From 6f32ae6829557a7d2eb8f5e410d1f9351ad27193 Mon Sep 17 00:00:00 2001 From: Fabian Arrotin Date: Jul 02 2021 13:52:12 +0000 Subject: More documentation, including for infra team member onboarding Signed-off-by: Fabian Arrotin --- diff --git a/docs/ansible/ara.md b/docs/ansible/ara.md index 3f0697d..05f5056 100644 --- a/docs/ansible/ara.md +++ b/docs/ansible/ara.md @@ -1 +1,16 @@ -# this is how we use ARA +# ARA and central mgmt node + +While `sysadmins` having ssh/sudo rights on servers can trigger themselves remotely ad-hoc or role tasks through ansible from their main station, that's *not* the best practice. + +Based on the Env, we have usually (can depend on ENV requirements), one [host](https://github.com/CentOS/ansible-role-ansible-host) that is used to control the whole Infra/ENV. + +On that host, we use [ARA](https://ara.recordsansible.org/) to keep track of playbooks execution on that host, while we also have `log_path` set to also log to on-disk log files (rotated) + +So the workflow goes like this : + + * sysadmin with RWC rights pushes needed change[s] to either `inventory`, `filestore` or `pkistore` git repo + * two cases : + * it can wait next automatic execution: do nothing and ansible will deploy your change (like for example a simple TLS cert replace and reload) when the next (cron) "play all roles on all nodes" task will run + * it has to be done `now` : you kick the role task from the central ansible host to be ran directly + + diff --git a/docs/ansible/topology.md b/docs/ansible/topology.md new file mode 100644 index 0000000..da3f6e5 --- /dev/null +++ b/docs/ansible/topology.md @@ -0,0 +1,13 @@ +# Ansible Environments + +As explained in the [Overview](/#available-environments), we use different ansible inventories based on the target infra that we'll target. + +Apart from the roles and playbooks which are all common for *all* environments, each env will use its own : + + * inventory + * filestore + * pkistore (shared though for `main`, `stg` and `dev`, but different for the other ones) + +Based on the requirements and on which git forge solution it's hosted on, people will be granted RWC (Read / Write / Commit) rights on the specific (non public, for obvious reasons) git repositories + +Worth also knowing that, depending on the Env requirements, some will be fully encrypted (with `git-crypt`) and other will have a mix of readable/encrypted content (with `ansible-vault`) so that (even if private repositories), some other team members can submit PR/MR against inventory without having access to the stored (and encrypted) credentials diff --git a/docs/apps/mailservers.md b/docs/apps/mailservers.md new file mode 100644 index 0000000..7bc91e7 --- /dev/null +++ b/docs/apps/mailservers.md @@ -0,0 +1,7 @@ +# Mail servers + +## @centos.org setup +### aliases + +## @centosproject.org setup +### aliases diff --git a/docs/index.md b/docs/index.md index 42ddf65..0c25115 100644 --- a/docs/index.md +++ b/docs/index.md @@ -45,3 +45,8 @@ Let's just have a quick look at the existing environments, *each* using its own * `CentOS dev` (DEV) : really ephemeral setup pointing to very low spec machines (usually VMs) to test new stack/applications and write automation before being then deployed in `CentOS staging` * `CentOS CI` : everything that is configuring/deploying the infra behind `ci.centos.org` domain (public or internal) * `CentOS Stream MVBE` : dedicated/isolated environment for CentOS Stream 9 buildsys and having its own inventory/rollout strategy + + +## CentOS Infra team + +See the [dedicated section](/infra/team/) diff --git a/docs/infra/authentication.md b/docs/infra/authentication.md index 9819a55..8662c6b 100644 --- a/docs/infra/authentication.md +++ b/docs/infra/authentication.md @@ -22,7 +22,7 @@ Same goes for the TLS certificates used on the haproxy reverse proxy : automatic ### Identity Provider (IdP) -We deploy our own IdP instance, based on [Ipsilon](https://ipsilon-project.org/) that is publicly available on https://id.centos.org. +We deploy our own IdP instance, based on [Ipsilon](https://ipsilon-project.org/) that is publicly available on [https://id.centos.org](https://id.centos.org). It's full deployed by the [ipsilon](https://github/centos/ansible-role-ipsilon) Ansible role but needs access through fedora network as it's not directly available from outside @@ -36,7 +36,7 @@ Applications using OpenID can point directly to https://id.centos.org and some a OpenIDC is preferred over OpenID but needs some configuration at both IdP and Application side : - * on https://id.centos.org : login as account with admin right in ipsilon (managed by Ansible inventory), and create new OpenIDC app / client ID / secret / oauth callback (basically original URL callback endpoint) + * on [https://id.centos.org](https://id.centos.org) : login as account with admin right in ipsilon (managed by Ansible inventory), and create new OpenIDC app / client ID / secret / oauth callback (basically original URL callback endpoint) * on the client application side : reflect all client id / secrets / oauth callback #### SAML diff --git a/docs/infra/monitoring.md b/docs/infra/monitoring.md index e69de29..c18f9b9 100644 --- a/docs/infra/monitoring.md +++ b/docs/infra/monitoring.md @@ -0,0 +1,10 @@ +# Monitoring stack + +## Overview + +We use [Zabbix](https://www.zabbix.com) to monitor the whole CentOS Infra/servers fleet since 2008, and have upgraded from major version to major version without any issue. + + + +## TLS usage + diff --git a/docs/infra/team.md b/docs/infra/team.md new file mode 100644 index 0000000..b44ea09 --- /dev/null +++ b/docs/infra/team.md @@ -0,0 +1,108 @@ +# CentOS Infrastructure Team + +## Overview + +CentOS Infra is mainly managed by the [Community Platform Engineering](https://docs.fedoraproject.org/en-US/cpe/) Team, but also accepting infra contributions from Community Contributors, which can be delegated rights on parts of the infra for service[s] they'd like to contribute to, or even be responsible for. + +All that is possible at various levels : + + * Application level : granted needed credentials (usually authenticated through [ACO](/infra/authentication/)) + * Ansible role level : through Pull Requests against our [Ansible](/ansible/) roles and/or inventories + * Machine/Infra level : granted shell/elevated rights based on Ansible rights + +## Onboarding infra members + +This is an overview of needed steps to onboard a new sysadmin, having so access everywhere : + +!!! info + Worth knowing that all explained steps *don't* have to be all applied. + Example: someone can be granted `koji` build right for the infra tags, because also of needed delegation for just koji/cbs.centos.org but not needing shell/access anywhere else, so don't apply *blindly* this process ! + + +### IPA Group membership + +While being part of the `sig-infra` group doesn't grant any shell/sudo permission *anywhere* , it at least reflect that new person joigning the team will be a PoC for infra and also automatically granted : + + * rights to build/promote pkgs on the [infra koji tags](https://cbs.centos.org/koji/search?match=glob&type=tag&terms=infra*) + * a `@centosproject.org` email address (see also the [postfix](/infra/mailservers) section) + +### Ansible inventory access (for "full" sysadmin) + +Based on the [Environment](/#available-environments) that new infra team member needs access to (delegation, as some are in charge of CentOS CI but not -yet- other parts, etc), one needs to be added in a specific ansible list/variables in inventory, the [admins_list](https://github.com/CentOS/ansible-role-baseline/blob/master/defaults/main.yml#L5) that contains list of shell accounts to create , with their ssh pub key and if they are granted sudo rights. + +From that point, next time Ansible will be ran across servers fleet (either automatically through central mgmt station *or* manually for the machines in the `manual-run` specific group), it will add the new sysadmin (or modify/remove) on the nodes. + +To retrieve a sysadmin ssh public key, gpg key or other needed informations, you can directly query IPA through fajson (it needs first to have a working kerberos ticket, so don't forget to `kinit` first) : + +``` +fas_user="arrfab" +curl --silent --fail --negotiate -u : https://fasjson.fedoraproject.org/v1/users/${fas_user}/|jq +``` + +!!! danger "" + It's *required* that CentOS Infra members, when they'll be in charge of multiple services and granted elevated rights, will : + + * use GnuPG for encryption (for mails and/or to be able to access ansible inventory git repositories) and so public gpg key available through fasjson/IPA/noggin portal + * have their ssh pub key also passphrase protected and with publicy key available on fasjson/IPA/noggin + * have enabled OTP on their account for authentication services + + +Once you have verified the GPG public key, you can (for `git-crypt`ed git repositories for ansible, add the new collaborator like this (do this on *each* [git repo](ansible/topology/) that the new sysadmin is granted access to). +So after you've added the gpg in your own keyring, you can add it to git-crypt + +``` +git-crypt add-gpg-user --trusted .gpg +git push +``` + +### Zabbix access (for "full" sysadmin) + +One existing administrator can create (through WebUI - Administration/Users - or through Zabbix API), a new user with the following settings: + + * User section + * Alias (username) => matching existing `nick` from authentication + * Groups => can be `Zabbix administrators` for full admins + * password => randomly generated *and* initial one sent to new sysadmin by gpg encrypted email, to be changed by user after (note that when we'll switch zabbix to new auth system, that step will be skipped completely) + * Media section + * At least add the `email` media type, sending to his registered email in IPA + * Filter out `Not classified` and `Information` severity levels but users can change/fine tune after + * Permissions section + * User Type => `Zabbix Super Admin` : Read-Write *everywhere* in Zabbix + +### Various notes + +#### IPA groups giving access + +##### git/pagure + +The `centos-git-admins` IPA group will give you all needed rights in pagure/git.centos.org, so add user in that group if he needs to be able to administer this git forge solution. + +##### Openshift + + +#### Git notifications + +For the current setup for inventories (not the ones hosted on gitlab that is) we can quickly enable commits notifications like this (adapt to the needs/users/projects) : + +``` +# Some variables +git_basedir="/repositories/git/centos" +git_repos="ansible-filestore-ci ansible-pkistore-ci ansible-inventory-ci" +git_mailto="rcpt_1@domain.com, rcpt2@otherdomain.com" +git_mailfrom="git@centosproject.org" + +pushd ${git_basedir} +for repo in ${git_repos}; do + pushd ${repo}.git + git config multimailhook.mailingList "${git_mailto}" + git config multimailhook.from ${git_mailfrom} + pushd hooks/post-receive.d; test -e git_multimail.py || ln -s /usr/bin/git_multimail.py ; popd + popd +done +popd +``` + + diff --git a/docs/operations/deploy/bare-metal.md b/docs/operations/deploy/bare-metal.md index b2177a6..dffa3a4 100644 --- a/docs/operations/deploy/bare-metal.md +++ b/docs/operations/deploy/bare-metal.md @@ -24,7 +24,7 @@ Once we have `dial tone` on the hardware side (oob/mgmt vlan), we need to ensure If we want ansible to automatically deploy it, we'll just have to add the node in the inventory and ensure that the /host_vars/ will have at least : - * following variables set : ` + * following variables set : * ipmi_ip`, `ipmi_user`, `ipmi_pass` : used to remotely pxe boot the node * `ip` , `gateway`, `netmask` and `dns` (usually apart from `ip`, which is unique, the rest is coming through inheritance * based on group inheritance, ensure that variables documented in [adhoc-provision-node.yml](https://github.com/CentOS/ansible-infra-playbooks/blob/master/adhoc-provision-node.yml) are also defined diff --git a/docs/security/iptables.md b/docs/security/iptables.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/security/iptables.md diff --git a/docs/security/ssh.md b/docs/security/ssh.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/security/ssh.md diff --git a/mkdocs.yml b/mkdocs.yml index 40d06b5..78d5dce 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -86,12 +86,16 @@ nav: - infra/sponsors.md - infra/centos-ci.md - infra/monitoring.md + - infra/team.md - Security: - security/index.md - security/tls.md - security/gpg.md + - security/iptables.md + - security/ssh.md - Ansible: - ansible/index.md + - ansible/topology.md - ansible/ara.md - Tips and Tricks: - tips/mdadm.md