CentOS Infrastructure Team

Overview

CentOS Infra is mainly managed by the Community Platform Engineering Team, but also accepting infra contributions from Community Contributors, which can be delegated rights on parts of the infra for service[s] they'd like to contribute to, or even be responsible for.

All that is possible at various levels :

Application level : granted needed credentials (usually authenticated through ACO)
Ansible role level : through Pull Requests against our Ansible roles and/or inventories
Machine/Infra level : granted shell/elevated rights based on Ansible rights

Onboarding infra members

This is an overview of needed steps to onboard a new sysadmin, having so access everywhere :

Info

Worth knowing that all explained steps don't have to be all applied. Example: someone can be granted koji build right for the infra tags, because also of needed delegation for just koji/cbs.centos.org but not needing shell/access anywhere else, so don't apply blindly this process !

IPA Group membership

While being part of the sig-infra group doesn't grant any shell/sudo permission anywhere , it at least reflect that new person joigning the team will be a PoC for infra and also automatically granted :

rights to build/promote pkgs on the infra koji tags
a @centosproject.org email address (see also the postfix section)

Ansible inventory access (for "full" sysadmin)

Based on the Environment that new infra team member needs access to (delegation, as some are in charge of CentOS CI but not -yet- other parts, etc), one needs to be added in a specific ansible list/variables in inventory, the admins_list that contains list of shell accounts to create , with their ssh pub key and if they are granted sudo rights.

From that point, next time Ansible will be ran across servers fleet (either automatically through central mgmt station or manually for the machines in the manual-run specific group), it will add the new sysadmin (or modify/remove) on the nodes.

To retrieve a sysadmin ssh public key, gpg key or other needed informations, you can directly query IPA through fajson (it needs first to have a working kerberos ticket, so don't forget to kinit first) :

fas_user="arrfab"
curl --silent --fail --negotiate -u : https://fasjson.fedoraproject.org/v1/users/${fas_user}/|jq

It's required that CentOS Infra members, when they'll be in charge of multiple services and granted elevated rights, will :

use GnuPG for encryption (for mails and/or to be able to access ansible inventory git repositories) and so public gpg key available through fasjson/IPA/noggin portal
have their ssh pub key also passphrase protected and with publicy key available on fasjson/IPA/noggin
have enabled OTP on their account for authentication services

Once you have verified the GPG public key, you can (for git-crypted git repositories for ansible, add the new collaborator like this (do this on each git repo that the new sysadmin is granted access to). So after you've added the gpg in your own keyring, you can add it to git-crypt

git-crypt add-gpg-user --trusted <replace_with_user_email_listed_in_gpg)
[master e28a784] Add 1 git-crypt collaborator
 1 file changed, 1 insertion(+)
 create mode 100644 .git-crypt/keys/default/0/<usually_user_gpg_pub_key>.gpg
git push

Zabbix access (for "full" sysadmin)

One existing administrator can create (through WebUI - Administration/Users - or through Zabbix API), a new user with the following settings:

User section
Alias (username) => matching existing nick from authentication
Groups => can be Zabbix administrators for full admins
password => randomly generated and initial one sent to new sysadmin by gpg encrypted email, to be changed by user after (note that when we'll switch zabbix to new auth system, that step will be skipped completely)
Media section
At least add the email media type, sending to his registered email in IPA
Filter out Not classified and Information severity levels but users can change/fine tune after
Permissions section
User Type => Zabbix Super Admin : Read-Write everywhere in Zabbix

Various notes

IPA groups giving access

git/pagure

The centos-git-admins IPA group will give you all needed rights in pagure/git.centos.org, so add user in that group if he needs to be able to administer this git forge solution.

Openshift

Git notifications

For the current setup for inventories (not the ones hosted on gitlab that is) we can quickly enable commits notifications like this (adapt to the needs/users/projects) :

# Some variables
git_basedir="/repositories/git/centos"
git_repos="ansible-filestore-ci ansible-pkistore-ci ansible-inventory-ci"
git_mailto="rcpt_1@domain.com, rcpt2@otherdomain.com"
git_mailfrom="git@centosproject.org"

pushd ${git_basedir}
for repo in ${git_repos}; do
  pushd ${repo}.git 
  git config multimailhook.mailingList "${git_mailto}"
  git config multimailhook.from ${git_mailfrom}
  pushd hooks/post-receive.d; test -e git_multimail.py || ln -s /usr/bin/git_multimail.py ; popd
  popd
done
popd