Blame docs/buildsys/koji-cbs-sigs.md

12bb45
# Special Interest Groups
12bb45
12bb45
CentOS clearly defines what a SIG (Special Interest Group) is : a group of people sharing interest into a specific task/tech area, and willing to build/test/deliver artifacts (like RPM packages) that CentOS users can then consume.
12bb45
12bb45
Each SIG *must* be approved by the CentOS board, and no action should be taken before that. Once approved and CentOS board meeting minutes published, we can (Infra and Releng) proceed with next steps :
12bb45
12bb45
## SIG onboarding
0c748e
0c748e
### Creating SIG authentication Group
12bb45
We first need to create the SIG group in IPA (if it doesn't exist yet) and it shouldn't of course conflict with any existing.
12bb45
Requirements:
12bb45
12bb45
 * a user with `admin` right in IPA to be able to create the group
12bb45
 * a shell account on an enrolled machine (like `ipa-shell.iad2.centos.org` but can be any enrolled node within the @FEDORAPROJECT.ORG REALM (verify that you have a `valid` krb5 ticket or next steps will fail)
12bb45
 * a SIG name and description
12bb45
 * at least one SIG member that will be the chair (meaning that he'll be himself then able to add/remove other SIG users in that group
12bb45
12bb45
We can use either the IPA UI, or for automation purposes,just use the ipa cli.
12bb45
Basicall three commands:
12bb45
12bb45
  * `ipa group-add` : to create the group
12bb45
  * `ipa group-add-member` : to add the first user
12bb45
  * `ipa group-add-member-manager` : to grant "sponsor" right to sig chair, himself then able to add users
12bb45
12bb45
Here is a full output/example about creating the `sig-kmods` with one chair (valid IPA/FAS account) :
12bb45
12bb45
```
12bb45
# Defininig first some variables
12bb45
sig_name="sig-kmods"
12bb45
sig_description="The CentOS kmods SIG"
12bb45
sig_chair="pjgeorg"
12bb45
12bb45
# Now creating group and granting rights
12bb45
ipa group-add --desc="${sig_description}" --fasgroup ${sig_name}
12bb45
-----------------------
12bb45
Added group "sig-kmods"
12bb45
-----------------------
12bb45
  Group name: sig-kmods
12bb45
  Description: The CentOS kmods SIG
12bb45
  GID: 1494370508
12bb45
  FAS group: True
12bb45
12bb45
ipa group-add-member ${sig_name} --users=${sig_chair}
12bb45
  Group name: sig-kmods
12bb45
  Description: The CentOS kmods SIG
12bb45
  GID: 1494370508
12bb45
  Member users: pjgeorg
12bb45
  Membership managed by users: pjgeorg
12bb45
-------------------------
12bb45
Number of members added 1
12bb45
-------------------------
12bb45
12bb45
ipa group-add-member-manager ${sig_name} --users=${sig_chair}
12bb45
  Group name: sig-kmods
12bb45
  Description: The CentOS kmods SIG
12bb45
  GID: 1494370508
12bb45
  Membership managed by users: pjgeorg
12bb45
-------------------------
12bb45
Number of members added 1
12bb45
-------------------------
12bb45
12bb45
```
12bb45
If you need to add later more "sponsor" user, just repeat the `ipa group-add-member-manager` operation with new user.
12bb45
12bb45
12bb45
At this stage, the group should appear throug portal on https://accounts.centos.org
12bb45
12bb45
Worth knowing also that in the next minutes, an automatic email alias is created for the group following the naming convention `sig-<name>@centosproject.org` and that will automatically include all sig members email addresses, once added (and removed when removed)
12bb45
12bb45
Same for SIG members : an email alias following the `<user>@centosproject.org` alias is also created for all SIG members (and pointing to the email address they used when they registered their ACO/FAS account)
12bb45
More informations about this in the [Ansible postfix role](https://github.com/CentOS/ansible-role-postfix/blob/master/defaults/main.yml#L71)
12bb45
8ab8e7
Don't forget to also add the new sig-<group> to the `centosproject-email-aliases` group as indirect member so that all new SIG members will be granted an email alias, based ont he role described above.
0c748e
0c748e
### Creating SIG namespace on git.centos.org
0c748e
0c748e
Each SIG can push to specific branches on git.centos.org (computed automatically from some sig prefixes and sig name).
0c748e
To allow a new SIG group to have the automatic right in git.centos.org (and reflect group membership there though centralized authentication), we need to just add `sig-<name>` to the existing `pagure_supported_sigs` ansible list/vars in Ansible inventory.
0c748e
0c748e
Once pushed and applied (automatically by Ansible), when a SIG member will login again, his group membership will be synced and such user will be allowed to :
0c748e
0c748e
  * push to some specific branches
0c748e
  * push to lookaside
0c748e
0c748e
More informations about this in the Git section.
0c748e
0c748e
0c748e
### Welcome email
0c748e
12bb45
While not fully needed, it's now a good idea to send mail to that new SIG to welcome them.
12bb45
Something like this : 
12bb45
12bb45
```
12bb45
To: <sig_name>@centosproject.org
12bb45
Subject: Welcome to the new <sig_name> SIG !
12bb45
12bb45
Hi,
12bb45
12bb45
This welcome email means that your new group <sig_name> was created and granted some management rights to at least the SIG Chair.
12bb45
The group should appear now on https://accounts.centos.org/group/<sig_name>/
12bb45
12bb45
Also worth knowing that each SIG member is also automatically granted an email alias following this convention : <user>@centosproject.org
12bb45
12bb45
If you have any needs, requests, feel free to contact the CentOS Infra and Releng team through https://pagure.io/centos-infra/issues.
12bb45
12bb45
Kind Regards,
12bb45
```
12bb45
12bb45
## Security GPG key for packages signing
12bb45
12bb45
Each SIG building RPM packages through https://cbs.centos.org community build service will need a GPG security key that will be then automatically be used by the signing service
12bb45
5377d5
!!! danger
5377d5
    Due to sensitive information ,the following steps are restricted to a very limited number of people and on a restrictive node that doesn't have *any* direct inbound connection. Worth knowing that such node isn't reachable directly by the central ansible automation system, so ansible automation has to be ran "on demand" by people taking following actions.
5377d5
5377d5
We first need to create a new GPG key on the signing machine , protected with a strong passphrase, that will be needed to unlock signing service/agent.
5377d5
5377d5
To generate a strong passphrase you can for example use `openssl rand -base64 30` (or any other strong/long passphrase generator)
5377d5
5377d5
Let's use the `sig-kmods` SIG as an example for the next steps (and of course adapt on each operation, manual step)
5377d5
We first need to ssh as the signing user (important for TTY) on the signing service machine. Then we can proceed like this : 
5377d5
5377d5
```
5377d5
# Let's define some variables first
5377d5
sig_name="CentOS Kmods SIG"
5377d5
sig_url="https://wiki.centos.org/SpecialInterestGroup/Kmods"
5377d5
sig_key_passphrase="Of_Course_A_Real_Passphrase_generated_as_described_above" 
5377d5
# Creating the key in keyring
5377d5
gpg --batch --gen-key <
5377d5
Key-Type: 1
5377d5
Key-Length: 2048
5377d5
Subkey-Type: 1
5377d5
Subkey-Length: 2048
5377d5
Name-Real: ${sig_name}
5377d5
Name-Comment: ${sig_url}
5377d5
Name-Email: security@centos.org
5377d5
Passphrase: ${sig_key_passphrase}
5377d5
Expire-Date: 0
5377d5
EOF
5377d5
5377d5
gpg: key 7AE06D54 marked as ultimately trusted
5377d5
```
5377d5
5377d5
As we can see the key was created but we have first to unlock it now (in gpg-agent) with the generated passphrase, otherwise CBS signing service will be blocked (replace obviously gpg key id with the output above) :
5377d5
5377d5
```
5377d5
# Unlocking the key
5377d5
key_id=$(gpg --list-keys | egrep -B 1 "${sig_name}"|head -n 1|awk '{print $2}'|cut -f 2 -d '/')
5377d5
test_file=$(mktemp)
a194fb
timeout 30 gpg --batch --sign -b --use-agent --local-user ${key_id}! ${test_file} && rm ${test_file} && echo "Signing OK" || echo "Signing failed" 
5377d5
```
5377d5
5377d5
Now that CBS signing service is ready, we have to export the new key and so push it to the website (so that people can see it publicly to verify and also have it embedded in a centos-release-* rpm pkg : 
5377d5
5377d5
```
5377d5
# extract key
5377d5
gpg --export --armor ${key_id} > /var/tmp/RPM-GPG-KEY-CentOS-SIG-$(echo ${sig_name/CentOS/}|sed 's/SIG//')
5377d5
gpg --with-fingerprint /var/tmp/RPM-GPG-KEY-CentOS-SIG-$(echo ${sig_name/CentOS/}|sed 's/SIG//')
5377d5
5377d5
pub  2048R/7AE06D54 2021-06-22 CentOS Kmods SIG (https://wiki.centos.org/SpecialInterestGroup/Kmods) <security@centos.org>
5377d5
      Key fingerprint = 48EF 712E C5DD B68B 5280  BE45 5B8E 1A76 7AE0 6D54
5377d5
sub  2048R/E46DCBCB 2021-06-22
5377d5
5377d5
```
5377d5
5377d5
Now either submit a PR or directly do a git commit/push to [CentOS.org git repository](https://git.centos.org/centos/centos.org) and key will be available on website automatically after.
5377d5
Simply drop the `RPM-GPG-KEY-CentOS-SIG-<name>` under /keys/ directory and update /keys.md to add the link to download the key and show fingerprint
5377d5
5377d5
Here is the Kmods SIG [commit](https://git.centos.org/centos/centos.org/c/fc51c0fc590ebdc734906343ccc392298fda930d?branch=master) given a reference
5377d5
5377d5
!!! warning
5377d5
    Don't forget to also put the gpg passphrase in the git-crypted pkistore as it will be needed each time that we'll have to unlock the gpg keys in keyring (like on a reboot of the signing machine/service)
12bb45
5f299a
Last step is to then play the [stylo](https://github.com/centos/ansible-role-stylo) role with ansible after having updated the signing node `stylo_gpg_sigs` list in host_vars. Attention that gpg key id has to be written in lowercase though : 
5f299a
5f299a
  * update ansible inventory with new gpg keyid / sig name
5f299a
  * `ansible-playbook playbooks/role-stylo.yml --tags "config"`
5f299a
5f299a
12bb45
27793d
## CBS Koji tags 
27793d
### Overview
ffbaa1
CBS uses a specific naming convention for the koji tags that will be created.
ffbaa1
It's important that we follow that specific convention because the signing/pushing automated process parses the tags to know which gpg SIG key to use, and where to push content that will appear on the mirror network.
ffbaa1
ffbaa1
Tag structure:
ffbaa1
ffbaa1
 * SIG name
ffbaa1
 * CentOS Linux/Stream release (abbreviated)
ffbaa1
 * SIG project
ffbaa1
 * SIG project release
ffbaa1
 * tag level that can be either `candidate` , `testing` or `release`
ffbaa1
ffbaa1
What's the influence of the `tag level` mentioned above ? 
27db0d
ffbaa1
   * candidate : ignored by signing service, build stays in koji, used to validate build itself or just internal unit-test happening through ci.centos.org for SIGs doing their tests in that environment
ffbaa1
   * testing : processed by signing service but packages aren't signed, and just land on https://buildlogs.centos.org CDN, so just to have publicly available pkgs that can be promoted to `release` by SIG later
ffbaa1
   * release : processed by signing service : packages are gpg signed with SIG key, and content appear on the mirror CDN and can be consummed by all
ffbaa1
fa7e9f
### SIG koji tags
fa7e9f
fa7e9f
#### Creating tags
27793d
ffbaa1
To create tags, we have in place a tool that is maintained through ansible-role-kojihub and distributed on the kojihub node itself.
ffbaa1
That means that to be able to use it, you need to have local shell access and kick the following commands as the `koji` user , *or* eventually have admin rights and kick the tool from outside 
ffbaa1
ffbaa1
```bash
ffbaa1
/opt/cbs-tools/scripts/create_sig.sh 
ffbaa1
usage: /opt/cbs-tools/scripts/create_sig.sh -d <distribution> -s <signame(s)> -t <tag(s)>
ffbaa1
ffbaa1
This script generate new build target in koji for SIGS.
ffbaa1
ffbaa1
OPTIONS:
ffbaa1
   -a   ARCHES                   : Force arches e.g : "x86_64","aarch64 i686", etc...
ffbaa1
   -b                            : Enable non public bootstrap repo (SCLO SIG only)
ffbaa1
   -c   COLLECTION               : Enable collection in the buildroot e.g : mariadb100
ffbaa1
   -d   DISTRIBUTION             : 5 6 7 8 8s
ffbaa1
   -p   SIG PROJECT NAME         : cloud6-<openstack>, sclo-<mariadb100>, etc...
ffbaa1
   -r   SIG PROJECT RELEASE NAME : cloud6-openstack-<juno>
ffbaa1
   -s   SIG NAME                 : cloud
ffbaa1
   -t   DISTTAGS                 : el7 el7.centos el8_0 el8s
ffbaa1
   -x                            : delete old -build tag and then recreate.
ffbaa1
```
ffbaa1
ffbaa1
Let's use the Kmods SIG again as an example : 
ffbaa1
ffbaa1
```
ffbaa1
/opt/cbs-tools/scripts/create_sig.sh -a "x86_64" -s kmods -d 8s -p packages -r main -t el8
ffbaa1
* Checking distribution el8s configuration...
ffbaa1
 -> Checking kmods config...
ffbaa1
Creating user : kmods
ffbaa1
Using default options for kmods/packages
ffbaa1
Creating tag  : kmods8s-packages-main-candidate
ffbaa1
Creating tag  : kmods8s-packages-main-testing
ffbaa1
Creating tag  : kmods8s-packages-main-release
ffbaa1
 -> creating kmods8s-packages-main-el8
ffbaa1
Added external repo centos8s-cr to tag kmods8s-packages-main-el8-build (priority 5)
ffbaa1
Added external repo centos8s-extras to tag kmods8s-packages-main-el8-build (priority 10)
ffbaa1
Added external repo centos8s-powertools to tag kmods8s-packages-main-el8-build (priority 15)
ffbaa1
Added external repo centos8s-appstream to tag kmods8s-packages-main-el8-build (priority 20)
ffbaa1
Added external repo centos8s-baseos to tag kmods8s-packages-main-el8-build (priority 25)
ffbaa1
ffbaa1
```
ffbaa1
2acb89
!!! danger
2acb89
    If you have to create tags for SCLs, see the note below as we have to consolidate tags into specific one for signing purpose. We'll eventually backport that into automation script later
2acb89
2acb89
##### SCLs specific tags
2acb89
The Software Collection SIG has some specific requirements and pkgs that need to be tagged in buildroot and also tags being consolidated when processed for signing (all lands into *one* repository and not multiple ones, even if there quite some SCL tags.
2acb89
2acb89
Example below that we need to do when we have a request for a new SCL , and let's assume that it's a RH collection.We have already created the tag as in the process above, but we'll need to tune these tags a little bit and let's so first declare some variables :
2acb89
2acb89
```
2acb89
tag="sclo7-rh-redis6-rh"
2acb89
collection="rh-redis6"
2acb89
origin="rh" # can be sclo or rh, needed below
2acb89
koji add-group-pkg ${tag}-el7-build srpm-build scl-utils-build ${collection}-build
2acb89
koji add-group-pkg ${tag}-el7-build build scl-utils-build ${collection}-build
2acb89
2acb89
for level in testing release ;  do
2acb89
  last_priority=$(koji taginfo sclo7-all-${origin}-${level}|tail -n 1 |awk '{print $1}')
2acb89
  koji add-tag-inheritance sclo7-all-${origin}-${level} ${tag}-${level}  --priority $(( $last_priority + 5 ))
2acb89
done
2acb89
2acb89
```
2acb89
2acb89
fa7e9f
#### Modifying existing tags
27793d
27793d
It's always possible to modify some existing tags through standard `koji` (or `cbs` alias) CLI calls.
27793d
Example : let's add additional architectures for the Kmods SIG :
27793d
27793d
``` 
27793d
for i in el8-build candidate testing release ; do koji edit-tag --arches="x86_64 ppc64le aarch64" kmods8s-packages-main-${i}; koji taginfo kmods8s-packages-main-${i}|grep Arches ; done
27793d
```
27793d
fa7e9f
If some SIGs want to depend on external repo, like to get all -devel packages from CentOS Stream, we can also add it :
fa7e9f
fa7e9f
```
2bfbad
externalrepo_name="centos8s-buildroot"
fa7e9f
externalrepo_url="http://kojifiles.rdu2.centos.org/kojifiles/repos/dist-c8-stream-build/latest/\$arch"
fa7e9f
sig_tag="hyperscale8s-packages-experimental-el8-build"
fa7e9f
koji add-external-repo --mode=bare --tag ${sig_tag} ${externalrepo_name} ${externalrepo_url}
fa7e9f
koji taginfo ${sig_tag}
fa7e9f
fa7e9f
```
fa7e9f
fa7e9f
If external repo is already defined, we just need to add it to tags and no need to redefine it : 
fa7e9f
fa7e9f
```
2bfbad
sig_tag="hyperscale8s-packages-main-el8-build"
429202
externalrepo_name="epel8"
2bfbad
koji add-external-repo --mode bare --tag=${sig_tag} ${externalrepo_name}
fa7e9f
```
27793d
94b4f8
We can also modify the rpm dist tag through rpm macro that will be added to the specific tag :
94b4f8
94b4f8
```
94b4f8
tag="automotive9s-packages-experimental-el9s-build"
94b4f8
cbs edit-tag ${tag} --extra rpm.macro.dist='.el9iv'
94b4f8
# confirm with `cbs taginfo $tag`
94b4f8
```
61a828
61a828
#### Media spins with Kiwi
61a828
61a828
If SIGs are asking to be able to produce media with kiwi plugin, we have to ensure that there is at least one (or more) kojid builders which has/have the `kiwi` plugin enabled (through role).
61a828
Then at the SIGs tag level, we need to ensure that we add the [mandatory](https://docs.pagure.org/koji/plugins/?highlight=kiwi#image-builds-using-kiwi) `kiwi-build` group. (RPM packages list can vary depending on SIG needs but kiwi-cli and kiwi-systemdeps are mandatory)
61a828
61a828
```
61a828
build_tag="hyperscale9s-spin_media-main-el9s-build"
61a828
koji add-group ${build_tag} kiwi-build
61a828
koji add-group-pkg ${build_tag} kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs
61a828
```
61a828
61a828
!!! warning
61a828
    Worth knowing that some of these deps are satisfied *only* through Epel repository so it should first be enabled as external repo to the build tag, otherwise they'll not be installed in the buildroot