Delivery

Here is a quick overview of the Delivery process , from storing sources in git, building on koji/cbs and then pushing packages out

Spins

In the following sections we'll focus on rpm packages and repositories but worth knowing that spins artifacts will be signed themselves, but the SHA256SUM files will be, allowing to verify images integrity

Following signing/push process status

When you'll tag/untag pkgs/builds to specific tags (see below) it will be triggering a sign+push process outside of cbs/koji. While (for obvious reasons) that signing infrastructure isn't publicly reachable, it still uses mqtt notifications to send status about queued jobs and the one[s] being processed or finished, (including number of other jobs in the signing queue)

It's a public (read-only) topic on MQTT broker , using your existing x509 TLS cert that you use to interact with cbs. There are multiple ways to subscribe to a mqtt topic, but if you just need something lightweight, you can just install the simple mosquitto rpm package (available on Fedora and/or EPEL repositories) and then proceed like this :

mosquitto_sub --cafile ~/.centos-server-ca.cert --cert ~/.centos.cert --key ~/.centos.cert -h mqtt.git.centos.org -p 8883 -t cbs-signing/# -v

This will stay connected to the topic and print received messages sent to mqtt cbs-signing topic. Example :

cbs-signing {"time": "20230525-11:33:01", "tag": "kmods9s-packages-main-release", "status": "processing", "jobs_in_queue": "40"}
cbs-signing {"time": "20230525-11:38:29", "tag": "kmods9s-packages-main-testing", "status": "processed", "jobs_in_queue": "40"}

Promoting to testing

By default, packages built on cbs are just tagged to candidate tag and stay in cbs/koji.

If you want your packages to get pushed to the buildlogs mirror pool, you can tag packages to testing

If you want to tag multiple specific packages/versions to testing, you can proceed with one koji/cbs call :

cbs tag-build <sig_name>-<project>-<version>-testing <pkg1>-1.0.1 <pkg2>-2.3.4 <and_so_on>

This will trigger a message on the mqtt-based message bus and intercepted by the isolated machine processing requests. At this stage it will :

• verify which packages need to be signed with the dedicated gpg key for the SIG
• download , sign and import back into koji signed packages
• call koji for a distRepo tasks (preparing a usable repository with your packages) and wait for it to finish
• sign repomd.xml file once repositories are all processed for all architectures
• push that to the buildlogs CDN

In the next minutes, your up2date repository will appear under /centos/ on the buildlogs nodes and so following the tag convention :

<sig_name>/
├── <architecture>
│   ├── <project>-<version>
│   │   ├── Packages
│   │   └── repodata

As buildlogs.centos.org has its own specific cdn, you can point your users willing to test your packages directly to such url (in your .repo, see below)

Promoting to release and mirrors

Once you're satisfied with your package[s] quality (after some testing/feedback, up2you to decide when/how), you can proceed with next step, aka pushing to mirror network.

Same process as for testing except that it's now release tag :

cbs tag-build <sig_name>-<project>-<version>-release <pkg1>-1.0.1 <pkg2>-2.3.4 <and_so_on>

This will trigger a message on the mqtt-based message bus and intercepted by the isolated machine processing requests. At this stage it will :

• verify which packages need to be signed with the dedicated gpg key for the SIG
• download , sign and import back into koji signed packages
• call koji for a distRepo tasks (preparing a usable repository with your packages) and wait for it to finish
• sign repomd.xml file once repositories are all processed for all architectures (with dedicated gpg key)
• push various packages to mirrors, depending on the CentOS Version (see below) :

CentOS Linux 7 and CentOS Stream 8

The packages will appear on the existing mirror network, divided into three categories :

• 'normal' rpm packages (that people will download/install) => pushed to mirror.centos.org (and picked up by external mirrors too)
• debuginfo rpm packages => pushed to debuginfo.centos.org
• src.rpm packages => pushed to Vault

CentOS stream 9 and above

Starting from CentOS Stream 9, all packages will be pushed out in one simple directory. All packages will be appearing on mirror.stream.centos.org, under the SIGs directory (separated from distro content, for a clear distinction about distro versus SIGs generated content)

This is how it would look like for Stream 9 :

SIGs/9-stream/<sig_name>/
├── <architecture>
│   └── <project>-<version>
│       ├── debug
│       │   └── repodata
│       ├── Packages
│       └── repodata

Consuming rpms packages through .repo definition

When packages are signed and pushed to mirror network, they are automatically (for the release level) checked by the mirror crawler[s] and so you don't need to point your users to either mirror.stream.centos.org or mirror.centos.org.

Instead you can point to the correct mirrorlist or metalink url instead, depending on the CentOS Linux/Stream version :

CentOS Linux 7 and CentOS Stream 8

You can call mirrorlist.centos.org by specifying the repo name, archictecture and centos version like this :

mirrorlist=http://mirrorlist.centos.org?release=<centos_release>&arch=<arch>&repo=<sig_name>-<project>-<version>

Example for the configmanagement sig producing the ansible (project) 29 (version) repo :

 curl 'http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=configmanagement-ansible-29'

CentOS Stream 9 and above

Starting from CentOS Stream 9 , mirrors are now added in Fedora Mirrormanager so you have to call metalink= instead of mirrorlist=

As MirrorManager has already plenty of fedora/epel repositories to track, the reponame to use to query mirrormanager for metalink is more complex than for previous mirrorlist.centos.org setup.

The logic goes like this and notice the difference for source and debuginfo packages in the metalink uris:

# packages
metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-9-stream&arch=<basearch>
# source packages (.src.rpm)
metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-source-9-stream&arch=source
# debuginfo packages
metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-debug-9-stream&arch=<basearch>

You can also always query MirrorManagers to see which metalinks/repositories it already knows about if you're unsure. Here is an example for the infra SIG producing the infra project with version common:

curl 'https://mirrors.centos.org/metalink?repo=unknown&arch=x86_64' -s | grep infra
# repo=centos-infra-sig-infra-common-9-stream&arch=aarch64
# repo=centos-infra-sig-infra-common-9-stream&arch=ppc64le
# repo=centos-infra-sig-infra-common-9-stream&arch=x86_64
# repo=centos-infra-sig-infra-common-debug-9-stream&arch=aarch64
# repo=centos-infra-sig-infra-common-debug-9-stream&arch=ppc64le
# repo=centos-infra-sig-infra-common-debug-9-stream&arch=x86_64
# repo=centos-infra-sig-infra-common-source-9-stream&arch=source

centos-release-<sig> package

To make it convenient for end-users to add both the .repo files used by dnf/yum to automatically find new repositories, and also to ship the dedicated rpm gpg public key to verify the gpg integrity of the shipped packages, SIGs can build and ship a centos-release-<sig> package.

Worth knowing that such packages have to be built through specific cbs tags (see below) and not your SIG tag. Indeed, SIGs content aren't "trusted" by default (at the rpm gpg level) but 8-strea/9-stream will start distributing the rpm gpg public key that will sign these specific centos-release-* packages, and so end-users will be able to dnf install centos-release-<blah> directly.

Once done, end-users will be able to download/consume your repositories.

To do so, you can create first an infra ticket to create a project under the /rpms/ namespace on https://git.centos.org (in case it doesn't exist yet)

How should you name your 'centos-release' package ? Basically following the centos-release-<project> naming convention (see for example the openstack project, built by the Cloud SIG, and having multiple <versions> , each version for each supported centos distribution being a different branch)

At the minimum, your git project for your centos-release package should look like :

├── .centos-release-<project>.metadata
├── SOURCES
│   ├── CentOS-SIG-<project>.repo
│   └── RPM-GPG-KEY-CentOS-SIG-<name>
└── SPECS
    └── centos-release-<project>.spec

You can then proceed as described previously to push to git and then submit a build against specific tags (verify through cbs list-permissions --mine that you can build/tag to specific 'extras' tags. If not, see with your SIG group chair/sponsor