# Delivery

Here is a quick overview of the Delivery process , from storing sources in git, building on koji/cbs and then pushing packages out

![CBS-SIGS-workflow](img/CBS-SIGs-workflow.png)

## Spins

In the following sections we'll focus on rpm packages and repositories but worth knowing that spins artifacts will be signed themselves, but the SHA256SUM files will be, allowing to verify images integrity

## Following signing/push process status

When you'll tag/untag pkgs/builds to specific tags (see below) it will be triggering a sign+push process outside of cbs/koji.

While (for obvious reasons) that signing infrastructure isn't publicly reachable, it still uses mqtt notifications to send status about queued jobs and the one[s] being processed or finished, (including number of other jobs in the signing queue)

It's a public (read-only) topic on MQTT broker , using your existing x509 TLS cert that you use to interact with cbs.

There are multiple ways to subscribe to a mqtt topic, but if you just need something lightweight, you can just install the simple `mosquitto` rpm package (available on Fedora and/or EPEL repositories) and then proceed like this : 

```

mosquitto_sub --cafile ~/.centos-server-ca.cert --cert ~/.centos.cert --key ~/.centos.cert -h mqtt.git.centos.org -p 8883 -t cbs-signing/# -v

```

This will stay connected to the topic and print received messages sent to mqtt cbs-signing topic. 

Example : 

```

cbs-signing {"time": "20230525-11:33:01", "tag": "kmods9s-packages-main-release", "status": "processing", "jobs_in_queue": "40"}

cbs-signing {"time": "20230525-11:38:29", "tag": "kmods9s-packages-main-testing", "status": "processed", "jobs_in_queue": "40"}

```

## Promoting to testing

By default, packages built on cbs are just tagged to `candidate` tag and stay in cbs/koji.

If you want your packages to get pushed to the [buildlogs](https://buildlogs.centos.org) mirror pool, you can tag packages to `testing`

!!! warning

    Worth knowing that only classical pkgs are pushed out, so no src.rpm nor debuginfo packages are sent to testing network (they'll be for `release` tag though)

If you want to tag multiple specific packages/versions to `testing`, you can proceed with one koji/cbs call :

```

cbs tag-build <sig_name>-<project>-<version>-testing <pkg1>-1.0.1 <pkg2>-2.3.4 <and_so_on>

```

This will trigger a message on the mqtt-based message bus and intercepted by the isolated machine processing requests. At this stage it will : 

  * verify which packages need to be signed with the [dedicated gpg key](https://www.centos.org/keys/#community-driven-project-keys) for the `SIG` 

  * download , sign and import back into koji signed packages

  * call koji for a `distRepo` tasks (preparing a usable repository with your packages) and wait for it to finish

  * sign repomd.xml file once repositories are all processed for all architectures

  * push that to the `buildlogs` CDN

In the next minutes, your up2date repository will appear under /centos/ on the [buildlogs](https://buildlogs.centos.org/centos/) nodes and so following the tag convention : 

```

<sig_name>/

├── <architecture>

│   ├── <project>-<version>

│   │   ├── Packages

│   │   └── repodata

```

As [buildlogs.centos.org](https://buildlogs.centos.org) has its own specific cdn, you can point your users willing to test your packages directly to such url (in your .repo, see below)

## Promoting to release and mirrors

Once you're satisfied with your package[s] quality (after some testing/feedback, up2you to decide when/how), you can proceed with next step, aka pushing to mirror network.

Same process as for `testing` except that it's now `release` tag : 

```

cbs tag-build <sig_name>-<project>-<version>-release <pkg1>-1.0.1 <pkg2>-2.3.4 <and_so_on>

```

This will trigger a message on the mqtt-based message bus and intercepted by the isolated machine processing requests. At this stage it will : 

  * verify which packages need to be signed with the [dedicated gpg key](https://www.centos.org/keys/#community-driven-project-keys) for the `SIG` 

  * download , sign and import back into koji signed packages

  * call koji for a `distRepo` tasks (preparing a usable repository with your packages) and wait for it to finish

  * sign repomd.xml file once repositories are all processed for all architectures (with dedicated gpg key)

  * push various packages to mirrors, depending on the CentOS Version (see below) :

### CentOS Linux 7 and CentOS Stream 8

The packages will appear on the existing mirror network, divided into three categories :

  * 'normal' rpm packages (that people will download/install) => pushed to [mirror.centos.org](http://mirror.centos.org) (and picked up by external mirrors too)

  * debuginfo rpm packages => pushed to [debuginfo.centos.org](https://debuginfo.centos.org)

  * src.rpm packages => pushed to [Vault](https://vault.centos.org)

### CentOS stream 9 and above

Starting from CentOS Stream 9, all packages will be pushed out in one simple directory.

All packages will be appearing on [mirror.stream.centos.org](http://mirror.stream.centos.org), under the SIGs directory (separated from distro content, for a clear distinction about distro versus SIGs generated content)

This is how it would look like for Stream 9 : 

```

SIGs/9-stream/<sig_name>/

├── <architecture>

│   └── <project>-<version>

│       ├── debug

│       │   └── repodata

│       ├── Packages

│       └── repodata

```

## Consuming rpms packages through .repo definition

When packages are signed and pushed to mirror network, they are automatically (for the `release` level) checked by the mirror crawler[s] and so you don't need to point your users to either mirror.stream.centos.org or mirror.centos.org.

Instead you can point to the correct `mirrorlist` or `metalink` url instead, depending on the CentOS Linux/Stream version : 

### CentOS Linux 7 and CentOS Stream 8

You can call mirrorlist.centos.org by specifying the repo name, archictecture and centos version like this : 

```

mirrorlist=http://mirrorlist.centos.org?release=<centos_release>&arch=<arch>&repo=<sig_name>-<project>-<version>

```

Example for the configmanagement sig producing the ansible (project) 29 (version) repo : 

```

 curl 'http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=configmanagement-ansible-29'

```

### CentOS Stream 9 and above

Starting from CentOS Stream 9 , mirrors are now added in [Fedora Mirrormanager](https://admin.fedoraproject.org/mirrormanager/mirrors/CentOS) so you have to call metalink= instead of mirrorlist=

As MirrorManager has already plenty of fedora/epel repositories to track, the reponame to use to query mirrormanager for metalink is more complex than for previous mirrorlist.centos.org setup.

The logic goes like this and notice the difference for `source` and `debuginfo` packages in the metalink uris:

```

# packages

metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-9-stream&arch=<basearch>

# source packages (.src.rpm)

metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-source-9-stream&arch=source

# debuginfo packages

metalink=https://mirrors.centos.org/metalink?repo=centos-<sig_name>-sig-<project>-<version>-debug-9-stream&arch=<basearch>

```

You can also always query MirrorManagers to see which metalinks/repositories it already knows about if you're unsure.

Here is an example for the `infra` SIG producing the `infra` project with version `common`: 

```

curl 'https://mirrors.centos.org/metalink?repo=unknown&arch=x86_64' -s | grep infra

# repo=centos-infra-sig-infra-common-9-stream&arch=aarch64

# repo=centos-infra-sig-infra-common-9-stream&arch=ppc64le

# repo=centos-infra-sig-infra-common-9-stream&arch=x86_64

# repo=centos-infra-sig-infra-common-debug-9-stream&arch=aarch64

# repo=centos-infra-sig-infra-common-debug-9-stream&arch=ppc64le

# repo=centos-infra-sig-infra-common-debug-9-stream&arch=x86_64

# repo=centos-infra-sig-infra-common-source-9-stream&arch=source

```

### centos-release-<sig> package

To make it convenient for end-users to add both the .repo files used by dnf/yum to automatically find new repositories, and also to ship the dedicated rpm gpg public key to verify the gpg integrity of the shipped packages, SIGs can build and ship a `centos-release-<sig>` package.

Worth knowing that such packages have to be built through specific cbs tags (see below) and not *your* SIG tag.

Indeed, SIGs content aren't "trusted" by default (at the rpm gpg level) but 8-stream/9-stream will start distributing the rpm gpg public key that will sign these specific centos-release-* packages, and so end-users will be able to `dnf install centos-release-<blah>` directly.

Once done, end-users will be able to download/consume your repositories.

To do so, you can create first an [infra ticket](https://pagure.io/centos-infra/issues) to create a project under the /rpms/ namespace on https://git.centos.org (in case it doesn't exist yet)

How should you name your 'centos-release' package ? Basically following the centos-release-<project> naming convention (see for example the [`openstack` project](https://git.centos.org/rpms/centos-release-openstack/branches), built by the Cloud SIG, and having multiple <versions> , each version for each supported centos distribution being a different branch)

At the minimum, your git project for your centos-release package should look like : 

```

├── .centos-release-<project>.metadata

├── SOURCES

│   ├── CentOS-SIG-<project>.repo

│   └── RPM-GPG-KEY-CentOS-SIG-<name>

└── SPECS

    └── centos-release-<project>.spec

```

You can then proceed as [described previously](git.md) to push to git and then [submit a build](cbs.md) against specific tags (verify through `cbs list-permissions --mine` that you can build/tag to specific 'extras' tags. If not, see with your SIG group chair/sponsor

!!! important

    Don't submit your build to your own SIG tag : instead use the dedicated `extras<8s,9s>-extras-common-release` tag, that each SIG chair will be able to build for