From 27db0d66cde24d18bed12ed09fbd480a3b4f1471 Mon Sep 17 00:00:00 2001 From: Fabian Arrotin Date: Jul 06 2021 13:13:35 +0000 Subject: more documentation on koji/cbs Signed-off-by: Fabian Arrotin --- diff --git a/docs/buildsys/intro.md b/docs/buildsys/intro.md new file mode 100644 index 0000000..93bff7d --- /dev/null +++ b/docs/buildsys/intro.md @@ -0,0 +1,31 @@ +# Build Systems overview + +To build artifacts that will be released/pushed to the mirror network, CentOS Project relies on distributed build systems, mostly : + + * [Koji](https://pagure.io/koji) + * [reimzul](/buildsys/reimzul/) + +While reimzul is still being used for CentOS Linux 7 rpm builds, koji is now the one being used for CentOS Linux 8 and all Stream releases (but different environments) + +RPM packages built by/for Special Interest Groups on our [Community Build Service](/buildsys/koji-cbs/) are also built on a koji instance, different from the one[s] used to build CentOS Linux/Stream distributions. + +We adviced you to read upstream koji documentation to have details of all needed components and how they can be combined all together to be able to build/distribute packages, but here is a quick overview : + + * kojihub/kojiweb : this is the main component, itself needing: + * authentication system in place (using either TLS/x509 certificates of kerberos tickets) + * postgresql database access : to store build results, activity, etc + * NFS storage: to store all builds logs and rpm packages and composed repositories + * httpd running with proper TLS certificates to encrypt traffic between hub and builders + * kojid : multiple builders, responsible to pick tasks (on the hub), and then use mock to build rpm packages and send results back to the kojihub + * depending if one of these nodes is also in the `createrepo` channel, it will itself RW rights on the NFS volume to be able to update repodata after some builds + +Now you can read other sub-sections about dedicated koji and some specific notes around these implementations + +It's through worth knowing that *all* koji enviroments (almost) are deployed from the same Ansible roles (but with different settings, per different inventory or group variables) + +Some links to ansible roles: + + * [kojihub](https://github.com/centos/ansible-role-kojihub) + * [kojid](https://github.com/centos/ansible-role-kojid) + * [nfs-server](https://github.com/centos/ansible-role-nfs-server) + * (only used for SIGs)[stylo](https://github.com/centos/ansible-role-stylo) diff --git a/docs/buildsys/koji-cbs-sigs.md b/docs/buildsys/koji-cbs-sigs.md index 3e20635..682bd39 100644 --- a/docs/buildsys/koji-cbs-sigs.md +++ b/docs/buildsys/koji-cbs-sigs.md @@ -192,6 +192,7 @@ Tag structure: * tag level that can be either `candidate` , `testing` or `release` What's the influence of the `tag level` mentioned above ? + * candidate : ignored by signing service, build stays in koji, used to validate build itself or just internal unit-test happening through ci.centos.org for SIGs doing their tests in that environment * testing : processed by signing service but packages aren't signed, and just land on https://buildlogs.centos.org CDN, so just to have publicly available pkgs that can be promoted to `release` by SIG later * release : processed by signing service : packages are gpg signed with SIG key, and content appear on the mirror CDN and can be consummed by all diff --git a/docs/buildsys/koji-cbs.md b/docs/buildsys/koji-cbs.md index 1648f26..80e2031 100644 --- a/docs/buildsys/koji-cbs.md +++ b/docs/buildsys/koji-cbs.md @@ -1 +1,32 @@ # Community Build Service (https://cbs.centos.org) + +That infra is hosted in RDU2c (Community Cage) DC, with kojihub being publicly reachable, and kojid/builders restricted to internal vlan/zone (no default route) +It does *not* build any official CentOS Linux/Stream package used in the distribution, but is used to let community members build additional packages that can be built against/for CentOS Linux/Stream releases. + + +## Authentication + +The whole CBS/koji infra is using the centralized [Authentication service](/infra/authentication/) so both the infra components (services/nodes) and the users are authenticated with TLS certificates. + +That means that for each node, we need a valid [TLS cert signed by IPA](/security/tls/#ipadogtag-central-authentication). + +Same rule applies for `users` : they need to be authenticated with valid TLS certificated signed by same CA but we'll consider two kind of users: + + * service account[s]: used to run services (not real users) so can be [created](/security/tls/#tls-service-account) by infra team + * real users: they can use instructions to create their [own TLS cert](/infra/authentication/#tlsx509-authentication) + +## Koji tags structure + +When the Special Interest Groups ([SIG](https://wiki.centos.org/SpecialInterestGroup) in short) wanted to start building , the idea discussed on the centos-devel list (back in 2014) was to create some koji [tags](https://docs.pagure.org/koji/HOWTO/#package-organization) that would let people build/test/promote their packages that would then be pushed to the external mirrors CDN (while being signed with a specific [GPG key](/koji-cbs-sigs/#security-gpg-key-for-packages-signing)) + +The proposed and agreed levels are : + + * candidate: just used to build initially a package, test that it build and minimal CI tests + * pushed to external mirror : no + * signed with gpg key: no + * testing: based on SIG decision than can `tag-build` pkg in -testing for more external tests + * pushed to external mirrors: yes (https://buildlogs.centos.org) + * signed gith gpg key: no + * release: consider stable and tested enough by SIG so ready for public consumption + * pushed to external mirrors: yes (all mirrors) + * signed with gpg key: yes diff --git a/mkdocs.yml b/mkdocs.yml index 5899d29..64bc76b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -58,6 +58,7 @@ nav: - Build and Delivery: - buildsys/git.md - Build Systems: + - buildsys/intro.md - buildsys/koji-cbs.md - buildsys/koji-cbs-sigs.md - buildsys/koji-mbox.md