From 2d970326b01d88416255ce772822ac4d3b3a963a Mon Sep 17 00:00:00 2001 From: Fabian Arrotin Date: Dec 13 2021 15:37:47 +0000 Subject: Some doc around git/cbs but more to come (WIP) Signed-off-by: Fabian Arrotin --- diff --git a/docs/cbs.md b/docs/cbs.md index f93eb0b..82cb32d 100644 --- a/docs/cbs.md +++ b/docs/cbs.md @@ -1 +1,97 @@ # Koji/CBS buildsystem + +## CBS permissions + +!!! important + Before being able to submit build tasks to koji/cbs, you need to have a valid [account](auth.md) and be member of a SIG group ! So be sure you followed all steps described in [doc](auth.md) before trying any other step + +If you followed the needed steps to get your account and your workstation prepared with needed packages and TLS certs in place, we can proceed with basic validation. + +Let's try first to confirm that we can reach CBS/koji and be authenticated (assuming that your username is `user` : + +``` +# cbs hello + +g'day, ! + +You are using the hub at https://cbs.centos.org/kojihub/ +Authenticated via client certificate /home//.centos.cert + +``` + +Now, if you're member of a SIG group (identified as `sig-` , the following command should list your all your rights and you should see a corresponding `build-` permission : + +``` +# cbs list-permissions --mine +build +livecd +appliance +image +build-configmanagement +livemedia + +``` + +As you can see, cbs/koji is using a 1:1 relationship between FAS/ACO group membership (sig-) and the corresponding permissions to submit tasks against specific tags (see below) associated with that SIG. + +!!! note + Just worth knowing that when you're added to a SIG group it can take just some minutes due to replication between FreeIPA servers and fasjson.fedoraproject.org to catch up, as that's the endpoint that cbs.centos.org is using to compute/modify rights. So just keep that in mind and wait for permissions to be returned as shown above before trying anything else + + +## Understanding tags and targets + +### Koji naming convention + +A `build target` specifies the buildroot and destination of a package for submission. That means that as a user you ask koji to build a package within a specific build environment (build target) that is using a specific buildroot to find dependencies, etc. +A `Tag` is how koji is grouping packages all together, and built packages can be automatically added to a tag after a successfull build, or added manually between other tags with the `tag-build` koji task. + +### CBS SIGs tags structure + +As you understood, each `build target` in CBS koji instance will have at least four tags : + + * the -build one (which is where pkg is built and using the configured buildroot) + * three levels that will be used to promote packages after : + * candidate (default tag where built packages are tagged to) + * testing + * release + +Our koji tag structure *must* follow always the following structure (really for CBS, not the rule for koji but it's important for automation and release to mirror network steps) : + +`---{candidate,testing,release}` , where: + + * SIG : SIG name , corresponding to IPA group, without the `sig-` prefix (so `sig-cloud` would become `cloud`) + * CentOS Version: .el version, so 7 means CentOS 7, 8s means CentOS Stream 8, and so on + * Project : the project that the SIG want to build + * Version : project version, can be a number or a string + * repo variant: one of the following : candidate, testing or release (see Delivery about the meaning for these levels) + + +Let's use one example, the `Cloud` SIG would like to build Openstack (Yoga release) for CentOS Stream 9 + +The build target would be `cloud9s-openstack-yoga-el9s` and the tags would be : cloud9s-openstack-yoga-{candidate,testing,release} + + + +### Submit a build on CBS + +Before being able to submit a build task in koji, one has to be the pkg owner for it for all possible tags, so if you want to become the "owner" of a pkg for your own SIG/group, you can proceed like this (and replace `ACO_name`, `pkg_name` and `basetag` with real values : + +``` +for level in candidate testing release ; do + cbs add-pkg --owner= ${}-${level} +done +``` + +Now you can submit a build task by pointing to git.centos.org URL, pointing to project to build and git commit hash (assuming that you proceeded with needed steps in the [git](git.md) part : + +``` +cbs build git+https://git.centos.org/rpms/.git# +``` + +!!! note + you can always submit a test build by adding `--scratch` to the build command as argument : that would really build the pkg but wouldn't make it available anywhere to be used. Please avoid using --scratch though as it consumes resources on the build systems for nothing + + +### Promoting packages to mirror network + +See dedicated [Delivery](delivery.md) documentation diff --git a/docs/git.md b/docs/git.md index 225bd90..ed4b27a 100644 --- a/docs/git.md +++ b/docs/git.md @@ -1 +1,102 @@ # Git/lookaside + +Package sources checked into git.centos.org are in exploded SRPM format. This means that the package working directory should have at least a SPECS/ subdirectory. + +## New Package (from source) + +If the package you'd like to build isn't yet on https://git.centos.org, create first a ticket on https://pagure.io/centos-infra/issues to request the git repo for /rpms/ to be created + +Once it will be created, it will be possible for you to push to *specific* branches, based on the following rules: c{7,8}-* + +So that means that if I'm member of the sig-core group (in ACO), I'll be able to commit/push to c8-sig-core or c8-sig-core-whatever-I-want branches + +Let's use the following example : I need to create a branch for pkg centpkg-minimal, and I'm member of the sig-core SIG group. We have to push two things to git.centos.org : + + * .spec files to build a src.rpm + * eventually blob (.tar.gz archive) that will need to be pushed to lookaside cache + +!!! warning + To push to lookaside cache you need to have already a local check out from https://git.centos.org/centos-git-common + +## Pushing first to lookaside cache + +Let's assume that my pkg centpkg-minimal that I want to build has an archive called centpkg.minimal.tar.gz. To be able to push to lookaside cache, we need the following : + + * a valid TLS cert (also needed to build on cbs.centos.org) obtained through [centos-cert](../auth/) util + * lookaside_upload script in your $PATH (coming from centos-git-common git repo, see above) + +This simple script would need some paramters: + +``` +lookaside_upload + +You need to call the script like this : ~/bin/lookaside_upload -arguments + + -f : filename/source to upload (required, default:none) + -n : package name for that source (requred, default:none, example "httpd") + -b : "branch" where to upload to (required, default:none, example "c7-sig-core") + -h : display this help +``` + +So back to our example, the pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz and I'm member of the sig-core group, and want to build it for c7, so we'll call it like this : + +``` +lookaside_upload -f centpkg-minimal.tar.gz -n centpkg-minimal -b c7-sig-core +[+] CentOS Lookaside upload tool -> Checking if file already uploaded +[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside +[+] CentOS Lookaside upload tool -> URL : https://git.centos.org +[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz +[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal +[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 +[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core +[+] CentOS Lookaside upload tool -> ====== Trying to upload ======= + +################################################################################################################ 100.0% +File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK +[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded .... +[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 + +``` + +Now that we have uploaded to lookaside cache, we can reference it in our /rpms/centpkg-minimal git repository on git.centos.org, see below + + +## Pushing to git.centos.org + +We should already have a local checkout of the git repository for the pkg we'd like to work on. In our case, the git repo url is https://git.centos.org/rpms/centpkg-minimal + +So the way to git clone/pull over ssh is so ssh://git@git.centos.org/rpms/centpkg-minimal.git : + +``` +git clone ssh://git@git.centos.org/rpms/centpkg-minimal.git +Cloning into 'centpkg-minimal'... +remote: Counting objects: 15, done. +remote: Compressing objects: 100% (11/11), done. +remote: Total 15 (delta 2), reused 0 (delta 0) +Receiving objects: 100% (15/15), done. +Resolving deltas: 100% (2/2), done. +``` + +!!! important + You can clone over https, but then it wouldn't let you push back to it, as it needs ssh with key verification and based on group membership for acls + +Let's create a new c7-sig-core branch (still based on the assumption that I want to build for c7) : + +``` +git checkout -b c7-sig-core +``` + +You can now create your SPECS/.spec and add also other text patches under SOURCES/ We still need also to point to /sources/ for the lookaside cache, so we'll use the value of the hash returned when we successfully uploaded to lookaside cache and we'll write a ..metadata file in the root dir that will look like this in our example: + +``` +d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 SOURCES/centpkg-minimal.tar.gz +``` + +Now that we have pointer to lookaside cache, and also .spec, we can push back to git and we should be able to proceed with the "build-from-git" on cbs.centos.org. Let's commit first : + +``` +# git add # if needed +git commit -a +git push origin c7-sig-core # to create the c7-sig-core branch on git.centos.org if not existing yet + +```