From fea7395741a814a6870d4540ac39cb9dc3bff59b Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Apr 21 2022 10:52:58 +0000 Subject: Start documenting the changes made to the build infrastructure This includes the changes made to the lookaside cache, the changes made to use the flat dist-git layout in the git repositories as well as the changes that using gitlab.com to host the git repositories bring Signed-off-by: Pierre-Yves Chibon --- diff --git a/docs/git.md b/docs/git.md index d43d6b9..a9f8cf0 100644 --- a/docs/git.md +++ b/docs/git.md @@ -1,19 +1,29 @@ -# Git/lookaside +# Lookaside/git -Package sources checked into git.centos.org are in exploded SRPM format. This means that the package working directory should have at least a SPECS/ subdirectory. +This page documents how SIGs can upload the sources of their RPM into the lookaside +cache and how they should set-up their corresponding git repository. -## New Package (from source) +It should be noted that two model exists, the "traditional" model and the one +based on CentOS Stream and Fedora. While both model can be intertwined in some +places, it is recommended that SIGs pick a model and follow it. This documentation +will not cover which aspect of which model is compatible with the other one. -If the package you'd like to build isn't yet on https://git.centos.org, create first a ticket on https://pagure.io/centos-infra/issues to request the git repo for /rpms/ to be created +## New Package (from source) -Once it will be created, it will be possible for you to push to *specific* branches, based on the following rules: c{7,8}-* +If you SIG uses [https://git.centos.org](https://git.centos.org) and the package +you want to build is not there yet, you will have to create a ticket first at: +[https://pagure.io/centos-infra/issues](https://pagure.io/centos-infra/issues) +to request that the git repo for `/rpms/` be created. -So that means that if I'm member of the sig-core group (in ACO), I'll be able to commit/push to c8-sig-core or c8-sig-core-whatever-I-want branches +If your SIG uses [https://gitlab.com/CentOS](https://gitlab.com/CentOS), you, +your SIG chair or people that have been granted access by the SIG chair, will be +able to create the project for your package directly. -Let's use the following example : I need to create a branch for pkg centpkg-minimal, and I'm member of the sig-core SIG group. We have to push two things to git.centos.org : +In the rest of the document we will use the following example : Build the package `centpkg-minimal`, +as a member of the `sig-core` SIG group. We have to push two things: - * .spec files to build a src.rpm - * eventually blob (.tar.gz archive) that will need to be pushed to lookaside cache + * the .spec file to build the rpm, to the git repository + * eventually an archive (.tar.gz, .tar, .xz...) of the sources, to lookaside cache !!! warning To push to lookaside cache you need to have already a local check out from [https://git.centos.org/centos-git-common](https://git.centos.org/centos-git-common) @@ -23,29 +33,51 @@ Let's use the following example : I need to create a branch for pkg centpkg-mini Let's assume that my pkg centpkg-minimal that I want to build has an archive called centpkg.minimal.tar.gz. To be able to push to lookaside cache, we need the following : * a valid TLS cert (also needed to build on cbs.centos.org) obtained through [centos-cert](../auth/) util - * lookaside_upload script in your $PATH (coming from centos-git-common git repo, see above) + * either script `lookaside_upload` or `lookaside_upload_sig` in your $PATH (coming from centos-git-common git repo, see above) + +There are two path available for the lookaside cache, each is available using a different script: + +- `lookaside_upload` allows uploading to the traditional CentOS Linux structure: `baseurl/pkgname/branch/hash` ([example]( +https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05eb775f)) +- `lookaside_upload_sig` allows uploading to the same lookaside structure as the one used by CentOS Stream and Fedora: `baseurl/pkgname/tarball/hashtype/hash/tarball` ([example]( +https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.tar.xz/sha512/f7aeac0fe5bf594933cd35b7ecc94ea8ddcbfedc04fa769c4da298e7bf105df116375d44711d944c748c85f61f96f6149be34c76eb37f28aa1f16359a9122abf/linux-5.14.0-62.el9.tar.xz)) + +Both scripts need some paramters, simply use `-h` to see them. -This simple script would need some paramters: +Back to our example. + +### Pushing to the "traditional" lookaside structure + +The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz and I'm member of the sig-core group, and want to build it for c7, so we'll call it like this : ``` -lookaside_upload +lookaside_upload -f centpkg-minimal.tar.gz -n centpkg-minimal -b c7-sig-core +[+] CentOS Lookaside upload tool -> Checking if file already uploaded +[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside +[+] CentOS Lookaside upload tool -> URL : https://git.centos.org +[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz +[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal +[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 +[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core +[+] CentOS Lookaside upload tool -> ====== Trying to upload ======= -You need to call the script like this : ~/bin/lookaside_upload -arguments +################################################################################################################ 100.0% +File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK +[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded .... +[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 - -f : filename/source to upload (required, default:none) - -n : package name for that source (requred, default:none, example "httpd") - -b : "branch" where to upload to (required, default:none, example "c7-sig-core") - -h : display this help ``` -So back to our example, the pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz and I'm member of the sig-core group, and want to build it for c7, so we'll call it like this : +### Pushing to the CentOS Stream/Fedora lookaside structure + +The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz, so we'll call it like this : ``` -lookaside_upload -f centpkg-minimal.tar.gz -n centpkg-minimal -b c7-sig-core +lookaside_upload_sig -f centpkg-minimal.tar.gz -n centpkg-minimal [+] CentOS Lookaside upload tool -> Checking if file already uploaded [+] CentOS Lookaside upload tool -> Initialing new upload to lookaside [+] CentOS Lookaside upload tool -> URL : https://git.centos.org -[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz +[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz [+] CentOS Lookaside upload tool -> Package name: centpkg-minimal [+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 [+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core @@ -58,14 +90,27 @@ File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc ``` -Now that we have uploaded to lookaside cache, we can reference it in our /rpms/centpkg-minimal git repository on git.centos.org, see below +Now that we have uploaded to lookaside cache, we can reference it in our /rpms/centpkg-minimal git repository on git.centos.org, see below + + +## Pushing to git + +SIGs have two options when it comes to the git repositories which can be used as part of dist-git: +- [git.centos.org](https://git.centos.org) +- [gitlab.com/CentOS](https://gitlab.com/CentOS) -## Pushing to git.centos.org +If your SIG wants to use gitlab, they will have to follow the [instructions](gitlab.md) to request one. +Here we'll cover both workflow, assuming the git repository exists on either platform. -We should already have a local checkout of the git repository for the pkg we'd like to work on. In our case, the git repo url is https://git.centos.org/rpms/centpkg-minimal +### Git clone -So the way to git clone/pull over ssh is so ssh://git@git.centos.org/rpms/centpkg-minimal.git : +The first thing you'll need is a local checkout of the git repository for the pkg we'd like to work on. +In our case, the git repo url is `https://git.centos.org/rpms/centpkg-minimal` or +`https://gitlab.com/CentOS//rpms/.git` + +So the way to git clone/pull over ssh is either `ssh://git@git.centos.org/rpms/centpkg-minimal.git` or +`git@gitlab.com:CentOS//rpms/.git`: ``` git clone ssh://git@git.centos.org/rpms/centpkg-minimal.git @@ -78,26 +123,119 @@ Resolving deltas: 100% (2/2), done. ``` !!! important - You can clone over https, but then it wouldn't let you push back to it, as it needs ssh with key verification and based on group membership for acls + On git.centos.org you can clone over https, but then it wouldn't let you push back to it, as it needs ssh with key verification and based on group membership for acls + +### Git branches + +Repositories git.centos.org are used by both Red Hat and CentOS SIG, the structure of the git +repository is used to control the access. SIG members are therefore only allowed to push to branches +following the pattern: `c-sig-*` -Let's create a new c7-sig-core branch (still based on the assumption that I want to build for c7) : +That means that if I'm member of the sig-core group (in ACO), I'll be able to +commit/push to `c7-sig-core`, `c8-sig-core` or `c9s-sig-core-whatever-I-want` branches + +For our example, on git.centos.org we'll have to create a new c7-sig-core branch +(with on the assumption that we want to build for c7) : ``` git checkout -b c7-sig-core ``` -You can now create your SPECS/.spec and add also other text patches under SOURCES/ We still need also to point to /sources/ for the lookaside cache, so we'll use the value of the hash returned when we successfully uploaded to lookaside cache and we'll write a .``.metadata file in the root dir that will look like this in our example: +Since SIGs have full control over their namespace on gitlab.com, that branching structure +is not mandatory. It will, however, be necessary if you plan on using the "traditional" lookaside +structure describe above as this relies on branch names. + +### Git repository layout + +There are two possible structure possible for the repository layout: + +- The exploded SRPM layout +- The flat dist-git layout + +#### The exploded SRPM layout + +This is the "traditional" layout used in CentOS Linux, it is articulated around two folders: ``` -d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 SOURCES/centpkg-minimal.tar.gz +├── SOURCES +└── SPECS ``` -Now that we have pointer to lookaside cache, and also .spec, we can push back to git and we should be able to proceed with the "build-from-git" on cbs.centos.org. Let's commit first : +The `SPECS` folder is meant to receive your spec file and all other text files and +patches would go under `SOURCES`: + +``` +├── SOURCES +│ ├── +│ └── +└── SPECS + └── .spec +``` + +Once done, we still need to link these files with the sources uploaded earlier in the +lookaside cache. See the next section. + +#### The flat dist-git layout + +This is the layout used in the dist-git repositories in CentOS Stream and Fedora. +All files (spec files, text files, patches...) are stored at the top level of +the repository: + +``` +├── +├── +└── .spec +``` + +### Linking to sources in the lookaside cache + +Here as well, there are two ways you can link a dist-git repository to sources +uploaded to the lookaside cache: + +- the "traditional" way using a `..metadata` file +- the CentOS Stream/Fedora way using a `sources` file !!! important - Even if your package doesn't contain any source pushed to lookaside cache (like for a package just having some small files in SOURCES/ dir, you *need* to have the `..metadata` package present and pushed in git repository + Even if your package doesn't contain any source pushed to lookaside cache + (like for a package just having some small files in SOURCES/ dir, you *need* to + have either a `..metadata` or `sources` file present and pushed in + git repository -Here is what the git repository for a pkg should look like : +#### Using a `..metadata` file + +To use this method, simply create a `..metadata` file with the following +content: +``` + SOURCES/ +``` + +For example: +``` +d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 SOURCES/centpkg-minimal.tar.gz +``` + +#### Using a `sources` file + +To use this method, simply create a `sources` file. The easiest way to achieve +this is: +``` +sha512sum --tag > sources +``` + +Example output: +``` +SHA512 (centpkg-minimal-1.1.0.tar.gz) = f7cfbc956199b1a0342f321a0e4cb055d6ac2c784a8faace43cf80772f0de34b58ede1a02a558cd03e25c14938ac6e17b3746b1919c0bbc1f5b2955472577e4f +``` + +Now that we have pointer to lookaside cache, and also .spec, we can push back to git and +we should be able to proceed with the "build-from-git" on cbs.centos.org. Let's commit first. + +### Git push + +Depending on if you picked the "traditional" structure or the CentOS Stream/Fedora +one, this is how your repository should look like: + +* Traditional layout ``` . @@ -107,21 +245,28 @@ Here is what the git repository for a pkg should look like : │ └── └── SPECS └── .spec +``` +* CentOS Stream/Fedora layout ``` +. +├── sources +├── +├── +└── .spec +``` -You can now push to to git, as usual: +We can now push to to git, as usual: ``` # git add # if needed -git commit -a -git push origin c7-sig-core # to create the c7-sig-core branch on git.centos.org if not existing yet - +git commit -a +git push origin # to create the remote branch if not existing yet ``` -Now that we have our sources pushed to both git.centos.org and lookaside cache, we can now proceed with a build in cbs/koji +Where `` would be `c7-sig-core` for our example using `git.centos.org`. -!!! note - It's worth knowing that the documented process and structure is the default one, aka `rpmbuild style` layout for how your files/sources should be declared in your git repository/branch. But since 2022-02-21, the `flat layout` (as used also on [Fedora](https://src.fedoraproject.org) dist-git server) can also be used on git.centos.org. You still need to push tarballs to lookaside but you can use the flat layout when importing from Fedora/Epel, and it's purely an opt-in choice, and so not the default one that SIGs are using for years now +Now that we have our sources pushed to both git.centos.org and lookaside cache, +we can now proceed with a [build in cbs/koji](cbs.md).