# Lookaside/git

This page documents how SIGs can upload the sources of their RPM into the lookaside
cache and how they should set-up their corresponding git repository.

It should be noted that two model exists, the "traditional" model and the one
based on CentOS Stream and Fedora. While both model can be intertwined  in some
places, it is recommended that SIGs pick a model and follow it. This documentation
will not cover which aspect of which model is compatible with the other one.

## New Package (from source)

If you SIG uses [https://git.centos.org](https://git.centos.org) and the package
you want to build is not there yet, you will have to create a ticket first at:
[https://pagure.io/centos-infra/issues](https://pagure.io/centos-infra/issues)
to request that the git repo for `/rpms/<your_package>` be created.

If your SIG uses [https://gitlab.com/CentOS](https://gitlab.com/CentOS), you,
your SIG chair or people that have been granted access by the SIG chair, will be
able to create the project for your package directly.

In the rest of the document we will use the following example : Build the package `centpkg-minimal`,
as a member of the `sig-core` SIG group. We have to push two things:

 * the .spec file to build the rpm, to the git repository
 * eventually an archive (.tar.gz, .tar, .xz...) of the sources, to lookaside cache

!!! warning
    To push to lookaside cache you need to have already a local check out from [https://git.centos.org/centos-git-common](https://git.centos.org/centos-git-common)

## Pushing first to lookaside cache

Let's assume that my pkg centpkg-minimal that I want to build has an archive called centpkg.minimal.tar.gz. To be able to push to lookaside cache, we need the following :

 * a valid TLS cert (also needed to build on cbs.centos.org) obtained through [centos-cert](../auth/) util
 * either script `lookaside_upload` or `lookaside_upload_sig` in your $PATH (coming from centos-git-common git repo, see above)

There are two path available for the lookaside cache, each is available using a different script:

- `lookaside_upload` allows uploading to the traditional CentOS Linux structure: `baseurl/pkgname/branch/hash` ([example](
https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05eb775f))
- `lookaside_upload_sig` allows uploading to the same lookaside structure as the one used by CentOS Stream and Fedora: `baseurl/pkgname/tarball/hashtype/hash/tarball` ([example](
https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.tar.xz/sha512/f7aeac0fe5bf594933cd35b7ecc94ea8ddcbfedc04fa769c4da298e7bf105df116375d44711d944c748c85f61f96f6149be34c76eb37f28aa1f16359a9122abf/linux-5.14.0-62.el9.tar.xz))

Both scripts need some paramters, simply use `-h` to see them.

Back to our example.

### Pushing to the "traditional" lookaside structure

The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz and I'm member of the sig-core group, and want to build it for c7, so we'll call it like this :

```
lookaside_upload -f centpkg-minimal.tar.gz -n centpkg-minimal -b c7-sig-core
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz
[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal
[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4
[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

################################################################################################################ 100.0%
File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK
[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded ....
[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4

```

### Pushing to the CentOS Stream/Fedora lookaside structure

The pkg name is centpkg-minimal, file is centpkg-minimal.tar.gz, so we'll call it like this :

```
lookaside_upload_sig -f centpkg-minimal.tar.gz -n centpkg-minimal
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : centpkg-minimal.tar.gz
[+] CentOS Lookaside upload tool -> Package name: centpkg-minimal
[+] CentOS Lookaside upload tool -> sha1sum: d6616b89617914a0dd0fd5cfa06b0afc7a4541c4
[+] CentOS Lookaside upload tool -> Remote branch: c7-sig-core
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======

################################################################################################################ 100.0%
File centpkg-minimal.tar.gz size 15178 CHECKSUM d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 stored OK
[+] CentOS Lookaside upload tool -> Validating that source was correctly uploaded ....
[+] CentOS Lookaside upload tool -> [SUCCESS] Source should be available at https://git.centos.org/sources/centpkg-minimal/c7-sig-core/d6616b89617914a0dd0fd5cfa06b0afc7a4541c4

```

Now that we have uploaded to lookaside cache, we can reference it in our /rpms/centpkg-minimal git repository on git.centos.org, see below


## Pushing to git

SIGs have two options when it comes to the git repositories which can be used as part of dist-git:

- [git.centos.org](https://git.centos.org)
- [gitlab.com/CentOS](https://gitlab.com/CentOS)

If your SIG wants to use gitlab, they will have to follow the [instructions](gitlab.md) to request one.
Here we'll cover both workflow, assuming the git repository exists on either platform.

### Git clone

The first thing you'll need is a local checkout of the git repository for the pkg we'd like to work on.
In our case, the git repo url is `https://git.centos.org/rpms/centpkg-minimal` or
`https://gitlab.com/CentOS/<sig>/rpms/<repo>.git`

So the way to git clone/pull over ssh is either `ssh://git@git.centos.org/rpms/centpkg-minimal.git` or
`git@gitlab.com:CentOS/<sign>/rpms/<repo>.git`:

```
git clone ssh://git@git.centos.org/rpms/centpkg-minimal.git
Cloning into 'centpkg-minimal'...
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 15 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (15/15), done.
Resolving deltas: 100% (2/2), done.
```

!!! important
    On git.centos.org you can clone over https, but then it wouldn't let you push back to it, as it needs ssh with key verification and based on group membership for acls

### Git branches

Repositories git.centos.org are used by both Red Hat and CentOS SIG, the structure of the git
repository is used to control the access. SIG members are therefore only allowed to push to branches
following the pattern: `c<version>-sig-<sig_name>*`

That means that if I'm member of the sig-core group (in ACO), I'll be able to
commit/push to `c7-sig-core`, `c8-sig-core` or `c9s-sig-core-whatever-I-want` branches

For our example, on git.centos.org we'll have to create a new c7-sig-core branch
(with on the assumption that we want to build for c7) :

```
git checkout -b c7-sig-core
```

Since SIGs have full control over their namespace on gitlab.com, that branching structure
is not mandatory. It will, however, be necessary if you plan on using the "traditional" lookaside
structure describe above as this relies on branch names.


### Git repository layout

There are two possible structure possible for the repository layout:

- The exploded SRPM layout
- The flat dist-git layout

#### The exploded SRPM layout

This is the "traditional" layout used in CentOS Linux, it is articulated around two folders:
```
├── SOURCES
└── SPECS
```

The `SPECS` folder is meant to receive your spec file and all other text files and
patches would go under `SOURCES`:

```
├── SOURCES
│   ├── <optional_file>
│   └── <optional_patch>
└── SPECS
    └── <pkg_name>.spec
```

Once done, we still need to link these files with the sources uploaded earlier in the
lookaside cache. See the next section.

#### The flat dist-git layout

This is the layout used in the dist-git repositories in CentOS Stream and Fedora.
All files (spec files, text files, patches...) are stored at the top level of
the repository:

```
├── <optional_file>
├── <optional_patch>
└── <pkg_name>.spec
```

### Linking to sources in the lookaside cache

Here as well, there are two ways you can link a dist-git repository to sources
uploaded to the lookaside cache:

- the "traditional" way using a `.<pkg_name>.metadata` file
- the CentOS Stream/Fedora way using a `sources` file

!!! important
    Even if your package doesn't contain any source pushed to lookaside cache
    (like for a package just having some small files in SOURCES/ dir, you *need* to
    have either a `.<pkg_name>.metadata` or `sources` file present and pushed in
    git repository

#### Using a `.<pkg_name>.metadata` file

To use this method, simply create a `.<pkg_name>.metadata` file with the following
content:
```
<sha1 of the tarball> SOURCES/<filename>
```

For example:
```
d6616b89617914a0dd0fd5cfa06b0afc7a4541c4 SOURCES/centpkg-minimal.tar.gz
```

#### Using a `sources` file

To use this method, simply create a `sources` file. The easiest way to achieve
this is:
```
sha512sum --tag <tarball> > sources
```

Example output:
```
SHA512 (centpkg-minimal-1.1.0.tar.gz) = f7cfbc956199b1a0342f321a0e4cb055d6ac2c784a8faace43cf80772f0de34b58ede1a02a558cd03e25c14938ac6e17b3746b1919c0bbc1f5b2955472577e4f
```

Now that we have pointer to lookaside cache, and also .spec, we can push back to git and
we should be able to proceed with the "build-from-git" on cbs.centos.org. Let's commit first.

### Git push

Depending on if you picked the "traditional" structure or the CentOS Stream/Fedora
one, this is how your repository should look like:

* Traditional layout

```
.
├── .<pkg_name>.metadata
├── SOURCES
│   ├── <optional_file>
│   └── <optional_patch>
└── SPECS
    └── <pkg_name>.spec
```

* CentOS Stream/Fedora layout

```
.
├── sources
├── <optional_file>
├── <optional_patch>
└── <pkg_name>.spec
```

We can now push to to git, as usual:

```
# git add <files> # if needed
git commit -a
git push origin <branch> # to create the remote branch if not existing yet
```

Where `<branch>` would be `c7-sig-core` for our example using `git.centos.org`.


Now that we have our sources pushed to both git.centos.org and lookaside cache,
we can now proceed with a [build in cbs/koji](cbs.md).