Blame docs/buildsys/mirror-network.md

a8c36c
The whole mirror network is a mix of sponsored/donated machines to the CentOS Project (so machines we install/control/monitor in the centos.org namespace) and external mirrors. Depending on CentOS artifacts (normal packages, iso images, cloud images, etc) they can land on multiple and different mirror networks
388fbe
388fbe
## CentOS Stream 9 and above
388fbe
### Overview
388fbe
More or less same process as for previous releases, except that the mirror pool is `mirror.stream.centos.org` and has less machines, due to hard-disk space constraint. Indeed, starting from Stream 9, all artifacts, including debuginfo and source packages, are all pushed out together to same mirror network (so not to `vault.centos.org` nor `debuginfo.centos.org` - see below)
388fbe
388fbe
### Operations
388fbe
As we don't use our previous mirror crawler for this, nothing is really needed, except third-party mirrors registering their mirror on Fedora MirrorManager, and someone from infra team adding it to the CentOS Category, after having checked that mirror isn't located in a restricted/embargoed country
388fbe
388fbe
## Cloud images
388fbe
### Overview
388fbe
The CentOS Project build some artifacts that can be directly consumed, like cloud images, in various formats : .raw, .qcow2 and vagrant boxes.
388fbe
Due to the initial size of these images, it was decided to create a specific `cloud.centos.org` small CDN (still based on donated/sponsored machines)
388fbe
The release process is the same as above though : artifacts are pushed to a specific mirror reference and machines in the cloud.centos.org pool are pulling content and expose it to public consumers.
388fbe
388fbe
Worth knowing though that in parallel some cloud images are also directly available on cloud infra, like our AMI images are automatically available on AWS and so don't need to be pulled from cloud.centos.org to be reimported back on AWS.
388fbe
388fbe
### Operations
388fbe
Nothing really to be done, except adding/removing from our PowerDNS setup nodes behind cloud.centos.org in case of migration or maintenance
388fbe
388fbe
## Vault network
388fbe
### Overview
388fbe
For some time, and due to disk space usage on donated machines, it was decided to push *src.rpm* packages to a different network, called `vault.centos.org` and so not overloading `mirror.centos.org` (and so not overloading third-party mirrors network too)
388fbe
As mentioned in the CentOS Stream 9 section, it was only the case up to release Stream 8, as starting from Stream 9, mirror.stream.centos.org (and mirrors getting content from it) would contain *all* packages, including src.rpm and debuginfo packages
388fbe
`vault.centos.org` is also still containing archived and EOL'ed CentOS versions
388fbe
388fbe
It's now a dedicated Cloudfront distribution on AWS, meaning that everybody should be getting content directly from AWS, and so from caching servers in AWS infra (edge locations).
388fbe
That Cloudfront setup is configured to use some specific http servers are Origin (in a failover group).
388fbe
The release process is the same as above though : artifacts are pushed to a specific mirror reference and machines in the Vault Origin pool are pulling content and expose it to cloudfront (while these servers are reachable, they'd prevent getting content directly, and only cloudfront is authorized, see the `mirror-vault` role to see how)
388fbe
388fbe
### Operations
388fbe
 * tuning WAF rules on AWS Cloudfront to eventuall reject DDoS and abusers and limit traffic
388fbe
 * adding/removing nodes used as Origin (all monitored already)
388fbe
388fbe
388fbe
## Debuginfo network
388fbe
### Overview
388fbe
For some time, and due to disk space usage on donated machines, it was decided to push *debuginfo* packages to a different network, called `debuginfo.centos.org` and so not overloading `mirror.centos.org` (and so not overloading third-party mirrors network too)
388fbe
The release process is the same as above though : artifacts are pushed to a specific mirror reference and machines in the debuginfo.centos.org pool are pulling content and expose it to public consumers.
388fbe
As mentioned in the CentOS Stream 9 section, it was only the case up to release Stream 8, as starting from Stream 9, mirror.stream.centos.org (and mirrors getting content from it) would contain *all* packages, including src.rpm and debuginfo packages
388fbe
388fbe
### Operations
388fbe
Nothing really to be done, except adding/removing from our PowerDNS setup nodes behind debuginfo.centos.org in case of migration or maintenance
388fbe
388fbe
## Buildlogs network
388fbe
### Overview
388fbe
For the release of CentOS 7, it was decided to expose publicly the status/progress, but also pushing out all built rpm packages directly, including the specific mock config file that was used, the build log files (reason why it's named `buildlogs` so that people could see what was already built, before a final CentOS 7 tree would be able to see the light and so being tested. As it was there, it was decided to still push all unsigned pkgs to that network, but also use it to push SIGs content that would be also push there to be tested by community, and so tagged as `testing`.
388fbe
388fbe
It's worth knowing that in the `mirror-buildlogs` role, you'll see some RewriteRules in httpd to redirect to a specific CDN : CDN77 was interested in sponsoring the CentOS Project back then, so all pkgs download should be redirected to them, and they use (like Cloudfront for AWS and vault.centos.org) a dedicated "origin" node to themselves get content in their caching network
388fbe
388fbe
The release process is the usual : artifacts are pushed to a specific mirror reference and machines in the buildlogs.centos.org pool are pulling content and expose it to public consumers and CDN77.
388fbe
388fbe
### Operations
388fbe
Apart from monitoring the infra, the only thing to do is adding/removing record from our PowerDNS setup (GeoIP) *and* verify that the origin node used by CDN77 is still reachable and working (if not, just update in public DNS the CNAME for the <fqdn> declared as origin (see ansible inventory and role for which record is used for that)
388fbe