diff --git a/docs/buildsys/mirror-network.md b/docs/buildsys/mirror-network.md index 0c62615..ac51cb0 100644 --- a/docs/buildsys/mirror-network.md +++ b/docs/buildsys/mirror-network.md @@ -1,27 +1,4 @@ -The whole mirror network is a mix of sponsored/donated machines to the CentOS Project (so machines we install/control/monitor in the centos.org namespace) and external mirrors. Depending on CentOS Project (Linux vs Stream) or artifacts (normal packages, iso images, cloud images, etc) it can land on multiple and different mirror networks - -## CentOS Linux 7 / Stream 8 -### Overview -This is the largest and oldest mirror network we have and serving CentOS community for multiple years now. -Workflow goes like this : - - * CentOS releases/updates packages are pushed on main mirrors (mirror-ref) - * Some other centos owned/controlled machines are getting content (machines in the mirror.centos.org pool) - * external third-party mirrors, when declared in a mirrors databases and with IP used in ACL, are authorized to get content through rsync - * our "legacy" mirrors crawler validates in loop these mirrors and produce up2date mirrorlists that are pushed to mirrorlists.centos.org node (see other link in this doc to explain how it works just for mirrorlists) - -### Operations -Third-party mirrors are supposed (from now on) to create ticket on the centos-infra tracker to ask for a new mirror to be registered, or existing one to be modified. -The current status can be seen on [https://mirror-status.centos.org](https://mirror-status.centos.org). -Worth reading the [mirrorlist doc](https://github.com/centos/mirrorlists-code) about the backend scripts, used to validate mirrors and so how to modify existing entries in DB. - -To add a new mirror, the `/var/lib/centos-mirrors/mirror-geo-check.py` helper script can be used to : - - * automatically detect continent/country (and state if in US) and verify restricted access - * validate mirror is (currently) reachable and check content - * prepare the needed sql query/statement to run on mysql - -PS : all modifications to existing mirrors need to be done (for now, no helper script) "by hands" +The whole mirror network is a mix of sponsored/donated machines to the CentOS Project (so machines we install/control/monitor in the centos.org namespace) and external mirrors. Depending on CentOS artifacts (normal packages, iso images, cloud images, etc) they can land on multiple and different mirror networks ## CentOS Stream 9 and above ### Overview diff --git a/docs/infra/mirrorlists.md b/docs/infra/mirrorlists.md deleted file mode 100644 index be7da9c..0000000 --- a/docs/infra/mirrorlists.md +++ /dev/null @@ -1,48 +0,0 @@ -# CentOS mirrorlist service - -!!! note - the mirrorlist.centos.org is *crucial* for all deployed CentOs instances all around the world as each deployed CentOS instance will query the mirrorlist webservice to get a list of validated and up2date mirrors to retrieve their content from. It's using GeoIP *or* checking if coming from a cloud provide (like EC2), in which case it would redirect to the nearest (GeoIP) or internal (Cloudfront setup for AWS/EC2) mirror - - -## Overview -![mirrorlists schema](../img/mirrorlists.png) - -It contains the following kind of scripts: - - * backend : so scripts used by our "crawler" node, validating in loop all the external mirrors through IPv4 and IPv6 and so producing the 'mirrorlists', each one per repo/arch/country - * frontend : python scripts used for : - * http://mirrorlist.centos.org - * http://isoredirect.centos.org - -## Backend (crawler) -There are two Perl scripts for checking mirrors: - - * makemirrorlists-combined.pl for creating files for mirrorlist.centos.org - * makeisolists-combined.pl for creating files for isoredirect.centos.org. - -Both scripts can create lists for all CentOS supported released ,including SIG and AltArch content. makemirrorlists-combined.pl will test each mirror separately for IPv4 and IPv6. - -mirrorlist.centos.org will then be able to present only IPv6-capable mirrors to the clients when mirrorlist.centos.org is accessed over IPv6. -More details about the internals of these scripts can be found in backend/mirrorlist_crawler_deployment_notes.txt - -## Frontend -All scripts are located in the frontend folder. -The following items are needed for the mirrorlist/isoredirect service: - - * A http server (apache) using mod_proxy_balancer (see frontend/httpd/mirrorlist.conf vhost example) - * python-bottle to run the {ml,isoredirect}.py code for various instances - * Maxmind Geolite2 database : [City version](https://dev.maxmind.com/geoip/geoip2/geolite2/) - * python-geoip2 pkg (to consume those Geolite2 DB) - * python-memcached (to cache results for GeoIP/Cloud providers) - * For each worker, a specific instance/port can be initialized and added to Apache config for the proxy-balancer (see frontend/systemd/centos-ml-worker@.service) - -Those services (mirrorlist/isoredirect) just consume mirrorlist files, pushed to those nodes, and updated in loop by the Crawler process (see Backend section above) - -When a request is made to the service, the python script : - - * checks for IPv4 or IPv6 connectivity - * checks if IP is in memcached (for country/cloud provider) - * searches if IP is from cloud provider - * computes Geolocation based on the origin IP - * searches for validated mirrors in the same country/state for the request arch/repo/release - * returns such list diff --git a/docs/operations/decommission.md b/docs/operations/decommission.md index c09dadc..3a8dfe5 100644 --- a/docs/operations/decommission.md +++ b/docs/operations/decommission.md @@ -11,7 +11,7 @@ This is an overview of the needed tasks to perform when we want to remove a node * Remove it from Zabbix monitoring * (optional and if sponsored : remove sponsor logo from [website git repo](https://git.centos.org/centos/centos.org) ) -Worth knowing that if node is used in one of the [mirror,mirror.stream,vault,debuginfo] roles, it needs to be also removed from the sqlite DB used to generate the dynamic powerdns setup. +Worth knowing that if node is used in one of the [mirror,mirror.stream,vault,debuginfo] roles, it needs to be also removed from the sqlite DB used to generate the dynamic powerdns setup. The following snippet can be applied on the host that is defined in ansible inventory as `pdns_db_host` : ``` host="centost8.centos.org" diff --git a/mkdocs.yml b/mkdocs.yml index 69f60bd..2161589 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -88,7 +88,6 @@ nav: - infra/authentication.md - infra/dns.md - infra/mailservers.md - - infra/mirrorlists.md - infra/sponsors.md - infra/centos-ci.md - infra/openshift.md