From 12bb4504ac64f5fee55d6c231284693d455bb194 Mon Sep 17 00:00:00 2001 From: Fabian Arrotin Date: Jun 21 2021 17:55:40 +0000 Subject: Initial commit and WIP for mkdocs skeleton Signed-off-by: Fabian Arrotin --- diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..bd816b7 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +site +site/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..b2b769a --- /dev/null +++ b/README.md @@ -0,0 +1,31 @@ +# CentOS Infra documentation + +This git repository contains the needed sources (markdown format) used to generate the CentOS Infra documentation website. + +It uses [mkdocs](https://mkdocs.org) but with a specific [material](https://squidfunk.github.io/mkdocs-material) theme + +## How to test locally +The easiest way is just to use podman and a container: + +``` +podman pull docker.io/squidfunk/mkdocs-material:latest +``` + +You can then enter the directory where you have cloned this repository (where mkdocs.yml is) and then you can run the following development site (that will automatically refresh on each new file/change) : + +``` +podman run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material +``` + +You can now open your browser to http://localhost:8000 and you'll be able to see `live` your edit/changes + +## How to render/build the site as static pages + +Still using same podman container, but instead call it like this : + +``` +podman run --rm -it -v ${PWD}:/docs squidfunk/mkdocs-material build + +``` + +Worth knowing that the goal is to just develop locally , and then git commit/push as the built site will automatically be rendered in the next minutes on public website diff --git a/docs/about.md b/docs/about.md new file mode 100644 index 0000000..672eeda --- /dev/null +++ b/docs/about.md @@ -0,0 +1,3 @@ +# This is my first title + +And this is my first content, yeah diff --git a/docs/ansible/ara.md b/docs/ansible/ara.md new file mode 100644 index 0000000..3f0697d --- /dev/null +++ b/docs/ansible/ara.md @@ -0,0 +1 @@ +# this is how we use ARA diff --git a/docs/ansible/index.md b/docs/ansible/index.md new file mode 100644 index 0000000..be03979 --- /dev/null +++ b/docs/ansible/index.md @@ -0,0 +1,74 @@ +# Ansible usage in CentOS infra + +## Contributing to Ansible infra (playbooks or roles) +When you want to contribute to playbooks or roles, you should always open a merge request (PR) against `staging` branch and not `master` branch. +One reviewer from the correct org will then get notification and will discuss/review your PR and eventually guide you. +Ideally just look at the common way roles are organised, to reuse other roles and convention. +Always have default variables for *Everything*, with safe default values (of course never the ones deployed for staging/prod) +When proposing a change in the behaviour, always make that change a opt-in, that defaults to "no" (safest) so that only that change would be applied on other nodes *if* variable used to include that task would be turned on. Of course we can have on real needs a default to `True` if we know that such change would need to be replicated by default on all nodes controlled by Ansible and using that role. + +## Naming convention +### Roles (regular playbooks used at regular interval) +The playbooks that will be played for roles will start with `role-` +A all-in-one roles-all.yml will just include all the role-.yml when we want to just ensure the whole infra is configured the way it should. +Each playbook for a role target a group called `hostgroup-role-`. + +There a small exceptions where some role- playbooks will be small variants of a role, so also with other tasks to call specific tasks for an existing role (so when for example a vhost for httpd is a variant of the httpd role) + +#### "pre-flight" check +For each playbook configuring a role, there is an option (in case of) to end the play if we have to. +Basically touching /etc/no-ansible on a managed node would ensure that the playbook is ended. That permits to have (in emergency for example) someone having a look at a node and ensuring that ansible doesn't modify the node at the same time. After each role configuration, a file is also created (monitored by Zabbix) to ensure that nodes are always configured as they have to + + +### Deploy (on demand/triggered) +Deploy playbooks (can combine also other playbooks) can be named `deploy-` + + +### Ad-Hoc tasks (on demand/triggered) +Simple ad-hoc playbooks can just be named/start with `adhoc-`. +Those specific playbooks can need some tasks/vars/handlers, so for those special ones (as each role has it own set) we'll include those in the same repository, but it's up to the process deploying those for the ansible-host role to setup correctly the needed symlinks for the normal hierarchy. + +## Complete needed structure (needed on ansible mgmt node) +The "on-disk" ansible directory should then look like this : + +``` +. +├── ansible.cfg +├── files -> playbooks/files +├── handlers -> playbooks/handlers +├── filestore +├── inventory +├── pkistore +├── playbooks +│   ├── files +│   ├── handlers +│   ├── requirements.yml +│   └── vars +│   └── templates +├── roles +│   ├── +└── templates -> playbooks/templates +└── vars -> playbooks/vars + +``` + +## Ansible roles setup +All roles will be deployed for a list of individual git repositories, each one being its own role. +A requirements.yml file will be used to declare which roles (and from where to get them) and so downloaded on the ansible host through ansible-galaxy + +## Inventory and encrypted files +Inventory is itself a different git repository, git-crypted and that will be checked-out on the ansible host +Same for the two following git (crypted) repositories: + + * pkistore (holding some PKI key/certs) + * filestore (holding some other files/secrets that aren't templates but that should be crypted/non public, so not in roles either) + +!!! note + This paragraph is a note and shouldn't be read + Seriously, try something else + +!!! warning + This is a warning + And you should be careful about it + +This is the **end** diff --git a/docs/apps/bugs.md b/docs/apps/bugs.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/apps/bugs.md diff --git a/docs/apps/forums.md b/docs/apps/forums.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/apps/forums.md diff --git a/docs/apps/mailman.md b/docs/apps/mailman.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/apps/mailman.md diff --git a/docs/apps/wiki.md b/docs/apps/wiki.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/apps/wiki.md diff --git a/docs/auth/kerberos.md b/docs/auth/kerberos.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/auth/kerberos.md diff --git a/docs/auth/openidc.md b/docs/auth/openidc.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/auth/openidc.md diff --git a/docs/auth/shell.md b/docs/auth/shell.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/auth/shell.md diff --git a/docs/auth/tls.md b/docs/auth/tls.md new file mode 100644 index 0000000..89454f8 --- /dev/null +++ b/docs/auth/tls.md @@ -0,0 +1 @@ +# TLS certificate auth diff --git a/docs/buildsys/koji-cbs-sigs.md b/docs/buildsys/koji-cbs-sigs.md new file mode 100644 index 0000000..dab017c --- /dev/null +++ b/docs/buildsys/koji-cbs-sigs.md @@ -0,0 +1,96 @@ +# Special Interest Groups + +CentOS clearly defines what a SIG (Special Interest Group) is : a group of people sharing interest into a specific task/tech area, and willing to build/test/deliver artifacts (like RPM packages) that CentOS users can then consume. + +Each SIG *must* be approved by the CentOS board, and no action should be taken before that. Once approved and CentOS board meeting minutes published, we can (Infra and Releng) proceed with next steps : + +## SIG onboarding +We first need to create the SIG group in IPA (if it doesn't exist yet) and it shouldn't of course conflict with any existing. +Requirements: + + * a user with `admin` right in IPA to be able to create the group + * a shell account on an enrolled machine (like `ipa-shell.iad2.centos.org` but can be any enrolled node within the @FEDORAPROJECT.ORG REALM (verify that you have a `valid` krb5 ticket or next steps will fail) + * a SIG name and description + * at least one SIG member that will be the chair (meaning that he'll be himself then able to add/remove other SIG users in that group + +We can use either the IPA UI, or for automation purposes,just use the ipa cli. +Basicall three commands: + + * `ipa group-add` : to create the group + * `ipa group-add-member` : to add the first user + * `ipa group-add-member-manager` : to grant "sponsor" right to sig chair, himself then able to add users + +Here is a full output/example about creating the `sig-kmods` with one chair (valid IPA/FAS account) : + +``` +# Defininig first some variables +sig_name="sig-kmods" +sig_description="The CentOS kmods SIG" +sig_chair="pjgeorg" + +# Now creating group and granting rights +ipa group-add --desc="${sig_description}" --fasgroup ${sig_name} +----------------------- +Added group "sig-kmods" +----------------------- + Group name: sig-kmods + Description: The CentOS kmods SIG + GID: 1494370508 + FAS group: True + +ipa group-add-member ${sig_name} --users=${sig_chair} + Group name: sig-kmods + Description: The CentOS kmods SIG + GID: 1494370508 + Member users: pjgeorg + Membership managed by users: pjgeorg +------------------------- +Number of members added 1 +------------------------- + +ipa group-add-member-manager ${sig_name} --users=${sig_chair} + Group name: sig-kmods + Description: The CentOS kmods SIG + GID: 1494370508 + Membership managed by users: pjgeorg +------------------------- +Number of members added 1 +------------------------- + +``` +If you need to add later more "sponsor" user, just repeat the `ipa group-add-member-manager` operation with new user. + + +At this stage, the group should appear throug portal on https://accounts.centos.org + +Worth knowing also that in the next minutes, an automatic email alias is created for the group following the naming convention `sig-@centosproject.org` and that will automatically include all sig members email addresses, once added (and removed when removed) + +Same for SIG members : an email alias following the `@centosproject.org` alias is also created for all SIG members (and pointing to the email address they used when they registered their ACO/FAS account) +More informations about this in the [Ansible postfix role](https://github.com/CentOS/ansible-role-postfix/blob/master/defaults/main.yml#L71) + +While not fully needed, it's now a good idea to send mail to that new SIG to welcome them. +Something like this : + +``` +To: @centosproject.org +Subject: Welcome to the new SIG ! + +Hi, + +This welcome email means that your new group was created and granted some management rights to at least the SIG Chair. +The group should appear now on https://accounts.centos.org/group// + +Also worth knowing that each SIG member is also automatically granted an email alias following this convention : @centosproject.org + +If you have any needs, requests, feel free to contact the CentOS Infra and Releng team through https://pagure.io/centos-infra/issues. + +Kind Regards, +``` + +## Security GPG key for packages signing + +Each SIG building RPM packages through https://cbs.centos.org community build service will need a GPG security key that will be then automatically be used by the signing service + + + +## Koji tags creation diff --git a/docs/buildsys/koji-cbs.md b/docs/buildsys/koji-cbs.md new file mode 100644 index 0000000..1648f26 --- /dev/null +++ b/docs/buildsys/koji-cbs.md @@ -0,0 +1 @@ +# Community Build Service (https://cbs.centos.org) diff --git a/docs/buildsys/koji-mbox.md b/docs/buildsys/koji-mbox.md new file mode 100644 index 0000000..f187853 --- /dev/null +++ b/docs/buildsys/koji-mbox.md @@ -0,0 +1 @@ +# Koji setup for CentOs 8/8-stream distro diff --git a/docs/buildsys/koji-stream.md b/docs/buildsys/koji-stream.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/buildsys/koji-stream.md diff --git a/docs/buildsys/reimzul.md b/docs/buildsys/reimzul.md new file mode 100644 index 0000000..3081aa8 --- /dev/null +++ b/docs/buildsys/reimzul.md @@ -0,0 +1,150 @@ +# Reimzul CentOS 7 distro build system + +!!! note + Reimzul is a distributed build system that is used to build CentOS Linux up to (and including CentOS 7). + For CentOs 8/8-stream/9-stream and beyond, see other koji system + +## Reimzul overview +![reimzul](../img/reimzul.png) + +Reimzul consists only of some python scripts, that are used as wrappers around mock. + +The following components are used to orchestrate the builds and notifications: + + * [beanstalkd](http://kr.github.io/beanstalkd/) : used as build queues, per arches + * [stunnel](http://www.stunnel.org/) : all components are using TLS connections , so including workers connecting to beanstalkd, to "watch" the queues and pick the build jobs landing in those queues, as well as for http (to retrieve the src.rpm to build) and then also for rsync (transferring artifacts - logs,rpm - to bstore node) + * MQTT : python-paho-mqtt : sending results to a mqtt broker, with other subscribers used for notifications (including IRC) + * [MongoDB](http://www.mongodb.org) : just use as document store to load build results and so be able to track/query build history + +## Controller +This is the node that runs beanstalkd for the various tubes/queues +To submit a pkg, we have two options: + + * build a .src.rpm, uploaded under /srv/reimzul/incoming and call (local) reimzul_submit.py script (see above) + * remotely trigger a build from git (including branch/tag/commit id) through reimzul_client (see above) + +Example when willing to submit same src.rpm (already uploaded !) to multiple arches : +``` +for arch in x86_64 aarch64 armhfp ; do /srv/reimzul/code/reimzul_submit.py -s time-1.7-45.el7.src.rpm -d .el7 -a $arch -t c7.1708.u ; done +``` +Important to know that there is no need to submit for i386 : it will be done automatically in parallel on the builder doing the x86_64 build, with the same timestamp + +The controller has also a dispatcher worker (msg_dispatcher.py) that watches the notify tube, and will : + + * send mails to specific rcpts for each failed/successful build + * log to /var/log/reimzul/reimzul.log + * add each job to local mongodb instance + * also send json payload over mqtt (so then various subscribers can reuse those payloads, including for example for irc notifications) + +To control those notifications, reimzul uses a config file /etc/reimzul/reimzul.ini (see reimzul.ini.sample for reference) + +### Processes : + + * reimzul-notifier.py (sending notifications for build results, log, mqtt, mails) + * reimzul-mqtt-pub.py (process subscribed to some topic to allow remote builds, and that process will retrieve automatically from git and submit) + +One reimzul notifier worker is enough to send notifications +``` +cp systemd/reimzul-notifier.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable reimzul-notifier --now + +``` + +One reimzul mqtt subscriber worker is enough to wait for incoming build requests +``` +cp systemd/reimzul-mqtt-sub.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable reimzul-mqtt-sub --now + +``` + + + +## Builders (workers) : + +These nodes are the ones that : + + * watch for jobs in $arch (exception being x86_64, watching also i386 tube) + * download the src.rpm from controller (stunnel) + * submit it to mock + * upload results to bstore node (central http repo holding all the repositories) + * sending results back in notify tube + +### Processes : + + * reimzul_worker.py + +You can launch as many workers/builder threads you want : there is a .service systemd unit file (see systemd/reimzul-worker@.service) that you can then launch multiple times. For example, let's assume that we want 4 parallel workers : +``` +cp systemd/reimzul-worker@.service /etc/systemd/system/ +systemctl daemon-reload +for i in {1..4} ; do systemctl enable reimzul-worker@${i} --now; done + +``` + +## Bstore : + +Central storage node that will accept all build artifacts under specific target repos. +Worth noting that all communication, including rsyncd, happen over tls (through stunnel) +It has also a worker itself, just watching the "createrepo" channel. +When a build finishes and has a successful build, one job is added to the createrepo tube (you can have multiple parallel workers for this too) and repo metadata is launched through createrepo_c, with multiple workers per process and cache directory + +### Processes + + * tosign_worker.py (will collect built pkgs in a staging area, waiting for pkgs to be then signed) + * repogen_worker.py (will regenerate repodata on each successful build) + +You can launch as many repogen workers threads as you want : there is a .service systemd unit file (see systemd/reimzul-repoogen-worker@.service) that you can then launch multiple times. For example, let's assume that we want 3 parallel workers : +``` +cp systemd/reimzul-repogen-worker@.service /etc/systemd/system/ +systemctl daemon-reload +for i in {1..3} ; do systemctl enable reimzul-repogen-worker@${i} --now; done + +``` + +One signer worker is enough (it will just copy the rpm files in a staging area, waiting to be collected for signing) +``` +cp systemd/reimzul-signer.service /etc/systemd/system/ +systemctl daemon-reload +systemctl enable reimzul-signer --now + +``` + +## Clients + +### Local client (on the controller, so need priv and local access) + +Once a .src.rpm is built and available under /srv/reimzul/incoming, you can submit builds to Reimzul is to launch reimzul_submit.py : +``` +usage: reimzul_submit.py [-h] -s SRPM -a ARCH -t TARGET -d DISTTAG [--now] + [--scratch] + +Reimzul CentOS distributed build client + +optional arguments: + -h, --help show this help message and exit + -s SRPM, --srpm SRPM The src.rpm pkg already uploaded in controller node + -a ARCH, --arch ARCH Defines the mock architecture to build against + [example: x86_64,armhfp,aarch64,i386,ppc64le,ppc64] + -t TARGET, --target TARGET + The target repo to build against/for, without any arch + specified [example: c7.1708.u] + -d DISTTAG, --disttag DISTTAG + Defines the mock disttag to use [example: .el7_4] + --now Will prioritize this job in front of the build queue + --scratch Will just build the pkg but not prepare it in staging- + tosign area +``` + +## Upstream git repository + + * [reimzul](https://github.com/CentOS/reimzul) + + +## Ansible automation roles + + * [ansible-role-reimzul-controller](https://github.com/CentOS/ansible-role-reimzul-controller) + * [ansible-role-reimzul-builder](https://github.com/CentOS/ansible-role-reimzul-builder) + * [ansible-role-reimzul-bstore](https://github.com/CentOS/ansible-role-reimzul-bstore) + diff --git a/docs/img/centos.png b/docs/img/centos.png new file mode 100644 index 0000000..fa510d6 Binary files /dev/null and b/docs/img/centos.png differ diff --git a/docs/img/mirrorlists.png b/docs/img/mirrorlists.png new file mode 100644 index 0000000..6b03bcb Binary files /dev/null and b/docs/img/mirrorlists.png differ diff --git a/docs/img/reimzul.png b/docs/img/reimzul.png new file mode 100644 index 0000000..b89578e Binary files /dev/null and b/docs/img/reimzul.png differ diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..f747c25 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,7 @@ +# CentOS Infra and Releng documentation + +This site is dedicated to CentOS Infrastructure documentation and is considered *Work in progress* + +The goal is to consolidate various documentation we have either through ansible inventories, some other git private repositories and/or elsewhere. + + diff --git a/docs/infra/dns.md b/docs/infra/dns.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/infra/dns.md diff --git a/docs/infra/ec2.md b/docs/infra/ec2.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/infra/ec2.md diff --git a/docs/infra/mirrorlists.md b/docs/infra/mirrorlists.md new file mode 100644 index 0000000..be7da9c --- /dev/null +++ b/docs/infra/mirrorlists.md @@ -0,0 +1,48 @@ +# CentOS mirrorlist service + +!!! note + the mirrorlist.centos.org is *crucial* for all deployed CentOs instances all around the world as each deployed CentOS instance will query the mirrorlist webservice to get a list of validated and up2date mirrors to retrieve their content from. It's using GeoIP *or* checking if coming from a cloud provide (like EC2), in which case it would redirect to the nearest (GeoIP) or internal (Cloudfront setup for AWS/EC2) mirror + + +## Overview +![mirrorlists schema](../img/mirrorlists.png) + +It contains the following kind of scripts: + + * backend : so scripts used by our "crawler" node, validating in loop all the external mirrors through IPv4 and IPv6 and so producing the 'mirrorlists', each one per repo/arch/country + * frontend : python scripts used for : + * http://mirrorlist.centos.org + * http://isoredirect.centos.org + +## Backend (crawler) +There are two Perl scripts for checking mirrors: + + * makemirrorlists-combined.pl for creating files for mirrorlist.centos.org + * makeisolists-combined.pl for creating files for isoredirect.centos.org. + +Both scripts can create lists for all CentOS supported released ,including SIG and AltArch content. makemirrorlists-combined.pl will test each mirror separately for IPv4 and IPv6. + +mirrorlist.centos.org will then be able to present only IPv6-capable mirrors to the clients when mirrorlist.centos.org is accessed over IPv6. +More details about the internals of these scripts can be found in backend/mirrorlist_crawler_deployment_notes.txt + +## Frontend +All scripts are located in the frontend folder. +The following items are needed for the mirrorlist/isoredirect service: + + * A http server (apache) using mod_proxy_balancer (see frontend/httpd/mirrorlist.conf vhost example) + * python-bottle to run the {ml,isoredirect}.py code for various instances + * Maxmind Geolite2 database : [City version](https://dev.maxmind.com/geoip/geoip2/geolite2/) + * python-geoip2 pkg (to consume those Geolite2 DB) + * python-memcached (to cache results for GeoIP/Cloud providers) + * For each worker, a specific instance/port can be initialized and added to Apache config for the proxy-balancer (see frontend/systemd/centos-ml-worker@.service) + +Those services (mirrorlist/isoredirect) just consume mirrorlist files, pushed to those nodes, and updated in loop by the Crawler process (see Backend section above) + +When a request is made to the service, the python script : + + * checks for IPv4 or IPv6 connectivity + * checks if IP is in memcached (for country/cloud provider) + * searches if IP is from cloud provider + * computes Geolocation based on the origin IP + * searches for validated mirrors in the same country/state for the request arch/repo/release + * returns such list diff --git a/docs/infra/sponsors.md b/docs/infra/sponsors.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/infra/sponsors.md diff --git a/docs/operations/decommission.md b/docs/operations/decommission.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/operations/decommission.md diff --git a/docs/operations/deploy/bare-metal.md b/docs/operations/deploy/bare-metal.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/operations/deploy/bare-metal.md diff --git a/docs/operations/deploy/common.md b/docs/operations/deploy/common.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/operations/deploy/common.md diff --git a/docs/operations/deploy/virtual-machine.md b/docs/operations/deploy/virtual-machine.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/docs/operations/deploy/virtual-machine.md diff --git a/docs/security/tls.md b/docs/security/tls.md new file mode 100644 index 0000000..fd94a41 --- /dev/null +++ b/docs/security/tls.md @@ -0,0 +1,127 @@ +# TLS certificates in Infra + +## Common naming convention. +Ansible roles are using the following logic to distribute .key/.cert files + +``` + * {{ public_name }}.key : TLS private key + * {{ public_name }}.crt : signed TLS certificate + * {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few) + +``` + +## Public certificates + +### DigiCert +We have some long-term certificates that we use for www/lists/ci/etc .centos.org +Those need to be required through DigiCert Web ui and through internal RH ticket. +Once we have the signed cert back, we can upload it in pkistore and deploy + + +### Letsencrypt + +!!! warning + This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services. + + +We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup). +Actually that node is `certbot.rdu2.centos.org`. + +#### How to obtain new cert (DNS challenge is the preferred way) + +##### For dns challenge +We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh +We just have once to add in our zone (main one, so centos.org) a CNAME pointing to the delegated zone acme.centos.org +Example (simple) for forums.centos.org : + +``` +_acme-challenge.forums IN CNAME _acme-challenge.forums.acme.centos.org. +``` +Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record + +``` +acme.sh --issue --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org +``` + +Let's see what this produces on our DNS node, basically updating TXT record: + +``` +Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved +Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ" +``` + +And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers: + +``` +[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org' +[Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain +[Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org' +[Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org +[Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ" +[Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success. +[Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first. +[Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org +[Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success. +[Tue 10 Dec 10:36:19 UTC 2019] All success, let's return +[Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org +[Tue 10 Dec 10:36:22 UTC 2019] Success +[Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records. +[Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org +[Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt +[Tue 10 Dec 10:36:22 UTC 2019] Removed: Success +[Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign. +[Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579 +[Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd +[Tue 10 Dec 10:36:24 UTC 2019] Cert success. + +``` + +PS : for a wildcard, just add multiple -d and '*.domain' +``` +acme.sh --issue --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org +``` +All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir + +##### For http challenge +Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip) + +You can add multiple SANs in the same certs. Here is one example with mon.centos.org and SAN mon.j7.centos.org, status.centos.org : + +``` +certbot certonly --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org -d mon.centos.org -d mon.j7.centos.org -d status.centos.org + +``` +All files (certs/keys) are then available under /etc/letsencrypt/live/ (you'll have to import those into this pkistore dir) + + + +#### How to renew existing certs +##### For DNS challenges (existing records) +For each cert/dns record, we have to ask for a renewal +``` +acme.sh --renew-all +``` + +##### For HTTP challange +For http challenge it's better to run first with --dry-run, then fix eventual issue and then launch it again for real operations +``` +time certbot renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org --dry-run ; echo return code $? + +certbot renew --force-renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org + +``` + +### Deploying through ansible +Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first. +Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository : + +``` +./validate_public_chain koji.mbox.centos.org.crt +Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle +koji.mbox.centos.org.crt: OK + +``` +Once it's committed/pushed to pkistore git repo, tobisna (ansible bot) will deploy the renewed TLS certs automatically + + + diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..6c8bbf9 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,61 @@ +site_name: CentOS Infra docs +site_url: https://docs.infra.centos.org/ +nav: + - Overview: index.md + - Operations: + - Common infra: + - operations/deploy/common.md + - operations/deploy/bare-metal.md + - operations/deploy/virtual-machine.md + - operations/sponsored/ec2.md + - operations/sponsored/bare-metal.md + - operations/decommission.md + - CI infra: + - test.md + - Build and Delivery: + - Build Systems: + - buildsys/reimzul.md + - buildsys/koji-cbs.md + - buildsys/koji-cbs-sigs.md + - buildsys/koji-mbox.md + - buildsys/koji-stream.md + - Pushing to mirror/CDN: + - buildsys/mirror-network.md + - Public Applications: + - apps/wiki.md + - apps/bugs.md + - apps/forums.md + - apps/mailman.md + - Infra services: + - infra/authentication.md + - infra/dns.md + - infra/web.md + - infra/mailservers.md + - infra/mirrorlists.md + - infra/sponsors.md + - infra/centos-ci.md + - infra/monitoring.md + - Security: + - security/tls.md + - Ansible: + - ansible/index.md + - ansible/ara.md + - Tips and Tricks: + - tips/hardware.md + - tips/ipmi.md +theme: + name: material + logo: img/centos.png + features: + - navigation.tabs + - navigation.tabs.sticky + - navigation.sections + - navigation.top + palette: + primary: black + +markdown_extensions: + - toc: + baselevel: 4 + - admonition +