Ansible roles are using the following logic to distribute .key/.cert files
* {{ public_name }}.key : TLS private key
* {{ public_name }}.crt : signed TLS certificate
* {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)
We have some long-term certificates that we use for www/lists/ci/etc .centos.org Those need to be required through DigiCert Web ui and through internal RH ticket. Once we have the signed cert back, we can upload it in pkistore and deploy
Warning
This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.
We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup).
Actually that node is certbot.rdu2.centos.org.
We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh We just have once to add in our zone (main one, so centos.org) a CNAME pointing to the delegated zone acme.centos.org Example (simple) for forums.centos.org :
_acme-challenge.forums IN CNAME _acme-challenge.forums.acme.centos.org.
Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record
acme.sh --issue --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org
Let's see what this produces on our DNS node, basically updating TXT record:
Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"
And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:
[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org' [Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain [Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org' [Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ" [Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success. [Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first. [Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success. [Tue 10 Dec 10:36:19 UTC 2019] All success, let's return [Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org [Tue 10 Dec 10:36:22 UTC 2019] Success [Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records. [Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt [Tue 10 Dec 10:36:22 UTC 2019] Removed: Success [Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign. [Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579 [Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd [Tue 10 Dec 10:36:24 UTC 2019] Cert success.
PS : for a wildcard, just add multiple -d and '*.domain'
acme.sh --issue --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org
All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir
Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)
You can add multiple SANs in the same certs. Here is one example with mon.centos.org and SAN mon.j7.centos.org, status.centos.org :
certbot certonly --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org -d mon.centos.org -d mon.j7.centos.org -d status.centos.org
All files (certs/keys) are then available under /etc/letsencrypt/live/ (you'll have to import those into this pkistore dir)
For each cert/dns record, we have to ask for a renewal
acme.sh --renew-all
For http challenge it's better to run first with --dry-run, then fix eventual issue and then launch it again for real operations
time certbot renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org --dry-run ; echo return code $? certbot renew --force-renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org
Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first. Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository :
./validate_public_chain koji.mbox.centos.org.crt Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle koji.mbox.centos.org.crt: OK
Once it's committed/pushed to pkistore git repo, tobisna (ansible bot) will deploy the renewed TLS certs automatically