Ansible roles are using the following logic to distribute .key/.cert files
* {{ public_name }}.key : TLS private key
* {{ public_name }}.crt : signed TLS certificate
* {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)
While IPA enrolled nodes can directly request TLS certificates, for CentOS infra, due to the fact that almost all nodes can't be enrolled (no direct link with IPA infra), we have to delegate this on an enrolled node where we can then be granted "delegation" rights in IPA, so that a "fake" enrolled node (created for the real target server that will need a TLS cert) can be "managed by" the enrolled node.
From that enrolled node, we'll then be able to retrieve the TLS cert/keytab and also then export/import into pkistore , following the same naming convention as described above.
Depending on the Env (Prod vs STG), you can find which node[s] is/are enrolled (through applied ipa-client client role applied with also the ipa_client_tls_delegated_host boolean set to True (needed for the following script to be present in the ipa-client role).
Pre-requisites:
ipa-client role applied with correct script deployedNote
The following steps are just for new certificates. As once you'll have requested this on the enrolled node, the certmonger process will automatically watch and request/renew new ones, so they'll land on the enrolled node automatically, from which you can then retrieve TLS files (from /etc/pki/tls/certs) and update pkistore (see above). To help with that see <pkistore>/koji/retrieve_from_ipa script
Once we have shell access on such enrolled node, we can proceed like this :
/usr/libexec/centos/generate_ipa_tls_krb5 You need to call the script like this : /usr/libexec/centos/generate_ipa_tls_krb5 -arguments -n : node name / fqdn ([REQUIRED], example 'ppc64-01.cbs.centos.org') -d : Description for that host ([REQUIRED], example 'cbs koji builder') -s : service for principal ([REQUIRED], example 'compile' would create compile/ppc64-01.cbs.centos.org service in IPA) -h : display this help You also need a valid kerberos ticket otherwise script will exit
Here is an output example for the command with arguments:
/usr/libexec/centos/generate_ipa_tls_krb5 -n mbs.mbox.stg.centos.org -d 'Koji mbox STG mbs' -s 'HTTP' [+] Adding host in IPA and adding delegation to retrieve certs/keytab ... ------------------------------------ Added host "mbs.mbox.stg.centos.org" ------------------------------------ Host name: mbs.mbox.stg.centos.org Description: Koji mbox STG mbs Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Password: False Keytab: False Managed by: mbs.mbox.stg.centos.org Host name: mbs.mbox.stg.centos.org Description: Koji mbox STG mbs Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org ------------------------- Number of members added 1 ------------------------- ------------------------------------------------------------------ Added service "HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG" ------------------------------------------------------------------ Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Managed by: mbs.mbox.stg.centos.org Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org ------------------------- Number of members added 1 ------------------------- Host name: mbs.mbox.stg.centos.org Description: Koji mbox STG mbs Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org Hosts allowed to retrieve keytab: <modified>.fedoraproject.org ------------------------- Number of members added 1 ------------------------- Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org Hosts allowed to retrieve keytab: <modified>.fedoraproject.org ------------------------- Number of members added 1 ------------------------- [+] Retrieving TLS and keytab files ... New signing request "20210705130958" added. Keytab successfully retrieved and stored in: /etc/pki/centos/certs/HTTP-mbs.mbox.stg.centos.org.keytab [+] Validating TLS against IPA CA ... /etc/pki/centos/certs/mbs.mbox.stg.centos.org.crt: OK
Tip
You need a valid kerberos ticket for the existing REALM, otherwise same script will exit 1. Worth knowing that if you have enabled OTP on your account (a must for admins) you can use the 2fa-kinit script, also deployed by ansible on each enrolled node by the ipa-client ansible role
You can now import into pkistore (and correct directory based on role) git-crypted repository the needed .crt and .key files from /etc/pki/centos/certs/ directory
While the mentioned above script is probably the one that we'll use the most for nodes, we can also have to create service account, just to retrieve TLS cert used to auth against other services. As we'll just do that on very limited use cases, we can just "manually" execute the following snippet, still with first a valid kerberos ticket to be able to add users in IPA (so also on an enrolled machine and ideally the same one we use for the node certificates) :
# Let's first define some variables
service_account="mbox_stg_kojira"
full_name="CentOS kojira mbox STG service user"
realm="STG.FEDORAPROJECT.ORG"
# Before next steps we *have* to have ipa rights to create ipa users and valid kerberos ticket
pushd /etc/pki/centos/certs >/dev/null
ipa user-show ${service_account} >/dev/null 2>&1 || ipa user-add --cn="${full_name}" --displayname="${full_name}" --password --first=${service_account} --last=${service_account} ${service_account}
# Now that we have created the account with strong and random password, we can create private key and csr and ask IPA CA to sign it
# Create a private key and csr first
test -e ${service_account}.key || openssl req -new -newkey rsa:2048 -days 3650 -nodes -keyout ${service_account}.key -out ${service_account}.csr -subj "/CN=${service_account}"
# kinit as user (will ask password and also twice to set new password which can be the same one we decided to use
kinit ${service_account}@${realm}
ipa cert-request ${service_account}.csr --principal=${service_account} --profile-id=userCerts --certificate-out=${service_account}.crt && rm ${service_account}.csr
You can now push both .key/.crt files into pkistore git-crypted repository and also record the service account password in the IPA-service-accounts file in that same git-crypted repository
Warning
This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.
We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup).
Actually that node is acme01.rdu2.centos.org.
We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh We just have once to add in our zone (main one, so centos.org) a CNAME pointing to the delegated zone acme.centos.org Example (simple) for forums.centos.org :
_acme-challenge.forums IN CNAME _acme-challenge.forums.acme.centos.org.
Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record
acme.sh --issue --keylength 2048 --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org
Let's see what this produces on our DNS node, basically updating TXT record:
Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"
And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:
[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org' [Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain [Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org' [Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ" [Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success. [Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first. [Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success. [Tue 10 Dec 10:36:19 UTC 2019] All success, let's return [Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org [Tue 10 Dec 10:36:22 UTC 2019] Success [Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records. [Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org [Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt [Tue 10 Dec 10:36:22 UTC 2019] Removed: Success [Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign. [Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579 [Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd [Tue 10 Dec 10:36:24 UTC 2019] Cert success.
PS : for a wildcard, just add multiple -d and *.domain
acme.sh --issue --keylength 2048 --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org
All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir
Note
worth knowing that for AWS/Route53 hosted zones (as we now have some for CI infra and openshift), one can use --dns dns_aws [option](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#10-use-amazon-route53-domain-api to request new TLS certs. And it will be automatic for renewal so nothing to worry aboutt
Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)
You can add multiple SANs in the same certs. Here is one example with buildlogs.centos.org and SAN cloud.centos.org :
acme.sh --issue --keylength 2048 -d buildlogs.centos.org -d cloud.centos.org -w /var/www/html/
All files (certs/keys) are then available under /root/.acme.sh/{hostname} (you'll have to import those into this pkistore dir)
Each pkistore git repository (based on the env) will have a ./tools/letsencrypt-renew-import wrapper tool that will :
acme.sh --renew -d ${domain} --forceOnce all done and validated, you can just git commit && git push back as usual
Example :
./tools/letsencrypt-renew-import [+] Analyzing TLS cert accounts.centos.org.crt ... Renewing [accounts.centos.org.crt] on [acme01.rdu2.centos.org] TLS cert accounts.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS [accounts.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.centos.org-CAChain.crt] : SUCCESS [+] Analyzing TLS cert accounts.dev.centos.org.crt ... TLS [accounts.dev.centos.org.crt] file is a symlink so ignoring ... SKIPPED [+] Analyzing TLS cert accounts.stg.centos.org.crt ... Renewing [accounts.stg.centos.org.crt] on [acme01.rdu2.centos.org] TLS cert accounts.stg.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS [accounts.stg.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.stg.centos.org-CAChain.crt] : SUCCESS <...>
Note
if you'll have an error on a specific cert, just ssh into delegate machine for acme.sh and manually kick acme.sh --renew -d <domain> --force to see the output and fix the underlying issue (if any)
Same as for dns challenges as we consolidated all under acme.sh (and no certbot anymore)
Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first. Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository :
./validate_public_chain koji.mbox.centos.org.crt Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle koji.mbox.centos.org.crt: OK
Let's consider now three infrastructures and how to push renewed certs :
Once it's committed/pushed to pkistore git repo, the ansible bot will deploy the renewed TLS certs automatically. You can still "force" the playbook execution if you want, from ansible bot host but should be done automatically and you can see reports through ARA.
Same as for other parts of infra, except that you have to encrypt with ansible-vault before git commit/git push operations (important). Once done :
ansible-playbook-stream playbooks/role-haproxy.yml --tags "tls,pki"
Note
Fedora Infra team is responsible for mirrors.centos.org one, as it's hosted on their infra. In the past we were generating/renewing cert but it seems now automated