Blame docs/security/tls.md

12bb45
# TLS certificates in Infra 
12bb45
12bb45
## Common naming convention.
12bb45
Ansible roles are using the following logic to distribute .key/.cert files
12bb45
12bb45
```
12bb45
  * {{ public_name }}.key : TLS private key
12bb45
  * {{ public_name }}.crt : signed TLS certificate
12bb45
  * {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)
12bb45
12bb45
```
12bb45
09ef13
## Internal certificates
09ef13
09ef13
### IPA/dogtag (central authentication)
09ef13
09ef13
While IPA enrolled nodes can directly request TLS certificates, for CentOS infra, due to the fact that almost *all* nodes *can't* be enrolled (no direct link with IPA infra), we have to delegate this on an enrolled node where we can then be granted "delegation" rights in IPA, so that a "fake" enrolled node (created for the real target server that will need a TLS cert) can be "managed by" the enrolled node.
09ef13
09ef13
From that enrolled node, we'll then be able to retrieve the TLS cert/keytab and also then export/import into `pkistore` , following the same naming convention as described above.
09ef13
09ef13
Depending on the Env (Prod vs STG), you can find which node[s] is/are enrolled (through applied `ipa-client` client role applied with also the `ipa_client_tls_delegated_host` boolean set to `True` (needed for the following script to be present in the ipa-client role).
09ef13
09ef13
Pre-requisites:
09ef13
09ef13
 * one node enrolled in the correct REALM we want to generate/retrieve TLS cert (and keytab) for
09ef13
 * an IPA account that has enough privileges to add hosts/services and local sudo rights on the intermediate enrolled node
09ef13
 * `ipa-client` role applied with correct script deployed
09ef13
474c50
!!! note
45279b
    The following steps are just for *new* certificates. As once you'll have requested this on the enrolled node, the `certmonger` process will automatically watch and request/renew new ones, so they'll land on the enrolled node automatically, from which you can then retrieve TLS files (from /etc/pki/tls/certs) and update pkistore (see above). To help with that see <pkistore>/koji/retrieve_from_ipa script
474c50
09ef13
Once we have shell access on such enrolled node, we can proceed like this :
43cbca
43cbca
```
43cbca
/usr/libexec/centos/generate_ipa_tls_krb5 
43cbca
43cbca
You need to call the script like this : /usr/libexec/centos/generate_ipa_tls_krb5 -arguments
43cbca
 -n : node name / fqdn ([REQUIRED], example 'ppc64-01.cbs.centos.org')
43cbca
 -d : Description for that host ([REQUIRED], example 'cbs koji builder')
43cbca
 -s : service for principal ([REQUIRED], example 'compile' would create compile/ppc64-01.cbs.centos.org service in IPA)
43cbca
 -h : display this help
43cbca
43cbca
You also need a valid kerberos ticket otherwise script will exit
43cbca
43cbca
```
43cbca
43cbca
Here is an output example for the command with arguments:
43cbca
09ef13
```
09ef13
/usr/libexec/centos/generate_ipa_tls_krb5 -n mbs.mbox.stg.centos.org -d 'Koji mbox STG mbs' -s 'HTTP'
09ef13
[+] Adding host in IPA and adding delegation to retrieve certs/keytab ...
09ef13
------------------------------------
09ef13
Added host "mbs.mbox.stg.centos.org"
09ef13
------------------------------------
09ef13
  Host name: mbs.mbox.stg.centos.org
09ef13
  Description: Koji mbox STG mbs
09ef13
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Password: False
09ef13
  Keytab: False
09ef13
  Managed by: mbs.mbox.stg.centos.org
09ef13
  Host name: mbs.mbox.stg.centos.org
09ef13
  Description: Koji mbox STG mbs
09ef13
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
09ef13
-------------------------
09ef13
Number of members added 1
09ef13
-------------------------
09ef13
------------------------------------------------------------------
09ef13
Added service "HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG"
09ef13
------------------------------------------------------------------
09ef13
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Managed by: mbs.mbox.stg.centos.org
09ef13
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
09ef13
-------------------------
09ef13
Number of members added 1
09ef13
-------------------------
09ef13
  Host name: mbs.mbox.stg.centos.org
09ef13
  Description: Koji mbox STG mbs
09ef13
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
09ef13
  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org
09ef13
-------------------------
09ef13
Number of members added 1
09ef13
-------------------------
09ef13
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
09ef13
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
09ef13
  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org
09ef13
-------------------------
09ef13
Number of members added 1
09ef13
-------------------------
09ef13
[+] Retrieving TLS and keytab files ...
09ef13
New signing request "20210705130958" added.
09ef13
Keytab successfully retrieved and stored in: /etc/pki/centos/certs/HTTP-mbs.mbox.stg.centos.org.keytab
09ef13
[+] Validating TLS against IPA CA ...
09ef13
/etc/pki/centos/certs/mbs.mbox.stg.centos.org.crt: OK
09ef13
```
09ef13
09ef13
!!! tip
09ef13
    You need a valid kerberos ticket for the existing REALM, otherwise same script will exit 1. Worth knowing that if you have enabled OTP on your account (a *must* for admins) you can use the `2fa-kinit` script, also deployed by ansible on each enrolled node by the `ipa-client` ansible role
09ef13
09ef13
You can now import into `pkistore` (and correct directory based on role) git-crypted repository the needed .crt and .key files from /etc/pki/centos/certs/ directory
09ef13
474c50
43cbca
#### TLS service account
43cbca
43cbca
While the mentioned above script is probably the one that we'll use the most for nodes, we can also have to create service account, just to retrieve TLS cert used to auth against other services. As we'll just do that on *very* limited use cases, we can just "manually" execute the following snippet, still with first a valid kerberos ticket to be able to add users in IPA (so also on an enrolled machine and ideally the same one we use for the node certificates) :
43cbca
43cbca
```
43cbca
# Let's first define some variables
43cbca
service_account="mbox_stg_kojira"
43cbca
full_name="CentOS kojira mbox STG service user"
43cbca
realm="STG.FEDORAPROJECT.ORG"
43cbca
43cbca
# Before next steps we *have* to have ipa rights to create ipa users and valid kerberos ticket
43cbca
pushd /etc/pki/centos/certs >/dev/null
43cbca
ipa user-show ${service_account} >/dev/null 2>&1 || ipa user-add --cn="${full_name}" --displayname="${full_name}" --password --first=${service_account} --last=${service_account} ${service_account}
43cbca
43cbca
# Now that we have created the account with strong and random password, we can create private key and csr and ask IPA CA to sign it
43cbca
# Create a private key and csr first
43cbca
test -e ${service_account}.key || openssl req -new -newkey rsa:2048 -days 3650 -nodes -keyout ${service_account}.key -out ${service_account}.csr -subj "/CN=${service_account}"
43cbca
# kinit as user (will ask password and also twice to set new password which can be the same one we decided to use
43cbca
kinit ${service_account}@${realm}
43cbca
ipa cert-request ${service_account}.csr --principal=${service_account} --profile-id=userCerts --certificate-out=${service_account}.crt && rm ${service_account}.csr
43cbca
```
43cbca
43cbca
You can now push both .key/.crt files into `pkistore` git-crypted repository *and* also record the service account password in the IPA-service-accounts file in that same git-crypted repository
12bb45
09ef13
### Red Hat CA (internal setup only)
09ef13
09ef13
 
09ef13
## Public certificates
12bb45
12bb45
### Letsencrypt
12bb45
12bb45
!!! warning
12bb45
    This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.
12bb45
12bb45
12bb45
We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup). 
c4af45
Actually that node is `acme01.rdu2.centos.org`.
12bb45
12bb45
#### How to obtain new cert (DNS challenge is the preferred way)
12bb45
12bb45
##### For dns challenge
12bb45
We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh
12bb45
We just have once to add in our zone (main one, so centos.org)  a CNAME pointing to the delegated zone acme.centos.org
12bb45
Example (simple) for forums.centos.org : 
12bb45
12bb45
```
12bb45
_acme-challenge.forums IN  CNAME _acme-challenge.forums.acme.centos.org.
12bb45
```
12bb45
Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record
12bb45
12bb45
```
55ea1e
acme.sh --issue --keylength 2048 --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org
12bb45
```
12bb45
12bb45
Let's see what this produces on our DNS node, basically updating TXT record:
12bb45
12bb45
```
12bb45
Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved
12bb45
Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"
12bb45
```
12bb45
12bb45
And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:
12bb45
12bb45
```
12bb45
[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org'
12bb45
[Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain
12bb45
[Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org'
12bb45
[Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain:  _acme-challenge.forums.acme.centos.org
12bb45
[Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"
12bb45
[Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success.
12bb45
[Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first.
12bb45
[Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org
12bb45
[Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success.
12bb45
[Tue 10 Dec 10:36:19 UTC 2019] All success, let's return
12bb45
[Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Success
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records.
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Removed: Success
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign.
12bb45
[Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579
12bb45
[Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd
12bb45
[Tue 10 Dec 10:36:24 UTC 2019] Cert success.
12bb45
12bb45
```
12bb45
09ef13
PS : for a wildcard, just add multiple -d and `*.domain`
09ef13
12bb45
```
55ea1e
acme.sh --issue --keylength 2048 --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org
12bb45
```
12bb45
All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir
12bb45
e51791
!!! note
e51791
    worth knowing that for AWS/Route53 hosted zones (as we now have some for CI infra and openshift), one can use `--dns dns_aws` [option](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#10-use-amazon-route53-domain-api to request new TLS certs. And it will be automatic for renewal so nothing to worry aboutt
e51791
12bb45
##### For http challenge 
12bb45
Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)
12bb45
 
c4af45
You can add multiple SANs in the same certs. Here is one example with buildlogs.centos.org and SAN cloud.centos.org :
12bb45
12bb45
```
55ea1e
acme.sh --issue --keylength 2048 -d buildlogs.centos.org -d cloud.centos.org -w /var/www/html/
12bb45
12bb45
```
c4af45
All files (certs/keys) are then available under /root/.acme.sh/{hostname} (you'll have to import those into this pkistore dir)
12bb45
12bb45
12bb45
12bb45
#### How to renew existing certs
12bb45
##### For DNS challenges (existing records)
fd8995
fd8995
Each `pkistore` git repository (based on the env) will have a `./tools/letsencrypt-renew-import` wrapper tool that will : 
fd8995
fd8995
 * inspect .crt TLS files in the pkistore git repo
fd8995
 * verify if that's signed by Let's Encrypt CA
fd8995
 * access the central acme machine through ssh (from which you initially create new cert) and renew with `acme.sh --renew -d ${domain} --force`
fd8995
 * retrieve the .crt and CA chain, and also corresponding .key
fd8995
 * reencrypt (if needed, based on env) with ansible-vault
fd8995
fd8995
Once all done and validated, you can just git commit && git push back as usual
fd8995
fd8995
Example : 
fd8995
12bb45
```
fd8995
./tools/letsencrypt-renew-import
fd8995
[+] Analyzing TLS cert accounts.centos.org.crt ...
fd8995
 Renewing [accounts.centos.org.crt] on [acme01.rdu2.centos.org]
fd8995
 TLS cert accounts.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 
fd8995
 [accounts.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.centos.org-CAChain.crt] : SUCCESS 
fd8995
fd8995
[+] Analyzing TLS cert accounts.dev.centos.org.crt ...
fd8995
 TLS [accounts.dev.centos.org.crt] file is a symlink so ignoring ... SKIPPED  
fd8995
fd8995
[+] Analyzing TLS cert accounts.stg.centos.org.crt ...
fd8995
 Renewing [accounts.stg.centos.org.crt] on [acme01.rdu2.centos.org]
fd8995
 TLS cert accounts.stg.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 
fd8995
 [accounts.stg.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.stg.centos.org-CAChain.crt] : SUCCESS 
fd8995
<...>
fd8995
12bb45
```
12bb45
fd8995
!!! note
fd8995
    if you'll have an error on a specific cert, just ssh into delegate machine for acme.sh and manually kick `acme.sh --renew -d <domain> --force` to see the output and fix the underlying issue (if any)
fd8995
    
fd8995
c4af45
##### For HTTP challenges
12bb45
c4af45
Same as for dns challenges as we consolidated all under `acme.sh` (and no certbot anymore)
12bb45
12bb45
### Deploying through ansible
12bb45
Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first.
12bb45
Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository : 
12bb45
12bb45
```
12bb45
./validate_public_chain koji.mbox.centos.org.crt 
12bb45
Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle
12bb45
koji.mbox.centos.org.crt: OK
12bb45
12bb45
```
12bb45
20ca83
Let's consider now three infrastructures and how to push renewed certs :
12bb45
46eecc
#### CentOS public infra (including .dev. , .stg. and ci infra)
46eecc
Once it's committed/pushed to pkistore git repo,  the ansible bot will deploy the renewed TLS certs automatically.
20ca83
You can still "force" the playbook execution if you want, from ansible bot host but should be done automatically and you can see reports through ARA.
20ca83
1843bb
20ca83
#### CentOS Stream infra
20ca83
Same as for other parts of infra, except that you *have* to encrypt with ansible-vault before git commit/git push operations (important).
20ca83
Once done : 
20ca83
20ca83
```
20ca83
ansible-playbook-stream playbooks/role-haproxy.yml --tags "tls,pki"
20ca83
```
12bb45
03001f
!!! note
8615fe
    Fedora Infra team is responsible for `mirrors.centos.org` one, as it's hosted on their infra. In the past we were generating/renewing cert but it seems now [automated](https://pagure.io/fedora-infrastructure/issue/10829)
46eecc