# TLS certificates in Infra 

## Common naming convention.

Ansible roles are using the following logic to distribute .key/.cert files

```

  * {{ public_name }}.key : TLS private key

  * {{ public_name }}.crt : signed TLS certificate

  * {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)

```

## Internal certificates

### IPA/dogtag (central authentication)

While IPA enrolled nodes can directly request TLS certificates, for CentOS infra, due to the fact that almost *all* nodes *can't* be enrolled (no direct link with IPA infra), we have to delegate this on an enrolled node where we can then be granted "delegation" rights in IPA, so that a "fake" enrolled node (created for the real target server that will need a TLS cert) can be "managed by" the enrolled node.

From that enrolled node, we'll then be able to retrieve the TLS cert/keytab and also then export/import into `pkistore` , following the same naming convention as described above.

Depending on the Env (Prod vs STG), you can find which node[s] is/are enrolled (through applied `ipa-client` client role applied with also the `ipa_client_tls_delegated_host` boolean set to `True` (needed for the following script to be present in the ipa-client role).

Pre-requisites:

 * one node enrolled in the correct REALM we want to generate/retrieve TLS cert (and keytab) for

 * an IPA account that has enough privileges to add hosts/services and local sudo rights on the intermediate enrolled node

 * `ipa-client` role applied with correct script deployed

!!! note

    The following steps are just for *new* certificates. As once you'll have requested this on the enrolled node, the `certmonger` process will automatically watch and request/renew new ones, so they'll land on the enrolled node automatically, from which you can then retrieve TLS files (from /etc/pki/tls/certs) and update pkistore (see above). To help with that see <pkistore>/koji/retrieve_from_ipa script

Once we have shell access on such enrolled node, we can proceed like this :

```

/usr/libexec/centos/generate_ipa_tls_krb5 

You need to call the script like this : /usr/libexec/centos/generate_ipa_tls_krb5 -arguments

 -n : node name / fqdn ([REQUIRED], example 'ppc64-01.cbs.centos.org')

 -d : Description for that host ([REQUIRED], example 'cbs koji builder')

 -s : service for principal ([REQUIRED], example 'compile' would create compile/ppc64-01.cbs.centos.org service in IPA)

 -h : display this help

You also need a valid kerberos ticket otherwise script will exit

```

Here is an output example for the command with arguments:

```

/usr/libexec/centos/generate_ipa_tls_krb5 -n mbs.mbox.stg.centos.org -d 'Koji mbox STG mbs' -s 'HTTP'

[+] Adding host in IPA and adding delegation to retrieve certs/keytab ...

------------------------------------

Added host "mbs.mbox.stg.centos.org"

------------------------------------

  Host name: mbs.mbox.stg.centos.org

  Description: Koji mbox STG mbs

  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Password: False

  Keytab: False

  Managed by: mbs.mbox.stg.centos.org

  Host name: mbs.mbox.stg.centos.org

  Description: Koji mbox STG mbs

  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org

-------------------------

Number of members added 1

-------------------------

------------------------------------------------------------------

Added service "HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG"

------------------------------------------------------------------

  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Managed by: mbs.mbox.stg.centos.org

  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org

-------------------------

Number of members added 1

-------------------------

  Host name: mbs.mbox.stg.centos.org

  Description: Koji mbox STG mbs

  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org

  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org

-------------------------

Number of members added 1

-------------------------

  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG

  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org

  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org

-------------------------

Number of members added 1

-------------------------

[+] Retrieving TLS and keytab files ...

New signing request "20210705130958" added.

Keytab successfully retrieved and stored in: /etc/pki/centos/certs/HTTP-mbs.mbox.stg.centos.org.keytab

[+] Validating TLS against IPA CA ...

/etc/pki/centos/certs/mbs.mbox.stg.centos.org.crt: OK

```

!!! tip

    You need a valid kerberos ticket for the existing REALM, otherwise same script will exit 1. Worth knowing that if you have enabled OTP on your account (a *must* for admins) you can use the `2fa-kinit` script, also deployed by ansible on each enrolled node by the `ipa-client` ansible role

You can now import into `pkistore` (and correct directory based on role) git-crypted repository the needed .crt and .key files from /etc/pki/centos/certs/ directory

#### TLS service account

While the mentioned above script is probably the one that we'll use the most for nodes, we can also have to create service account, just to retrieve TLS cert used to auth against other services. As we'll just do that on *very* limited use cases, we can just "manually" execute the following snippet, still with first a valid kerberos ticket to be able to add users in IPA (so also on an enrolled machine and ideally the same one we use for the node certificates) :

```

# Let's first define some variables

service_account="mbox_stg_kojira"

full_name="CentOS kojira mbox STG service user"

realm="STG.FEDORAPROJECT.ORG"

# Before next steps we *have* to have ipa rights to create ipa users and valid kerberos ticket

pushd /etc/pki/centos/certs >/dev/null

ipa user-show ${service_account} >/dev/null 2>&1 || ipa user-add --cn="${full_name}" --displayname="${full_name}" --password --first=${service_account} --last=${service_account} ${service_account}

# Now that we have created the account with strong and random password, we can create private key and csr and ask IPA CA to sign it

# Create a private key and csr first

test -e ${service_account}.key || openssl req -new -newkey rsa:2048 -days 3650 -nodes -keyout ${service_account}.key -out ${service_account}.csr -subj "/CN=${service_account}"

# kinit as user (will ask password and also twice to set new password which can be the same one we decided to use

kinit ${service_account}@${realm}

ipa cert-request ${service_account}.csr --principal=${service_account} --profile-id=userCerts --certificate-out=${service_account}.crt && rm ${service_account}.csr

```

You can now push both .key/.crt files into `pkistore` git-crypted repository *and* also record the service account password in the IPA-service-accounts file in that same git-crypted repository

### Red Hat CA (internal setup only)

## Public certificates

### Letsencrypt

!!! warning

    This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.

We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup). 

Actually that node is `acme01.rdu2.centos.org`.

#### How to obtain new cert (DNS challenge is the preferred way)

##### For dns challenge

We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh

We just have once to add in our zone (main one, so centos.org)  a CNAME pointing to the delegated zone acme.centos.org

Example (simple) for forums.centos.org : 

```

_acme-challenge.forums IN  CNAME _acme-challenge.forums.acme.centos.org.

```

Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record

```

acme.sh --issue --keylength 2048 --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org

```

Let's see what this produces on our DNS node, basically updating TXT record:

```

Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved

Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"

```

And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:

```

[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org'

[Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain

[Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org'

[Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain:  _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"

[Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success.

[Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first.

[Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success.

[Tue 10 Dec 10:36:19 UTC 2019] All success, let's return

[Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org

[Tue 10 Dec 10:36:22 UTC 2019] Success

[Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records.

[Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt

[Tue 10 Dec 10:36:22 UTC 2019] Removed: Success

[Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign.

[Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579

[Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd

[Tue 10 Dec 10:36:24 UTC 2019] Cert success.

```

PS : for a wildcard, just add multiple -d and `*.domain`

```

acme.sh --issue --keylength 2048 --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org

```

All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir

!!! note

    worth knowing that for AWS/Route53 hosted zones (as we now have some for CI infra and openshift), one can use `--dns dns_aws` [option](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#10-use-amazon-route53-domain-api to request new TLS certs. And it will be automatic for renewal so nothing to worry aboutt

##### For http challenge 

Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)

You can add multiple SANs in the same certs. Here is one example with buildlogs.centos.org and SAN cloud.centos.org :

```

acme.sh --issue --keylength 2048 -d buildlogs.centos.org -d cloud.centos.org -w /var/www/html/

```

All files (certs/keys) are then available under /root/.acme.sh/{hostname} (you'll have to import those into this pkistore dir)

#### How to renew existing certs

##### For DNS challenges (existing records)

Each `pkistore` git repository (based on the env) will have a `./tools/letsencrypt-renew-import` wrapper tool that will : 

 * inspect .crt TLS files in the pkistore git repo

 * verify if that's signed by Let's Encrypt CA

 * access the central acme machine through ssh (from which you initially create new cert) and renew with `acme.sh --renew -d ${domain} --force`

 * retrieve the .crt and CA chain, and also corresponding .key

 * reencrypt (if needed, based on env) with ansible-vault

Once all done and validated, you can just git commit && git push back as usual

Example : 

```

./tools/letsencrypt-renew-import

[+] Analyzing TLS cert accounts.centos.org.crt ...

 Renewing [accounts.centos.org.crt] on [acme01.rdu2.centos.org]

 TLS cert accounts.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 

 [accounts.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.centos.org-CAChain.crt] : SUCCESS 

[+] Analyzing TLS cert accounts.dev.centos.org.crt ...

 TLS [accounts.dev.centos.org.crt] file is a symlink so ignoring ... SKIPPED  

[+] Analyzing TLS cert accounts.stg.centos.org.crt ...

 Renewing [accounts.stg.centos.org.crt] on [acme01.rdu2.centos.org]

 TLS cert accounts.stg.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 

 [accounts.stg.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.stg.centos.org-CAChain.crt] : SUCCESS 

<...>

```

!!! note

    if you'll have an error on a specific cert, just ssh into delegate machine for acme.sh and manually kick `acme.sh --renew -d <domain> --force` to see the output and fix the underlying issue (if any)

##### For HTTP challenges

Same as for dns challenges as we consolidated all under `acme.sh` (and no certbot anymore)

### Deploying through ansible

Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first.

Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository : 

```

./validate_public_chain koji.mbox.centos.org.crt 

Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle

koji.mbox.centos.org.crt: OK

```

Let's consider now three infrastructures and how to push renewed certs :

#### CentOS public infra (including .dev. , .stg. and ci infra)

Once it's committed/pushed to pkistore git repo,  the ansible bot will deploy the renewed TLS certs automatically.

You can still "force" the playbook execution if you want, from ansible bot host but should be done automatically and you can see reports through ARA.

#### CentOS Stream infra

Same as for other parts of infra, except that you *have* to encrypt with ansible-vault before git commit/git push operations (important).

Once done : 

```

ansible-playbook-stream playbooks/role-haproxy.yml --tags "tls,pki"

```

!!! note

    Fedora Infra team is responsible for `mirrors.centos.org` one, as it's hosted on their infra. In the past we were generating/renewing cert but it seems now [automated](https://pagure.io/fedora-infrastructure/issue/10829)