# Adding Compute/Worker nodes 

This SOP should be used in the following scenario:

- Red Hat OpenShift Container Platform 4.x cluster has been installed some time ago (1+ days ago) and additional worker nodes are required to increase the capacity for the cluster.

## Steps

1. Add the new nodes being added to the cluster to the appropriate inventory file in the appropriate group.

eg:

```

# ocp, compute/worker:

[ocp-ci-compute]

newnode1.example.centos.org

newnode2.example.centos.org

newnode3.example.centos.org

newnode4.example.centos.org

newnode5.example.centos.org

```

eg:

```

# ocp.stg, compute/worker:

[ocp-stg-ci-compute]

newnode6.example.centos.org

newnode7.example.centos.org

newnode8.example.centos.org

newnode9.example.centos.org

# ocp.stg, master/control plane

[ocp-stg-ci-master]

newnode10.example.centos.org

```

2. Examine the `inventory` file for `ocp` or `ocp.stg` and determine which management node corresponds with the group `ocp-ci-management`.

eg:

```

[ocp-ci-management]

some-managementnode.example.centos.org

```

3. Find the OCP admin user which is contained in the hostvars for this management node at the key `ocp_service_account`.

eg:

```

host_vars/some-managementnode.example.centos.org:ocp_service_account: adminuser

```

4. SSH to the node identified in step `2`, and become the user identified in step `3`.

eg:

```

ssh some-managementnode.example.centos.org

sudo su - adminuser

```

5. Verify that you are authenticated correctly to the Openshift cluster as the `system:admin`.

```

oc whoami

system:admin

```

6. Retrieve the certificate from the internal API and convert the contents to base64 string like so.

eg:

```

echo "q" | openssl s_client -connect api-int.ocp.ci.centos.org:22623  -showcerts | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/' | base64 --wrap=0

DONE

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXCERTSTOREDASABASE64ENCODEDSTRING=

```

7. Replace the cert in the compute/worker ignition file, at the `XXXXXXXXREPLACEMEXXXXXXXX=` point, be sure to save this change in SCM, and push.

```

cat filestore/rhcos/compute.ign

{"ignition":{"config":{"append":[{"source":"https://api-int.ocp.ci.centos.org:22623/config/worker","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"data:text/plain;charset=utf-8;base64,XXXXXXXXREPLACEMEXXXXXXXX=","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{"disks":[{"device":"/dev/sdb","wipeTable":true}]},"systemd":{}}

```

8. Once the ignition file has been updated, run the `adhoc-provision-ocp4-node` playbook to copy the updated ignition files up to the http server, and install the new node(s). When prompted, specify the hostname of the new node. Best to do one at a time, it takes a minute or two per new node being added at this step.

eg:

```

ansible-playbook playbooks/adhoc-provision-ocp4-node.yml

[WARNING] Nodes to be fully wiped/reinstalled with OCP => : newnode6.example.centos.org

```

9. As the new nodes are provisioned, they will attempt to join the cluster. They must first be accepted.

```

# List the certs. If you see status pending, this is the worker/compute nodes attempting to join the cluster. It must be approved.

oc get csr

# Accept all node CSRs one liner

oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

```

10. Finally run the playbook to update haproxy config to monitor the new nodes.

```

ansible-playbook playbooks/role-haproxy.yml --tags="config"

```

To see more information about adding new worker/compute nodes to a user provisioned infrastructure based OCP4 cluster see the detailed steps at [1],[2].

# Adding/Replacing etcd/control plane nodes 

Depending on the scenario (just adding more control planes nodes or just installing new one as one failed), you'll need to take some actions first (or not)

## Deleting from cluster a dead node (hardware issue) (and only if needed)

If you have one unrecoverable node and that you don't even want to reinstall on same node (same hostname/ip address/etc), you can start by following [official doc](https://docs.openshift.com/container-platform/4.9/backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.html#restore-identify-unhealthy-etcd-member_replacing-unhealthy-etcd-member)

So basically :

  * reviewing which node to remove with from etcd cluster (`oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd`)

  * taking remote shell on one of the remaining etcd nodes (`oc rsh -n openshift-etcd <one of the remaining nodes still reachable>`)

    * delete it from cluster (`etcdctl member remove <node_id>`)

  * remove secrets for *that* node from openshit (` oc get secrets -n openshift-etcd | grep <failed_hostname> |awk '{print $1}'|while read secret ; do oc delete secret -n openshift-etcd ${secret};done` )

  * delete node from openshift (`oc delete node 

  * We can now go to next step to install a new one as replacement

## Deploying a new control plane node

From this step it's the same methodology to deploy an additional etcd node or just (re)install a failed node : 

  * *first* step is to reflect etcd current nodes in ansible inventory (important) and play the haproxy role so that load-balancer will point to new and future solution (including node not yet installed)

  * same for dns zone : update both forward *and* reverse for the etcd records (it's using SRV type lookup to find other etcd nodes in etcd cluster)

  * only once dns and haproxy config are applied by ansible you can proceed by just installing a new node

### Installing control plane node 

More or less same thing as for compute node :

 * retrieve api tls cert

 * update ignition file (except that it's `master.ign` in this case)

 * deploy the node

 * wait for it to be installed and updated to cluster version (matching deployed openshift version)

 * classical `oc get csr` and process pending requests

 * node should then be listed as `<new_hostname>   NotReady   master `  through `oc get nodes`

 * once all signed csr are processed, you should see activity through `oc get pods -n openshift-etcd ` and some containers being created and finally appearing as `Ready` 

### Resources

- [1] [How to add Openshift 4 RHCOS worker nodes in UPI <24 hours](https://access.redhat.com/solutions/4246261)

- [2] [How to add Openshift 4 RHCOS worker nodes to UPI >24 hours](https://access.redhat.com/solutions/4799921)