# CentOS CI Infra Outage Preparation

During a scheduled outage where it is likely we will lose network access entirely to the entire rack, or between racks, it is advisable to shutdown the following services:

- Duffy

- CentOS CI Openshift prod/stg

- Legacy CI Jenkins

- Legacy OKD

- keepalived on gateway02.ci.centos.org

## Legacy OKD

1. bstinson, as the only person on the team which has access to the legacy OKD cluster, must handle tasks related to this cluster.

## OCP

https://github.com/centosci/ocp4-docs/blob/master/sops/create_etcd_backup.md

https://github.com/centosci/ocp4-docs/blob/master/sops/cordoning_nodes_and_draining_pods.md

https://github.com/centosci/ocp4-docs/blob/master/sops/graceful_shutdown_ocp_cluster.md

Admin nodes

Prod: ocp-admin.ci.centos.org

Stg: n4-136.cloud.ci.centos.org

2. Take etcd backup to the admin node associated with prod/stg

3. Cordon and drain all nodes

4. gracefully shutdown

## Duffy

5. switch off duffy - workers

    * source duffy2-venv/bin/activate; FLASK_APP=duffy DUFFY_SETTINGS=/etc/duffy.conf python scripts/worker.py

6. switch off duffy server

    * FLASK_APP=duffy DUFFY_SETTINGS=/etc/duffy.conf flask run -h 0.0.0.0 -p 8080

7. ci.centos.org legacy jenkins: manage jenkins, prepare for shutdown

    * ssh jenkins - systemctl restart jenkins

## keepalived on Gateway nodes

8. Shutdown keepalived on gateway02.ci.centos.org

   *  sudo systemctl stop keepalived