# CentOS CI Infra Outage Preparation
During a scheduled outage where it is likely we will lose network access entirely to the entire rack, or between racks, it is advisable to shutdown the following services:

- Duffy
- CentOS CI Openshift prod/stg
- Legacy CI Jenkins
- Legacy OKD
- keepalived on gateway02.ci.centos.org


## Legacy OKD
1. bstinson, as the only person on the team which has access to the legacy OKD cluster, must handle tasks related to this cluster.


## OCP
https://github.com/centosci/ocp4-docs/blob/master/sops/create_etcd_backup.md
https://github.com/centosci/ocp4-docs/blob/master/sops/cordoning_nodes_and_draining_pods.md
https://github.com/centosci/ocp4-docs/blob/master/sops/graceful_shutdown_ocp_cluster.md

Admin nodes
Prod: ocp-admin.ci.centos.org
Stg: n4-136.cloud.ci.centos.org

2. Take etcd backup to the admin node associated with prod/stg
3. Cordon and drain all nodes
4. gracefully shutdown


## Duffy

5. switch off duffy - workers
    * source duffy2-venv/bin/activate; FLASK_APP=duffy DUFFY_SETTINGS=/etc/duffy.conf python scripts/worker.py
6. switch off duffy server
    * FLASK_APP=duffy DUFFY_SETTINGS=/etc/duffy.conf flask run -h 0.0.0.0 -p 8080
7. ci.centos.org legacy jenkins: manage jenkins, prepare for shutdown
    * ssh jenkins - systemctl restart jenkins

## keepalived on Gateway nodes

8. Shutdown keepalived on gateway02.ci.centos.org
   *  sudo systemctl stop keepalived