From 1b039c5a66f5c806370843c75f1b2201b3e1b457 Mon Sep 17 00:00:00 2001 From: Fabian Arrotin Date: Oct 27 2021 12:05:18 +0000 Subject: Some notes about backups in infra Signed-off-by: Fabian Arrotin --- diff --git a/docs/infra/backup.md b/docs/infra/backup.md new file mode 100644 index 0000000..ec8d9be --- /dev/null +++ b/docs/infra/backup.md @@ -0,0 +1,44 @@ +# CentOS Backups + +## Overview + +We don't have a lot of real data to backup in the CentOS Infra, so we started with just something simple and easy that was just extended over the years. +We'll describe below the various methods/levels of backup that are complementary and used in serial. + +Basically just using following tools: + + * tar/gzip for quick compressed archives + * mysqldump/pg_dump/mongodump for various DB backup operations + * rsync + * over ssh for remote nodes + * over plain rsyncd for nodes in same DC and through specific module/target + * restic for rotation and multiple backup storage pools + +Let's have a look at involved steps : + +### Node/server level + +We have a dedicated ansible role [centos-backup](https://github.com/CentOS/ansible-role-centos-backup) that is used for items to backup/archive on each node that needs a backup. +That role can be declared directly on a node/group in ansible inventory and have `centos_backup_folders` ansible variable/list filled with some folders to locally compress. +All local backups will be saved under dedicated unprivileged home folder on same node, with some local retention (so works for very small disk space usage) + +Also worth noticing that other roles can import dynamically the centos-backup role (like for example the `mysql` one, etc) and can also drop a snippet under /etc/centos-backup.d/ so that it would be added to existing backup plan. + +Having local backup isn't enough though (of course !), but that's where next step kicks in : + +### Central server level + +A dedicated node having the same `centos-backup` role applied *but* with the `centos_backup_server` boolean set to True would be dedicated to centrally backup nodes from previous steps. +It's basically the node having the needed key for unprivileged backup user that will be use to rsync locally the archives generated at the node/server level (see previous step). It can also use plain rsync:// url if there is a valid rsyncd target/module at the server level. + +All the hosts to backup are declared in the `centos_backup_server_nodes_list` ansible list +Other settings also permit to use or not local restic tool for local encryption/rotation/dedup. + +The central node will also report backup status to zabbix and all logs are available in `centos_backup_server_logdir` + + +### Offsite level + +For some nodes (including but not limited to central server node) we can also export encrypted backup on a remote storage space. +That's what the ansible [restic](https://github.com/CentOS/ansible-role-restic) role will do and that permits to also have remote (encrypted) backup using rotation on (for example) dedicated and private S3 bucket on AWS + diff --git a/mkdocs.yml b/mkdocs.yml index b9345be..c7bb0ec 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -92,6 +92,7 @@ nav: - infra/centos-ci.md - infra/monitoring.md - infra/team.md + - infra/backup.md - Security: - security/index.md - security/tls.md