Blame SOURCES/rear-bz1743303-rubrik.patch

67735e
diff --git a/.gitignore b/.gitignore
67735e
index 5e3dc940..a644c865 100644
67735e
--- a/.gitignore
67735e
+++ b/.gitignore
67735e
@@ -4,3 +4,6 @@
67735e
 build-stamp
67735e
 /var
67735e
 /etc/rear/site.conf
67735e
+.DS_Store
67735e
+.vscode
67735e
+servers
67735e
diff --git a/doc/user-guide/16-Rubrik-CDM.adoc b/doc/user-guide/16-Rubrik-CDM.adoc
67735e
new file mode 100644
67735e
index 00000000..41f37d20
67735e
--- /dev/null
67735e
+++ b/doc/user-guide/16-Rubrik-CDM.adoc
67735e
@@ -0,0 +1,106 @@
67735e
+= Documentation for the Rubrik Cloud Data Management (CDM) Backup and Restore Method
67735e
+
67735e
+== Summary
67735e
+
67735e
+The Rubrik CDM backup and restore method for ReaR allows Rubrik CDM to perform bare metal recovery of Linux systems that are supported by ReaR. It does this by including the installed Rubrik CDM RBS agent files in the ISO that is created by `rear mkrescue` via a pre-script in the fileset. The ISO is left in place under `/var/lib/rear/output/rear-<hostname>.iso` by default. During the fileset backup Rubrik will backup the main operating system files as well as the ReaR ISO file. 
67735e
+
67735e
+Bare Metal Recovery is performed by first restoring the ReaR ISO file from Rubrik CDM to an alternate host. Next the host being restored is booted from the ISO via CD/DVD, USB, vSphere Datastore ISO, etc... Once booted running `rear recover` will prepare the host for restore and start the Rubrik CDM RBS agent. If the host has a new IP address the new RBS agent will need to be registered with the Rubrik cluster. Registration is not necessary if the recovery host is reusing the same IP address as the original. All of the files for the host are then recovered from Rubrik CDM to the recovery host's `/mnt/local` directory by the user. Once complete the user exit's ReaR and reboots the host. 
67735e
+
67735e
+== Configuration
67735e
+
67735e
+1. Install and configure ReaR in accordance with:
67735e
+- Red Hat 
67735e
+   * https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-relax-and-recover_rear
67735e
+- Ubuntu
67735e
+   * http://manpages.ubuntu.com/manpages/disco/en/man8/rear.8.html
67735e
+- SUSE
67735e
+   * https://en.opensuse.org/SDB:Disaster_Recovery
67735e
+   * https://documentation.suse.com/sle-ha/15-SP1/html/SLE-HA-all/cha-ha-rear.html
67735e
+- Generic
67735e
+   * https://github.com/rear/rear
67735e
+
67735e
+   NOTE: Ignore any instructions to configure external storage like NFS, CIFS/SMB or ftp. Also ignore any instructions to configure a specific backup method. This will be taken care of in the next steps. 
67735e
+
67735e
+   NOTE: Ignore any instructions to schedule ReaR to run via the host based scheduler (cron). Rubrik CDM will run ReaR via a pre-script in the fileset. If this is not preferred ReaR can be scheduled on the host, however, the ISOs created may not be in sync with the backups.
67735e
+
67735e
+   NOTE: If installing the pre-release or development version for which there is no installer, copy the repo to the host being protected. Then run `make install` from its root directory of the repo. 
67735e
+
67735e
+1. Install the Rubrik CDM RBS agent as directed by the Rubrik documentation.
67735e
+1. Edit `/etc/rear/local.conf` and enter:
67735e
+
67735e
+   OUTPUT=ISO
67735e
+   BACKUP=CDM
67735e
+
67735e
+1. Test `ReaR` by running `rear -v mkrescue`
67735e
+1. Configure fileset backup of the host and add `/usr/sbin/rear mkrescue` as a prescript. 
67735e
+1. ISOs will be saved as `/var/lib/rear/output/*.iso`
67735e
+
67735e
+- Recovery 
67735e
+
67735e
+1. Recover `/var/lib/rear/output/rear-<hostname>.iso` from host to be restored. 
67735e
+1. Boot recovery machine using recovered ISO.
67735e
+   
67735e
+   NOTE: Recovered system will use the same networking as the original machine. Verify no IP conflicts will occur. 
67735e
+
67735e
+   NOTE: If the same static IP address may be used it will need to be changed if the original machine is still running.
67735e
+
67735e
+1. Verify Firewall is down on recovery host.
67735e
+1. Run `rear recover`
67735e
+1. Answer inline questions until `rear>` prompt appears.
67735e
+1. Run `ps -eaf` and verify that `backup_agent_main` and `bootstrap_agent_main` are running.
67735e
+1. Get the IP address of the system using `ip addr`
67735e
+1. Register the new IP with the Rubrik appliance (if needed)
67735e
+1. Perform a re-directed export of `/` to `/mnt/local`
67735e
+1. Reboot
67735e
+1. Recover other file systems as needed.
67735e
+
67735e
+   Note: that the Rubrik RBS agent will connect as the original machine now. The host may need to be reinstalled and re-registered if the original machine is still running. 
67735e
+
67735e
+== Known Issues
67735e
+
67735e
+* Recovery via IPv6 is not yet supported.
67735e
+* Automatic recovery from replica CDM cluster is not supported
67735e
+* CDM may take some time to recognize that the IP address has moved from one system to another. When restoring using the same IP give CDM up to 10 minutes to recognize that the agent is running on another machine. This usually comes up during testing when the original machine is shutdown but not being restored to. 
67735e
+* Recovery from a replica CDM cluster is only supported with CDM v4.2.1 and higher.
67735e
+* Care must be taken with SUSE systems on DHCP. They tend to request the same IP as the original host. If this is not the desired behavior the system will have to be adjusted after booting from the ReaR ISO.  
67735e
+* If multiple restores are performed using the same temporary IP, the temporary IP must first be deleted from Servers & Apps -> Linux and Unix Servers and re-added upon each reuse.
67735e
+* ReaR's `ldd` check of other binaries or libraries may result in libraries not being found. This can generally be fixed by adding the path to those libraries to the `LD_LIBRARY_PATH` variable in `/etc/rear/local.conf`. Do this by adding the following line in `/etc/rear/local.conf`:
67735e
++
67735e
+  export LD_LIBRARY_PATH-"$LD_LIBRARY_PATH:<path>"
67735e
++
67735e
+To make CentoOS v7.7 work the following line was needed:
67735e
++
67735e
+  export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export"
67735e
++
67735e
+To make CentOS v8.0 work the following line was needed:
67735e
++
67735e
+  export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export:/usr/lib64/eog:/usr/lib64/python3.6/site-packages:/usr/lib64/samba:/usr/lib64/firefox"
67735e
+
67735e
+== Troubleshooting
67735e
+
67735e
+* Verify that ReaR will recover your system without using the CDM backup and restore method. Most errors are due to configuration with ReaR itself and not Rubrik CDM. Use the default ReaR backup and restore method to test with. 
67735e
+* Follow the OS specific configuration guides as mentioned at the beginning of this document. 
67735e
+
67735e
+== Test Matrix
67735e
+
67735e
+.Test Matrix
67735e
+[%header,format=csv]
67735e
+|===
67735e
+Operating System,DHCP,Static IP,Virtual,Physical,LVM Root Disk,Plain Root Disk,EXT3,EXT4,XFS,BTRFS,Original Cluster,Replication Cluster
67735e
+CentOS 7.3,,pass,Pass,,Pass,,,,Pass,,Pass,
67735e
+CentOS 7.6,Pass,,Pass,,Pass,,,,Pass,,Pass,
67735e
+CentOS 7.7,Pass,,Pass,Pass,Pass,,,,Pass,,Pass,
67735e
+CentOS 8.0,Pass,,Pass,,Pass,,,,Pass,,Pass,
67735e
+CentOS 5.11,,,,,,,,,,,,
67735e
+CentOS 6.10,,,,,,,,,,,,
67735e
+RHEL 7.6,Pass,,Pass,,Pass,,,,,,,
67735e
+RHEL 7.4,,,,,,,,,,,,
67735e
+RHEL 6.10,,,,,,,,,,,,
67735e
+SUSE 11 SP4,,,,,,,,,,,,
67735e
+SUSE 12 SP4,Pass (uses same IP as original),,Pass,,,,,,,Pass,Pass,
67735e
+Ubuntu 14.04 LTS,,,,,,,,,,,,
67735e
+Ubuntu 16.04 LTS,Pass,,,,Pass,,,Pass,,,Pass,
67735e
+Ubuntu 17.04 LTS,,,,,,,,,,,,
67735e
+|===
67735e
+
67735e
+* Empty cells indicate that no tests were run.
67735e
diff --git a/usr/share/rear/conf/default.conf b/usr/share/rear/conf/default.conf
107ca0
index 0f0d0675..56967132 100644
67735e
--- a/usr/share/rear/conf/default.conf
67735e
+++ b/usr/share/rear/conf/default.conf
67735e
@@ -1334,6 +1334,17 @@ LANG_RECOVER=C
67735e
 # low-quality master encryption key. For details, see the cryptsetup(8) manual page.
67735e
 LUKS_CRYPTSETUP_OPTIONS="--iter-time 2000 --use-random"
67735e
 
67735e
+##
67735e
+# BACKUP=CDM (Rubrik CDM; Cloud Data Managemnt)
67735e
+##
67735e
+# ReaR support for Rubrik Cloud Data Managment (CDM). 
67735e
+# ReaR will copy the Rubrk RBS agent and required OS binaries to its ISO for incluson on boot. 
67735e
+# ReaR will start the Rubrik RBS agent when 'rear recover' is run.
67735e
+COPY_AS_IS_CDM=( /etc/rubrik /usr/bin/rubrik /var/log/rubrik /etc/pki /usr/lib64 )
67735e
+COPY_AS_IS_EXCLUDE_CDM=( /var/log/rubrik/* )
67735e
+PROGS_CDM=( /usr/bin/rubrik/backup_agent_main /usr/bin/rubrik/bootstrap_agent_main openssl uuidgen )
67735e
+
67735e
+
67735e
 ##
67735e
 # BACKUP=FDRUPSTREAM stuff
67735e
 ##
67735e
diff --git a/usr/share/rear/prep/CDM/default/400_prep_cdm.sh b/usr/share/rear/prep/CDM/default/400_prep_cdm.sh
67735e
new file mode 100644
67735e
index 00000000..d3fd11b7
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/prep/CDM/default/400_prep_cdm.sh
67735e
@@ -0,0 +1,7 @@
67735e
+#
67735e
+# prepare stuff for CDM
67735e
+#
67735e
+
67735e
+COPY_AS_IS=( "${COPY_AS_IS[@]}" "${COPY_AS_IS_CDM[@]}" )
67735e
+COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" "${COPY_AS_IS_EXCLUDE_CDM[@]}" )
67735e
+PROGS=( "${PROGS[@]}" "${PROGS_CDM[@]}" fmt )
67735e
diff --git a/usr/share/rear/prep/CDM/default/450_check_cdm_client.sh b/usr/share/rear/prep/CDM/default/450_check_cdm_client.sh
67735e
new file mode 100644
67735e
index 00000000..637fac5f
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/prep/CDM/default/450_check_cdm_client.sh
67735e
@@ -0,0 +1,13 @@
67735e
+# 450_check_cdm_client.sh
67735e
+# 
67735e
+# This script checks if a Rubrik CDM client is installed and running
67735e
+#
67735e
+
67735e
+Log "Backup method is Rubrik (CDM): check backup_agent_main"
67735e
+if [ ! -x /usr/bin/rubrik/backup_agent_main ]; then
67735e
+    StopIfError 1 "Please install Rubrik (CDM) RBS client software."
67735e
+fi
67735e
+
67735e
+ps ax | grep -v grep | grep backup_agent_main
67735e
+StopIfError $? "Rubrik (CDM) RBS backup_agent_main was not running on this client."
67735e
+
67735e
diff --git a/usr/share/rear/restore/CDM/default/400_restore_with_cdm.sh b/usr/share/rear/restore/CDM/default/400_restore_with_cdm.sh
67735e
new file mode 100644
67735e
index 00000000..bc4811c4
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/restore/CDM/default/400_restore_with_cdm.sh
67735e
@@ -0,0 +1,19 @@
67735e
+# 400_restore_with_cdm.sh
67735e
+#
67735e
+#
67735e
+
67735e
+LogPrint "Please start the restore process on the Rubrik (CDM) cluster."
67735e
+
67735e
+if is_true $CDM_NEW_AGENT_UUID; then
67735e
+  LogPrint ""
67735e
+  LogPrint "Register the appropriate IP address from this list with Rubrik (CDM):"
67735e
+  LogPrint "$( ip addr | grep inet | cut -d / -f 1 | grep -v 127.0.0.1 | grep -v ::1 )"
67735e
+  LogPrint ""
67735e
+fi
67735e
+LogPrint "Make sure all required data is restored to $TARGET_FS_ROOT ."
67735e
+LogPrint ""
67735e
+LogPrint "Next type 'exit' to continue the recovery."
67735e
+LogPrint "Info: You can check the recovery process i.e. with the command 'df'."
67735e
+LogPrint ""
67735e
+
67735e
+rear_shell "Has the restore been completed and are you ready to continue the recovery?"
67735e
diff --git a/usr/share/rear/verify/CDM/default/410_use_replica_cdm_cluster_cert.sh b/usr/share/rear/verify/CDM/default/410_use_replica_cdm_cluster_cert.sh
67735e
new file mode 100644
107ca0
index 00000000..518387e3
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/verify/CDM/default/410_use_replica_cdm_cluster_cert.sh
107ca0
@@ -0,0 +1,88 @@
67735e
+# 410_use_replica_cdm_cluster_cert.sh
107ca0
+# If restoring from a replica Rubrik (CDM) cluster use its cert for RBS.
107ca0
+
107ca0
+LogPrint "If restoring from a replica Rubrik (CDM) cluster its cert will be downloaded and used for RBS"
67735e
+
67735e
+CDM_RBA_DIR=/etc/rubrik
67735e
+CDM_KEYS_DIR=${CDM_RBA_DIR}/keys
67735e
+
107ca0
+local prompt="Is the data being restored from the original CDM Cluster?"
107ca0
+local input_value=""
107ca0
+local wilful_input=""
67735e
+while true ; do
67735e
+    # Find out if the restore is being done from the original CDM cluster or a Replica
107ca0
+    # the default (i.e. the automated response after the timeout) should be 'no':
107ca0
+    input_value="$( UserInput -I CDM_REPLICA_CLUSTER -p "$prompt" -D 'no' )" && wilful_input="yes" || wilful_input="no"
107ca0
+    if is_false "$input_value" ; then
107ca0
+        if is_true "$wilful_input" ; then
107ca0
+            LogPrint "User confirmed the data is not being restored from the original CDM Cluster"
107ca0
+        else
107ca0
+            LogPrint "Assuming the data is not being restored from the original CDM Cluster"
107ca0
+        fi
67735e
+        break
67735e
+    fi
107ca0
+    if is_true "$input_value" ; then
107ca0
+        LogPrint "User confirmed the data is being restored from the original CDM Cluster"
107ca0
+        return 0
107ca0
+    fi
67735e
+done
67735e
+
107ca0
+LogPrint "Downloading cert from replica CDM cluster"
107ca0
+# The name of the tar file that is being downloaded has changed in Rubrik CDM v5.1.
107ca0
+# Before Rubrik CDM v5.1 it was rubrik-agent-sunos5.10.sparc.tar.gz
107ca0
+# since Rubrik CDM v5.1 it is rubrik-agent-solaris.sparc.tar.gz
107ca0
+# cf. https://github.com/rear/rear/issues/2441
67735e
+CDM_SUNOS_TAR=rubrik-agent-sunos5.10.sparc.tar.gz
107ca0
+CDM_SOLARIS_TAR=rubrik-agent-solaris.sparc.tar.gz
107ca0
+pushd $TMPDIR
107ca0
+while true ; do
107ca0
+    prompt="Enter one of the IP addresses for the replica CDM cluster (or 'no' to cancel)"
107ca0
+    CDM_CLUSTER_IP="$( UserInput -I CDM_CLUSTER_IP -r -t 0 -p "$prompt" )"
107ca0
+    test $CDM_CLUSTER_IP || continue
107ca0
+    if is_false "$CDM_CLUSTER_IP" ; then
107ca0
+        LogPrint "User canceled downloading cert from replica CDM cluster (data restore may fail now)"
107ca0
+        popd
107ca0
+        return 0
107ca0
+    fi
107ca0
+    # When curl fails for all files continue with an empty CDM_TAR_FILE to denote that nothing was downloaded:
107ca0
+    for CDM_TAR_FILE in $CDM_SOLARIS_TAR $CDM_SUNOS_TAR '' ; do
107ca0
+        test $CDM_TAR_FILE || continue
107ca0
+        curl $v -fskLOJ https://${CDM_CLUSTER_IP}/connector/${CDM_TAR_FILE} && break
107ca0
+    done
107ca0
+    if ! test -s "$CDM_TAR_FILE" ; then
107ca0
+        LogPrintError "Could not download Rubrik agent from https://${CDM_CLUSTER_IP}/connector/${CDM_SOLARIS_TAR} or https://${CDM_CLUSTER_IP}/connector/${CDM_SUNOS_TAR}"
107ca0
+        while true ; do
107ca0
+            prompt="Enter URL to download Rubrik agent tar archive (or 'no' to cancel)"
107ca0
+            CDM_AGENT_URL="$( UserInput -I CDM_AGENT_URL -r -t 0 -p "$prompt" )"
107ca0
+            test $CDM_AGENT_URL || continue
107ca0
+            if is_false "$CDM_AGENT_URL" ; then
107ca0
+                LogPrint "User canceled downloading Rubrik agent (data restore may fail now)"
107ca0
+                popd
107ca0
+                return 0
107ca0
+            fi
107ca0
+            curl $v -fskLOJ $CDM_AGENT_URL && break
107ca0
+            LogPrintError "Could not download Rubrik agent from $CDM_AGENT_URL"
107ca0
+        done
107ca0
+        CDM_TAR_FILE=$( basename "$CDM_AGENT_URL" )
107ca0
+    fi
107ca0
+    if ! tar $v -xzf $CDM_TAR_FILE ; then
107ca0
+        LogPrintError "Could not extract Rubrik agent (failed to 'tar -xzf $CDM_TAR_FILE')"
107ca0
+        continue
107ca0
+    fi
107ca0
+    CDM_CERT_FILE=$(find ./ -name "rubrik.crt")
107ca0
+    mv $v ${CDM_KEYS_DIR}/rubrik.crt ${CDM_KEYS_DIR}/rubrik.crt.orig
107ca0
+    if ! cp $v $CDM_CERT_FILE $CDM_KEYS_DIR ; then
107ca0
+        LogPrintError "Could not copy replica CDM cluster certificate"
107ca0
+        continue
107ca0
+    fi
107ca0
+    chmod $v 600 ${CDM_KEYS_DIR}/rubrik.crt
107ca0
+    mv $v ${CDM_KEYS_DIR}/agent.crt ${CDM_KEYS_DIR}/agent.crt.orig
107ca0
+    mv $v ${CDM_KEYS_DIR}/agent.pem ${CDM_KEYS_DIR}/agent.pem.orig
107ca0
+    # TODO: Actually do something if /etc/rubrik/rba-keygen.sh failed.
107ca0
+    # Is /etc/rubrik/rba-keygen.sh perhaps only optional?
107ca0
+    # cf. https://github.com/rear/rear/pull/2445#discussion_r448217873
107ca0
+    /etc/rubrik/rba-keygen.sh || LogPrintError "/etc/rubrik/rba-keygen.sh failed (data restore may also fail)"
107ca0
+    break
107ca0
+done
107ca0
+popd
107ca0
+LogPrint "Replica Rubrik (CDM) cluster certificate installed"
67735e
diff --git a/usr/share/rear/verify/CDM/default/430_gen_rbs_uuid_for_cdm.sh b/usr/share/rear/verify/CDM/default/430_gen_rbs_uuid_for_cdm.sh
67735e
new file mode 100644
67735e
index 00000000..5e99b79c
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/verify/CDM/default/430_gen_rbs_uuid_for_cdm.sh
67735e
@@ -0,0 +1,29 @@
67735e
+# 430_gen_rbs_uuid_for_cdm.sh
67735e
+# Reset the UUID used by RBS if the IP address has changed
67735e
+
67735e
+CDM_RBA_DIR=/etc/rubrik
67735e
+CDM_AGENT_UUID=${CDM_RBA_DIR}/conf/uuid
67735e
+
67735e
+# When USER_INPUT_CDM_SAME_AGENT_UUID has Does this client have the same IP address as the original 'y' was actually meant:
67735e
+LogPrint ""
67735e
+LogPrint "Found the following IP addresses on this system:"
67735e
+LogPrint "$( ip addr | grep inet | cut -d / -f 1 | grep -v 127.0.0.1 | grep -v ::1 )"
67735e
+LogPrint ""
67735e
+is_true "$USER_INPUT_CDM_SAME_AGENT_UUID" && USER_INPUT_SAME_AGENT_UUID="y"
67735e
+while true ; do
67735e
+    # Find out if the IP address has changed from the original. If so generate a new UUID.
67735e
+    # the default (i.e. the automated response after the timeout) should be 'n':
67735e
+    answer="$( UserInput -I CDM_SAME_AGENT_UUID -p "Does this client have the same IP address as the original? (y/n)" -D 'y' -t 300 )"
67735e
+    is_true "$answer" && return 0
67735e
+    if is_false "$answer" ; then
67735e
+        break
67735e
+    fi
67735e
+    UserOutput "Please answer 'y' or 'n'"
67735e
+done
67735e
+
67735e
+mv $v ${CDM_AGENT_UUID} ${CDM_AGENT_UUID}.old
67735e
+/usr/bin/uuidgen | tee -a ${CDM_AGENT_UUID} >&2
67735e
+StopIfError "Unable to generate new UUID"
67735e
+
67735e
+CDM_NEW_AGENT_UUID="true"
67735e
+LogPrint "Rubrik (CDM) RBS agent now has new UUID."
67735e
diff --git a/usr/share/rear/verify/CDM/default/450_start_cdm_rbs.sh b/usr/share/rear/verify/CDM/default/450_start_cdm_rbs.sh
67735e
new file mode 100644
67735e
index 00000000..571da1da
67735e
--- /dev/null
67735e
+++ b/usr/share/rear/verify/CDM/default/450_start_cdm_rbs.sh
67735e
@@ -0,0 +1,17 @@
67735e
+# 450_start_cdm_rbs.sh
67735e
+# Start the Rubrik (CDM) RBS Agent
67735e
+
67735e
+RBA_DIR=/etc/rubrik
67735e
+RBA_BIN_DIR=/usr/bin/rubrik
67735e
+
67735e
+BOOTSTRAP_DAEMON_OPTS="$( < ${RBA_DIR}/conf/bootstrap_flags.conf )"
67735e
+AGENT_DAEMON_OPTS="$( < ${RBA_DIR}/conf/agent_flags.conf )"
67735e
+BOOTSTRAP_DAEMON=$RBA_BIN_DIR/bootstrap_agent_main
67735e
+AGENT_DAEMON=$RBA_BIN_DIR/backup_agent_main
67735e
+
67735e
+$BOOTSTRAP_DAEMON $BOOTSTRAP_DAEMON_OPTS
67735e
+StopIfError "Unable to start RBS Bootstrap service"
67735e
+$AGENT_DAEMON $AGENT_DAEMON_OPTS
67735e
+StopIfError "Unable to start RBS Agent service"
67735e
+
67735e
+LogPrint "Rubrik (CDM) RBS agent started."