SAPHanaSR(7)

NAME
SAPHanaSR - Tools for automating SAP HANA system replication in scale-up setups.

DESCRIPTION
Overview

The SAPHanaSR package provides resource agents (RA) and tools for setting up and managing automation of SAP HANA system replication (SR) in scale-up setups. System replication will help to replicate the database data from one site to another site in order to compensate for database failures. With this mode of operation, internal SAP HANA high-availability (HA) mechanisms and the Linux cluster have to work together.

The SAPHana RA performs the actual check of the SAP HANA database instances and is configured as a master/slave resource. Managing the two SAP HANA instances means that the resource agent controls the start/stop of the instances. In addition the resource agent is able to monitor the SAP HANA databases on landscape host configuration level. For this monitoring the resource agent relies on interfaces provided by SAP. As long as the HANA landscape status is not "ERROR" the Linux cluster will not act. The main purpose of the Linux cluster is to handle the take-over to the other site. Only if the HANA landscape status indicates that HANA can not recover from the failure and the replication is in sync, then Linux will act.

An important task of the resource agent is to check the synchronisation status of the two SAP HANA databases. If the synchronisation is not "SOK", then the cluster avoids to takeover to the secondary side, if the primary fails. This is to improve the data consistency.

Note: To automate SAP HANA SR in scale-out setups, please use the package SAPHanaSR-ScaleOut.

Scenarios

In order to illustrate the meaning of the above overview, some important situations are described below. This is not a complete description of all situations.

1. Site take-over after primary has failed
This is the basic use case for HANA SR automation.
1. site_A is primary, site_B is secondary - they are in sync.
2. site_A crashes.
3. site_B does the take-over and runs now as new primary. The Linux cluster always makes sure the node at site_B is shut off before any other action is initiated.

2. Prevention against dual-primary
A primary absolutely must never be started, if the cluster does not know anything about the other site. On initial cluster start, the cluster needs to detect a valid HANA system replication setup, including system replication status (SOK) and last primary timestamp (LPT). This is neccessary to ensure data integrity.

The rational behind this is shown in the following scenario:
1. site_A is primary, site_B is secondary - they are in sync.
2. site_A crashes (remember the HANA ist still marked primary).
3. site_B does the take-over and runs now as new primary.
4. DATA GETS CHANGED ON NODE2 BY PRODUCTION
5. The admin also stops the cluster on site_B (we have two HANAs both
   internally marked down and primary now).
6. What, if the admin would now restart the cluster on site_A?
6.1 site_A would take its own CIB after waiting for the initial fencing
    time for site_B.
6.2 It would "see" its own (cold) primary and the fact that there was a
    secondary.
6.3 It would start the HANA from point of time of step 1.->2. (the crash),
    so all data changed inbetween would be lost.
This is why the Linux cluster needs to enforce a restart inhibit.

There are two options to get back both, SAP HANA SR and the Linux cluster, into a fully functional state:
a) the admin starts both nodes again
b) In the situation where the site_B is still down, the admin starts the
   primary on site_A manually.
The Linux cluster will follow this administrative decision. In both cases the administrator should register and start a secondary as soon as posible. This avoids a full log partition with consequence of a DATABASE STUCK.

3. Automatic registration as secondary after site failure and takeover
The cluster can be configured to register a former primary database automatically as secondary. If this option is set, the resource agent will register a former primary database as secondary during cluster/resource start.

4. Site take-over not preferred over local re-start
SAPHanaSR-ScaleOut allows to configure, if you prefer to takeover to the secondary after the primary landscape fails. The alternative is to restart the primary landscape, if it fails and only to takeover when no local restart is possible anymore. This can be tuned by SAPHana(7) parameters.
The current implementation only allows to takeover in case the landscape status reports 1 (ERROR). The cluster will not takeover, when the SAP HANA still tries to repair a local failure.

Implementation

The two HANA database systems (primary and secondary site) are managed by the same single Linux cluster. The number of nodes in that single Linux cluster usually is two.

The HANA consists of two sites with one node each.
Note: A third site can be added, with another system replication (aka replication chaining). This will be tolerated by the cluster. Nevertheless, this additional pieces are not managed by the cluster. E.g. if the direction of system replication between the two cluster nodes changes, the third site must be disconnected before hand. So, such a replication chaining does not allow to use the AUTOMATED_REGISTER feature of SAPHanaSR.

A common STONITH mechanism is set up for all nodes across all the sites.

Since the IP address of the primary HANA database system is managed by the cluster, only that single IP address is needed to access any nameserver candidate.

Best Practice

* Use two independent corosync rings, at least one of them on bonded network. Resulting in at least three physical links. Unicast is preferred.

* Use Stonith Block Device (SBD), shared LUNs across all nodes on all sites. Of course, together with hardware watchdog.

* Align all timeouts in the Linux cluster with the timeouts of the underlying storage and multipathing.

* Check the installation of OS and Linux cluster on all nodes before doing any functional tests.

* Carefully define, perform, and document tests for all scenarios that should be covered.

* Test HANA features without Linux cluster before doing the overall cluster tests.

* Test basic Linux cluster features without HANA before doing the overall cluster tests.

* Be patient. For detecting the overall HANA status, the Linux cluster needs a certain amount of time, depending on the HANA and the configured intervalls and timeouts.

* Before doing anything, always check for the Linux cluster's idle status, left-over migration constraints, and resource failures as well as the HANA landscape status, and the HANA SR status.

REQUIREMENTS

For the current version of the package SAPHanaSR, the support is limited to the following scenarios and parameters:

1. HANA scale-up cluster with system replication. The two HANA database systems (primary and secondary site) are managed by one Linux cluster. The number of nodes in that single Linux cluster is two. Note: A three-node Linux cluster is possible, but not covered by current best practices.

2. Technical users and groups such as sidadm are defined locally in the Linux system.

3. Strict time synchronization between the cluster nodes, e.g. NTP.

4. For scale-up the following SAP HANA SR scenarios are supported with the SAPHanaSR package:
    4.1 performance-optimized (memory preload on secondary)
    4.2 cost-optimized (e.g. with QA on secondary)
    4.3 multi-tier (replication chaining, third node connected to second one)
    4.4 single-tenant or multi-tenant (MDC) for all of the above
    4.5 multiple independent HANA SR pairs (MCOS) in one cluster
Note: For MCOS, there must be no constraints between HANA SR pairs.

5. Only one system replication for the SAP HANA database.

6. Both SAP HANA database systems have the same SAP Identifier (SID) and Instance Number.

7. Besides SAP HANA you need SAP hostagent to be installed and started on your system.

8. Automated start of SAP HANA database systems during system boot must be switched off.

9. The RA's monitoring operations have to be active.

10. For scale-up, the current resource agent supports SAP HANA in system replication beginning with HANA version 1.0 SPS 7 patch level 70, recommended is SPS 11. HANA version 2.0 is supported as well.

BUGS
In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.

SEE ALSO
ocf_suse_SAPHanaTopology(7) , ocf_suse_SAPHana(7) , ocf_heartbeat_IPaddr2(7) , SAPHanaSR-monitor(8) , SAPHanaSR-showAttr(8) , ntp.conf(5) , stonith(8) , sbd(8) , stonith_sbd(7) , crm(8) , corosync.conf(5) , crm_no_quorum_policy(7) , cs_precheck_for_hana(8) , cs_add_watchdog_to_initrd(8)
https://www.suse.com/products/sles-for-sap/resource-library/sap-best-practices.html ,
https://www.suse.com/releasenotes/ ,
https://www.susecon.com/doc/2015/sessions/TUT19921.pdf ,
https://www.suse.com/media/presentation/TUT90846_towards_zero_downtime%20_how_to_maintain_sap_hana_system_replication_clusters.pdf ,
http://scn.sap.com/community/hana-in-memory/blog/2014/04/04/fail-safe-operation-of-sap-hana-suse-extends-its-high-availability-solution ,
http://scn.sap.com/docs/DOC-60334 ,
http://scn.sap.com/community/hana-in-memory/blog/2015/12/14/sap-hana-sps-11-whats-new-ha-and-dr--by-the-sap-hana-academy ,
https://wiki.scn.sap.com/wiki/display/ATopics/HOW+TO+SET+UP+SAPHanaSR+IN+THE+COST+OPTIMIZED+SAP+HANA+SR+SCENARIO+-+PART+I

AUTHORS
F.Herschel, L.Pinne.

COPYRIGHT
(c) 2015-2018 SUSE Linux GmbH, Germany.
(c) 2019 SUSE LLC
The package SAPHanaSR comes with ABSOLUTELY NO WARRANTY.
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html