ocf_suse_SAPHanaController(7)

NAME
SAPHanaController - Manages take-over between two SAP HANA databases with system replication.

SYNOPSYS
SAPHanaController [start | stop | status | monitor | promote | demote | notify | meta-data | validate-all | methods | usage ]

DESCRIPTION
SAPHanaController is a resource agent for SAP HANA databases in scale-out setups. It manages take-over for a SAP HANA database with system replication in an OCF multi-state configuration.

System replication will help to replicate the database data from one site to another site in order to compensate for database failures. With this mode of operation, internal SAP HANA high-availability (HA) mechanisms and the Linux cluster have to work together.

An HANA scale-out setup already is, to some degree, an HA cluster on its own. The HANA is able to replace failing nodes with standby nodes or to restart certain sub-systems on other nodes. As long as the HANA landscape status is not "ERROR" the Linux cluster will not act. The main purpose of the Linux cluster is to handle the take-over to the other site. Only if the HANA landscape status indicates that HANA can not recover from the failure and the replication is in sync, then Linux will act. As an exception, the Linux cluster will react if HANA moves the master nameserver role to another candidate. SAPHanaController is also able to restart former failed worker nodes as standby. In addition to the SAPHanaTopology RA, the SAPHanaSR-ScaleOut solution uses a "HA/DR provider" API provided by HANA to get informed about the current state of the system replication.

On initial cluster start, the cluster needs to detect a valid HANA system replication setup, including system replication status (SOK) and last primary timestamp (LPT). This is neccessary to ensure data integrity.

The SAPHanaController resource agent (RA) performs the actual check of the SAP HANA database instances and is configured as a multi-state resource. The Linux cluster resource's master status will follow the HANA's active master name server role. Resources (f.e. IP address) that need to run on the same node with that HANA role should be co-located with the master target.

Managing the two SAP HANA instances in scale-out setup means that the resource agent controls the start/stop of the instances. In addition the resource agent is able to monitor the SAP HANA databases on landscape host configuration level. For this monitoring the resource agent relies on interfaces provided by SAP. A third task of the resource agent is to also check the synchronisation status of the two SAP HANA databases. If the synchronisation is not "SOK", then the cluster avoids to take-over to the secondary side, if the primary fails. This is to improve the data consistency.

The resource agent uses the following five interfaces provided by SAP:

1. sapcontrol/sapstartsrv
The interface sapcontrol/sapstartsrv is used to start/stop a HANA database instance/system. SAPHanaController also checks via this interface the instance status to allow a start of HANA standby nodes.

2. landscapeHostConfiguration
The interface is used to monitor an entire HANA system. The python script is named landscapeHostConfiguration.py. landscapeHostConfiguration.py has some detailed output about HANA system status and node roles. For our monitor the overall status is relevant. This overall status is reported by the return code of the script: 0: Internal Fatal, 1: ERROR, 2: WARNING, 3: INFO, 4: OK . The SAPHanaController resource agent will interpret return code 0 as FATAL, 1 as not-running (or ERROR) and return codes 2+3+4 as RUNNING.

3. hdbnsutil
The interface hdbnsutil is used to check the "topology" of the system replication as well as the current configuration (primary/secondary) of a SAP HANA database instance. A second task of the interface is the posibility to run a system replication take-over (sr_takeover) or to register a former primary to a newer one (sr_register). In scale-out setups, hdbnsutil sometimes might take some time.

4. systemReplicationStatus
The interface systemReplicationStatus is a special query to check the HANA system replication status via a python library. This interface exists since SAP HANA SPS9.

5. saphostctrl
The interface saphostctrl uses the function ListInstances to figure out the virtual host name of the SAP HANA instance. This is typically the hostname used during the HANA installation.

To make configuring the cluster as simple as possible, the additional SAPHanaTopology resource agent runs on all nodes of the Linux cluster and gathers information about the statuses and configurations of SAP HANA system replication. The SAPHanaTopology RA is designed as a normal (stateless) clone.

Please see also the REQUIREMENTS section below.

SUPPORTED PARAMETERS
This resource agent supports the following parameters:

SID
    SAP System Identifier. Has to be same on both instances. Example "SID=SLE".

InstanceNumber
    Number of the SAP HANA database. Has to be same on both instances. For system replication also Instance Number+1 is blocked. Example "InstanceNumber=00".

DIR_EXECUTABLE
    The full qualified path where to find sapstartsrv and sapcontrol. Specify this parameter, if you have changed the SAP kernel directory location after the default SAP installation.
    Optional, well known directories will be searched by default.

DIR_PROFILE
    The full qualified path where to find the SAP START profile. Specify this parameter, if you have changed the SAP profile directory location after the default SAP installation.
    Optional, well known directories will be searched by default.

HANA_CALL_TIMEOUT
    Define timeout how long a call to HANA to receive information can take. This could be e.g. landscapeHostConfiguration.py. There are some specific calls to HANA which have their own timeout values. For example the sr_takeover command does not timeout (inf). If the timeout is reached, the return code will be 124. If you increase the timeouts for HANA calls you should also adjust the operation timeouts of your Linux cluster resources.
    Optional. Default value: 120.

INSTANCE_PROFILE
    The name of the SAP HANA instance profile. Specify this parameter, if you have changed the name of the SAP HANA instance profile after the default SAP installation. Normally you do not need to set this parameter.
    Optional, well known directories will be searched by default.

PREFER_SITE_TAKEOVER
    Defines whether RA should prefer to take-over to the secondary database instead of restarting master locally. However a take-over will only be triggered, if the SAP HANA landscape status is on "ERROR". Example: "PREFER_SITE_TAKEOVER=true".
    Optional. Default value: false.

DUPLICATE_PRIMARY_TIMEOUT
    Time difference needed between two primary time stamps (LPTs), in case a dual-primary situation occurs. If the difference between both node's last primary time stamps is less than DUPLICATE_PRIMARY_TIMESTAMP, then the cluster holds one or both instances in a "WAITING" status. This is to give an admin the chance to react on a failover. Note: How the cluster proceeds after the DUPLICATE_PRIMARY_TIMESTAMP has passed, depends on the parameter AUTOMATED_REGISTER. See also the examples section below.
    Optional. Default value: 7200.

AUTOMATED_REGISTER
    Defines whether a former primary database should be registered automatically by the resource agent during cluster/resource start, if the DUPLICATE_PRIMARY_TIMEOUT condition is met. Example: "AUTOMATED_REGISTER=true".
    Default value: false.

SUPPORTED PROPERTIES
hana_${sid}_glob_filter
    Global cluster property hana_${sid}_glob_filter . This property should only be set if requested by support engineers. The default is sufficient for normal operation.

SUPPORTED ACTIONS
This resource agent supports the following actions (operations):

start
    Starts the HANA instance or bring the "clone instance" to a WAITING status. Suggested minimum timeout: 3600.

stop
    Stops the HANA instance. Suggested minimum timeout: 3600.

promote
    Either runs a take-nover for a secondary or a just-nothing for a primary. Suggested minimum timeout: 900.

demote
    Nearly does nothing and just marks the instance as demoted. Suggested minimum timeout: 320.

notify
    Always returns SUCCESS. Suggested minimum timeout: 10.

status
    Reports whether the HANA instance is running. Suggested minimum timeout: 60.

monitor (Master role)
    Reports whether the HANA database seems to be working in multi-state. It also needs to check the system replication status. Suggested minimum timeout: 700. Suggested interval: 60.

monitor (Slave role)
    Reports whether the HANA database seems to be working in multi-state. It also needs to check the system replication status. The secondary role's monitor interval has to be different from the master role. Suggested minimum timeout: 700. Suggested interval: 61.

validate-all
    Reports whether the parameters are valid. Suggested minimum timeout: 5.

meta-data
    Retrieves resource agent metadata (internal use only). Suggested minimum timeout: 5.

methods
    Suggested minimum timeout: 5.

RETURN CODES
The return codes are defined by the OCF cluster framework. Please refer to the OCF definition on the website mentioned below.
In addition, log entries are written, which can be scanned by using a pattern like "SAPHanaController.*RA.*rc=[1-7,9]" for errors. Regular operations might be found with "SAPHanaController.*RA.*rc=0".

EXAMPLES
* Below is an example configuration for a SAPHanaController multi-state resource in a HANA scale-out scenario.
    The HANA consists of two sites with five nodes each. An additional cluster node is used as majority maker for split-brain situations. In addition, a SAPHanaTopology clone resource is needed to make this work.

    primitive rsc_SAPHanaCon_SLE_HDB00 ocf:suse:SAPHanaController \
    op start interval="0" timeout="3600" \
    op stop interval="0" timeout="3600" \
    op promote interval="0" timeout="900" \
    op demote interval="0" timeout="320" \
    op monitor interval="60" role="Master" timeout="700" \
    op monitor interval="61" role="Slave" timeout="700" \
    params SID="SLE" InstanceNumber="00" PREFER_SITE_TAKEOVER="true" \
    DUPLICATE_PRIMARY_TIMEOUT="7200" AUTOMATED_REGISTER="true" \
    HANA_CALL_TIMEOUT="120"

    ms msl_SAPHanaCon_SLE_HDB00 rsc_SAPHanaCon_SLE_HDB00 \
    master-node-max="1" master-max="1" clone-node-max="1" interleave="true"

    location SAPHanaCon_not_on_majority_maker msl_SAPHanaCon_HAE_HDB00 -inf: vm-majority

* The following shows the filter for log messages set to the defaults.
This property should only be set if requested by support engineers. The default is sufficient for normal operation.

    property $id="SAPHanaSR" \
    hana_SLE_glob_filter="ra-act-dec-lpa"

* Search for log entries of the resource agent, show errors only:

    # grep "SAPHanaController.*RA.*rc=[1-7,9]" /var/log/messages

* Check for working NTP service:

    # ntpq -p

* Use of DUPLICATE_PRIMARY_TIMEOUT and Last Primary Timestamp (LPT) in case the primary node has been crashed completely. Typically on each side where the RA detects a running primary a time stamp is written to the node's attributes (last primary seen at time: lpt). If the timestamps ("last primary seen at") differ less than the DUPLICATE_PRIMARY_TIMEOUT than the RA could not automatically decide which of the two primaries is the better one.
    1. nodeA is primary and has a current time stamp, nodeB is secondary and has a secondary marker set:
    nodeA: 1479201695
    nodeB: 30 2. Now nodeA crashes and nodeB takes over:
    (nodeA: 1479201695)
    nodeB: 1479201700 3. A bit later nodeA comes back into the cluster:
    nodeA: 1479201695
    nodeB: 1479202000
    You see while nodeA keeps its primary down the old timestamp is kept. NodeB increases its timestamp on each monitor run. 4. After some more time (depending on the parameter DUPLICATE_PRIMARY_TIMEOUT)
    nodeA: 1479201695
    nodeB: 1479208895
    Now the time stamps differ >= DUPLICATE_PRIMARY_TIMEOUT. The algorithm defines nodeA now as "the looser" and depending on the AUTOMATED_REGISTER the nodeA will become the secondary. 5. NodeA would be registered:
    nodeA: 10
    nodeB: 1479208900 6. Some time later the secondary gets into sync
    nodeA: 30
    nodeB: 1479209100

* Use of DUPLICATE_PRIMARY_TIMEOUT and Last Primary Timestamp (LPT) in case the the database on primary node has been crashed, but the node is still alive. Typically on each side where the RA detects a running primary a time stamp is written to the node's attributes (last primary seen at time: lpt). If the timestamps ("last primary seen at") differ less than the DUPLICATE_PRIMARY_TIMEOUT than the RA could not automatically decide which of the two primaries is the better one.
    1. nodeA is primary and has a current time stamp, nodeB is secondary and has a secondary marker set:
    nodeA: 1479201695
    nodeB: 30 2. Now HANA on nodeA crashes and nodeB takes over:
    nodeA: 1479201695
    nodeB: 1479201700 3. As the cluster could be sure to properly stopped the HANA instance at nodeA it *immediately* marks the old primary to be a register candidate, if AUTOMATED_REGISTER is true:
    nodeA: 10
    nodeB: 1479201760 4. Depending on the AUTOMATED_REGISTER parameter the RA will also immediately regisiter the former primary to become the new secondary:
    nodeA: 10
    nodeB: 1479201820 5. And after a while the secondary gets in sync
    nodeA: 30
    nodeB: 1479202132

* Overview on maintenance procedure for HANA, the Linux cluster remains running.
    1. set the Linux cluster into maintenance mode (crm configure property maintenance-mode=true)
    2. perform the HANA maintenance, e.g. update to latest SPS
    3. set the SAPHanaController multi-state resource to unmanaged (crm resource unmanage ...)
    4. set the Linux cluster back into ready mode (crm configure property maintenance-mode=false)
    5. cleanup the SAPHanaController multi-state resource (crm resource cleanup ...)
    6. set the SAPHanaController multi-state resource back to managed (crm resource manage ...)

FILES

/usr/lib/ocf/resource.d/suse/SAPHanaController
        the resource agent

/usr/lib/ocf/resource.d/suse/SAPHanaTopology
        the also needed topology resource agent

/usr/sap/$SID/$InstanceName/exe
        default path for DIR_EXECUTABLE

/usr/sap/$SID/SYS/profile
        default path for DIR_PROFILE

REQUIREMENTS
For the current version of the SAPHanaController resource agent that comes with the software package SAPHanaSR-ScaleOut, the support is limited to the scenarios and parameters described in the respective manual page SAPHanaSR-ScaleOut(7).

SEE ALSO
ocf_suse_SAPHanaTopology(7) , ocf_heartbeat_IPaddr2(8) , SAPHanaSR-monitor(8) , SAPHanaSR-showAttr(8) , SAPHanaSR-ScaleOut(7) , ntp.conf(5) , stonith(8) , crm(8)
https://www.suse.com/products/sles-for-sap/resource-library/sap-best-practices.html ,
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-ocf-return-codes.html ,
http://scn.sap.com/community/hana-in-memory/blog/2014/04/04/fail-safe-operation-of-sap-hana-suse-extends-its-high-availability-solution ,
http://www.saphana.com/docs/DOC-2775 ,
http://scn.sap.com/docs/DOC-60334 ,
http://scn.sap.com/docs/DOC-65899

AUTHORS
F.Herschel, L.Pinne.

COPYRIGHT
(c) 2014 SUSE Linux Products GmbH, Germany.
(c) 2015-2017 SUSE Linux GmbH, Germany.
(c) 2018-2019 SUSE LLC
The resource agent SAPHanaController comes with ABSOLUTELY NO WARRANTY.
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html