susChkSrv.py(7)

NAME
susChkSrv.py - Provider for SAP HANA srHook method srServiceStateChanged().

DESCRIPTION
susChkSrv.py can be used to provide a script for the SAP HANA srHook method srServiceStateChanged(). The SAP HANA nameserver provides a Python-based API ("HA/DR providers"), which is called at important points of the host auto-failover and system replication takeover processes. These so called hooks can be used for arbitrary operations that need to be executed. The method srServiceStateChanged() is called when HANA processes are failing, starting or stopping. Purpose of susChkSrv.py is to detect failing HANA indexserver processes and trigger a fast takeover to the secondary site. With regular configuration of an HANA database, the resource agent (RA) for HANA in a Linux cluster does not trigger a takeover to the secondary site when:
- A software failure causes one or more HANA processes to be restarted in place by the HANA daemon (hdbdaemon).
- A hardware error (e.g. SIGBUS from an uncorrectable memory error) causes the indexserver to restart locally.
See also SAPHanaSR(7) or SAPHanaSR-ScaleOut(7). The hook script susChkSrv.py is called on any srServiceStateChanged() event. The script checks for
 'isIndexserver and serviceRestart and serviceWasActiveBefore and hostActive and databaseActive'. If it finds the correct entries, it executes the predefined action. As soon as the HANA landscapeHostConfiguration status changes to 1, the Linux cluster will take action. The action depends on HANA system replication status and the RA´s configuration parameters PREFER_SITE_TAKEOVER and AUTOMATED_REGISTER, see manual page ocf_suse_SAPHana(7) or ocf_suse_SAPHanaController(7). Customising of HANA daemon timeout parameters might be needed for adapting the solution to a given environment. Please refer to SAP HANA documentation. This hook script needs to be installed, configured and activated on all HANA nodes.

SUPPORTED PARAMETERS
* The "HA/DR providers" API accepts the following parameters for the ha_dr_provider_suschksrv section in global.ini:

[ha_dr_provider_suschksrv]

provider = susChkSrv
    Mandatory. Must not be changed.

path = [ /usr/share/SAPHanaSR | /usr/share/SAPHanaSR-ScaleOut ]
    Mandatory. Depending on path delivered within RPM package. Please change only if requested.

execution_order = [ INTEGER ]
    Mandatory. Order might depend on other hook scripts.

action_on_lost = [ ignore | stop | kill | fence ]
    Action to be processed when a lost indexserver is identified.
    - ignore: do nothing, just write to tracefiles.
    - stop: do 'sapcontrol ... StopSystem'. If this is combined with SAPHana or SAPHanaController RA parameter 'AUTOMATED_REGISTER=true', HANA needs to release all OS resources prior to the automated registering. See also manual page ocf_suse_SAPHana(7) or ocf_suse_SAPHanaController(7).
    - kill: do 'HDB kill-<signal>'. The signal can be defined by parameter 'kill_signal'. If this is combined with SAPHana or SAPHanaController RA parameter 'AUTOMATED_REGISTER=true', HANA needs to release all OS resources prior to the automated registering.
    - fence: do 'crm node fence <host>'. This needs a Linux cluster STONITH
     method and sudo permission. This action is primarily meant for scale-up. If it happens on a scale-out worker node, the remaining master needs to time out before the Linux cluster will react.
    Optional. Default is ignore.

kill_signal = [ INTEGER ]
    Signal to be used with 'HDB kill-<signal>'.
    Optional. Default is 9.

stop_timeout = [ INTEGER ]
    How many seconds to wait for 'sapcontrol ... StopSystem' to return? Should be greater than value of daemon.ini parameter 'forcedtimeout'.
    Optional. Default is 20 seconds.

* The "HA/DR providers" API accepts the following parameter for the trace section in globnal.ini:

[trace]

ha_dr_suschksrv = [ info | debug ]
    Optional. Default is info. Will be added automatically if not set.

* The HANA daemon TODO for the daemon section of daemon.ini:

[daemon]

terminationtimeout = [ INTEGER ]
    TODO Should be 45000.
    Optional. Timeout in milliseconds. Default is TODO.

forcedterminationtimeout = [ INTEGER ]
    TODO Should be 15000.
    Optional. Timeout in milliseconds. Default is TODO.

* The HANA daemon TODO for the indexserver.<tenant> section of daemon.ini:

[indexserver.<tenant>]

gracetime = [ INTEGER ]
    TODO Should be 6000.
    Optional. Timeout in milliseconds. Default is 2000.

RETURN CODES
0 Successful program execution.
>0 Usage, syntax or execution errors.

EXAMPLES

* Example for minimal entry in SAP HANA scale-up global configuration /hana/shared/$SID/global/hdb/custom/config/global.ini

In case of a failing indexserver, the event is logged. No action is performed. The section ha_dr_provider_suschksrv is needed on all HANA nodes. The HANA has to be stopped before the file can be changed.

    [ha_dr_provider_suschksrv]
    provider = susChkSrv
    path = /usr/share/SAPHanaSR/
    execution_order = 3

* Example for entry in SAP HANA scale-up global configuration /hana/shared/HA1/global/hdb/custom/config/global.ini

In case of a failing indexserver, the complete node will get fenced by calling 'crm node fence ...'. Unlike fence actions from inside the Linux HA cluster, this hook script fence action will be issued even when the cluster is in maintenance and stonith is temporarily disabled. The fence request then is queued until the cluster is set back into active state. See example below for removing queued fence actions.
The section ha_dr_provider_suschksrv is needed on all HANA nodes. The HANA has to be stopped before the file can be changed. Alterntively use SAPHanaSR-manageProvider for applying an HA/DR provider hook configuration, see manual page SAPHanaSR-manageProvider(8).

    [ha_dr_provider_suschksrv]
    provider = susChkSrv
    path = /usr/share/SAPHanaSR/
    execution_order = 2
    action_on_lost = fence

* Example for entry in SAP HANA scale-out global configuration /hana/shared/HA1/global/hdb/custom/config/global.ini

In case of a failing indexserver, the HANA instance will be stopped with
 'sacpcontrol ... StopSystem'. HANA timeouts might be adapted to speed up the stop. If this is combined with SAPHanaController parameter 'AUTOMATED_REGISTER=true', HANA needs to release all OS resources prior to the automated registering.
The hook script should wait maximum 25 seconds on the sapcontrol command to return.
The section ha_dr_provider_suschksrv is needed on all HANA nodes. The HANA has to be stopped before the file can be changed.
Note: HANA scale-out is supported only with exactly one master nameserver. No HANA host auto-failover.

    [ha_dr_provider_suschksrv]
    provider = susChkSrv
    path = /usr/share/SAPHanaSR-ScaleOut/
    execution_order = 2
    action_on_lost = stop
    stop_timeout = 25

* Example for entry in SAP HANA daemon configuration /hana/shared/HA1/global/hdb/custom/config/daemon.ini

TODO Example SID is HA1, tenant is HA1.
The sections daemon and indexserver.HA1 are needed on all HANA nodes. The HANA has to be stopped before the file can be changed.

    [daemon]
    terminationtimeout = 45000
    forcedterminationtimeout = 15000

    [indexserver.HA1]
    gracetime = 6000

* Example for sudo permissions in /etc/sudoers.d/SAPHanaSR .

SID is HA1. See also manual page SAPHanaSR-hookHelper(8).

    # SAPHanaSR and SAPHanaSR-ScaleOut need for susChkSrv
    ha1adm ALL=(ALL) NOPASSWD: /usr/sbin/SAPHanaSR-hookHelper --sid=HA1 --case=fenceMe

* Example for looking up the sudo permission for the hook script.

All related files (/etc/sudoers and /etc/sudoers.d/*) are scanned. Example SID is HA1.

    # sudo -U ha1adm -l | grep "NOPASSWD.*/usr/sbin/SAPHanaSR-hookHelper"

* Example for checking the HANA tracefiles for srServiceStateChanged() events.

Example SID is HA1. To be executed on the respective HANA master nameserver.
If the HANA nameserver process is killed, in some cases hook script actions do not make it into the nameserver tracefile. In such cases the hook script´s own tracefile might help, see respective example.

    # su - ha1adm
    ~> cdtrace
    ~> grep susChkSrv.*srServiceStateChanged nameserver_*.trc
    ~> grep -C2 Executed.*StopSystem nameserver_*.trc

* Example for checking the HANA tracefiles for when the hook script has been loaded.

Example SID is HA1. To be executed on both sites' master nameservers.

    # su - ha1adm
    ~> cdtrace
    ~> grep HADR.*load.*susChkSrv nameserver_*.trc
    ~> grep susChkSrv.init nameserver_*.trc

* Example for checking the hook script tracefile for actions.

Example SID is HA1. To be executed on all nodes. All incidents are logged on the node where it happens.

    # su - ha1adm
    ~> cdtrace
    ~> egrep '(LOST:|STOP:|START:|DOWN:|init|load|fail)' nameserver_suschksrv.trc

* Example for checking the hook script tracefile for node fence actions.

Example SID is HA1. To be executed on both sites' master nameservers. See also manual page SAPHanaSR-hookHelper(8).

    # su - ha1adm
    ~> cdtrace
    ~> grep fence.node nameserver_suschksrv.trc

* Example for revoking a queued fence request from the Linux cluster.

This could be done if an HANA indexserver failure has triggerd an node fence action while the Linux cluster is in maintenance. Before revoking a fence request, be sure it has been issued by the HA/DR provider hook script. See example above for checking the hook script tracefile for node fence actions. Example node is node2. To be executed on that node. See also manual pages SAPHanaSR-hookHelper(8) and crm_attribute(8).
Note: This removes the node attribute terminate=true from the Linux cluster CIB. It does not touch any fencing device.

    # grep fenced:.termination.was.requested /var/log/pacemaker/pacemaker.log
    # crm_attribute -t status -N 'node2' -D -n terminate
    # crm_attribute -t status -N 'node2' -G -n terminate

FILES

/usr/share/SAPHanaSR/susChkSrv.py or /usr/share/SAPHanaSR-ScaleOut/susChkSrv.py
    the hook provider, delivered with the RPM

/usr/sbin/SAPHanaSR-hookHelper
    the external script for node fencing

/etc/sudoers, /etc/sudoers.d/*
    the sudo permissions configuration

/hana/shared/$SID/global/hdb/custom/config/global.ini
    the on-disk representation of HANA global system configuration

/hana/shared/$SID/global/hdb/custom/config/daemon.ini
    the on-disk representation of HANA daemon configuration

/usr/sap/$SID/HDB$nr/$HOST/trace
    path to HANA tracefiles

/usr/sap/$SID/HDB$nr/$HOST/trace/nameserver_suschksrv.trc
    HADR provider hook script tracefile

REQUIREMENTS
1. SAP HANA 2.0 SPS05 or later provides the HA/DR provider hook method srServiceStateChanged() with needed parameters.

2. No other HADR provider hook script should be configured for the srServiceStateChanged() method. Hook scripts for other methods, provided in SAPHanaSR and SAPHanaSR-ScaleOut, can be used in parallel to susChkSrv.py, if not documented contradictingly.

3. The user ${sid}adm needs execution permission as user root for the command SAPHanaSR-hookHelper.

4. The hook provider needs to be added to the HANA global configuration, in memory and on disk (in persistence).

5. HANA daemon timeout TODO

6. The hook script runs in the HANA nameserver. It runs on the node where the event srServiceStateChanged() occurs.

7. HANA scale-out is supported only with exactly one master nameserver. HANA host auto-failover is not supported. Thus no standby nodes.

8. A Linux cluster STONITH method for all nodes is needed, particularly if susChkSrv.py parameter 'action_on_lost=fence' is set.

9. If susChkSrv.py parameter 'action_on_lost=stop' is set and the RA SAPHana or SAPHanaController parameter 'AUTOMATED_REGISTER=true' is set, it depends on HANA to release all OS resources prior to the registering attempt.

10. If the hook provider should be pre-compiled, the particular Python version that comes with SAP HANA has to be used.

BUGS
In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.

SEE ALSO
SAPHanaSR(7) , SAPHanaSR-ScaleOut(7) , SAPHanaSR.py(7) , ocf_suse_SAPHanaTopology(7) , ocf_suse_SAPHana(7) , ocf_suse_SAPHanaController(7) , SAPHanaSR-hookHelper(8) , SAPHanaSR-manageProvider(8) , crm(8) , crm_attribute(8) , python3(8) ,
https://help.sap.com/docs/SAP_HANA_PLATFORM?locale=en-US
https://help.sap.com/docs/SAP_HANA_PLATFORM/42668af650f84f9384a3337bcd373692/e2064c4aa47f443ab6a107f9ab7f5edd.html?version=2.0.01
https://help.sap.com/docs/SAP_HANA_PLATFORM/6b94445c94ae495c83a19646e7c3fd56/5df2e766549a405e95de4c5d7f2efc2d.html?locale=en-US
SAP note 2177064

AUTHORS
A.Briel, F.Herschel, L.Pinne.

COPYRIGHT
(c) 2022 SUSE LLC
susChkSrv.py comes with ABSOLUTELY NO WARRANTY.
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html