SAPHanaSR-ScaleOut(7)

NAME
SAPHanaSR-ScaleOut_basic_cluster - SAP HANA System Replication scale-out basic cluster configuration.

DESCRIPTION
The SAP HANA System Replication scale-out scenario needs a certain basic cluster configuration. Besides this necessary settings, some additional configurations might match specific needs. * CRM Basics no-quorum-policy = freeze The crm basic parameter no-quorum-policy defines how the cluster should act in case of quorum loss. With more than two nodes, the cluster must not ignore the quorum loss. For SAPHanaSR-ScaleOut, an odd number of nodes is required. Setting no-quorum-policy to 'freeze' won't allow the partition to shoot any other node when it doesn't have quorum. Cluster will not be able to add and start new resources, but running will stay alive. If the cluster uses disk-less SBD, the no-quorum-policy 'suicide' is required. default-resource-stickiness = 1000 The crm basic parameter default-resource-stickiness defines the 'stickiness' score a resource gets on the node where it is currently running. This prevents the cluster from moving resources around whithout an urgent need during a cluster transition. The correct value depends on number of resources, colocation rules and resource groups. Particularly additional groups colocated to the HANA primary master resource can affect cluster decisions. Too high value might prevent not only unwanted but also useful actions. This is because SAPHanaSR uses an internal scoring table for placing the HANA roles on the right nodes. concurrent-fencing = true The crm basic parameter concurrent-fencing allows the cluster to fence more than one node at a time. This helps to reduce the time needed for a take over in case a whole data center is lost. If nodes are fenced one by one, the time needed would be equal to the number of nodes * stonith timeout. With concurrent-fencing enabled the time needed is in the range of 2 * stonith timeout, independent of the number of nodes. See also pcmk_action_limit below. failure-timeout = 86400s The crm basic parameter failure-timeout defines how long failed actions will be kept in the CIB. After that time the failure record will be deleted. See also migration-threshold below. migration-threshold = 50 The crm basic parameter migration-threshold defines how many errors on a resource can be detected before this resource will be migrated to another node. See also failure-timeout. record-pending = false The op_default record-pending defines, whether the intention of an action upon the resource is recorded in the Cluster Information Base (CIB). Setting this parameter to 'true' allows the user to see pending actions like 'starting' and 'stopping' in crm_mon and Hawk.

* SBD STONITH Basics pcmk_action_limit = -1 The sbd stonith parameter pcmk_action_limit defines the maximum number of concurrent fencing actions. It allows parallel fencing of multiple nodes. A value of '-1' means virtually unlimited. See also concurrent-fencing above. pcmk_delay_max = 1s The sbd stonith parameter pcmk_delay_max defines an upper limit for waiting before a fencing/stonith request will be triggerd. This parameter should prevent the cluster from unwanted double fencing in case of spilt-brain. A value around 30 seconds required in two-node clusters. It is not needed in usual SAPHanaSR-ScaleOut setups.

EXAMPLES
* crm basic configuration Below is an example crm basic configuration for SAPHanaSR-ScaleOut. Shown are specific parameters which are needed. Some general parameters are left out.
This example has been taken from SLE-HA 11 SP4 with disk-based SBD:

    property $id="cib-bootstrap-options" \
     expected-quorum-votes="31" \
     no-quorum-policy="freeze" \
     dc-deadtime="20s" \
     default-resource-stickiness="1000" \
     stonith-enabled="true" \
     stonith-timeout="180s" \
     concurrent-fencing="true"
    rsc_defaults $id="rsc_defaults-options" \
     resource-stickiness="1000" \
     migration-threshold="50" \
     failure-timeout="86400s"
    op_defaults $id="op-options" \
     record-pending="false" \
     timeout="600s"
    op_defaults $id="op_defaults-options" \
     timeout="300s"

The following example is for SLE-HA 12 SP4 and 15 SP1 with disk-based SBD:

    property cib-bootstrap-options: \
     have-watchdog=true \
     cluster-infrastructure=corosync \
     cluster-name=hacluster \
     stonith-enabled=true \
     placement-strategy=balanced \
     stonith-timeout=180 \
     no-quorum-policy=freeze \
     concurrent-fencing=true
    rsc_defaults rsc-options: \
     resource-stickiness=120 \
     migration-threshold=50 \
     failure-timeout=86400
    op_defaults op-options: \
     timeout=600 \
     record-pending=true

* crm SBD stonith configuration To complete the SBD setup, it is necessary to activate SBD as STONITH/fencing mechanism in the CIB. The SBD is normally used for SAPHanaSR-ScaleOut instead of any other fencing/stonith mechanism. Example for a basic disk-based SBD resource:

    primitive rsc_stonith_sbd stonith:external/sbd \
     params pcmk_action_limit="-1" pcmk_delay_max="1"

* crm simple IP address resource configuration Let the Linux cluster manage one IP address and move that address along with the HANA primary master nameserver.

    primitive rsc_ip_SLE_HDB00 IPAddr2 \
     op monitor interval=10s timeout=20s \
     params ip=192.168.178.188
    colocation col_ip_with_SLE_HDB00 \
     2000: rsc_ip_SLE_HDB00:Started msl_SAPHanaCon_SLE_HDB00:Master

* crm IP address for active/active read-enabled resource configuration Let the Linux cluster manage an additional IP address and move that address along with the HANA secondary master nameserver.
Note: This example works for two-node HANA scale-out.

    primitive rsc_ip_ro_SLE_HDB00 IPAddr2 \
     op monitor interval=10s timeout=20s \
     params ip=192.168.178.199
    colocation col_ip_ro_with_secondary_SLE_HDB00 \
     2000: rsc_ip_ro_SLE_HDB00:Started msl_SAPHanaCon_SLE_HDB00:Slave
    location loc_ip_ro_not_master_SLE_HDB00 \
     rsc_ip_ro_SLE_HDB00 \
     rule -inf: hana_sle_roles ne master1:master:worker:master

* crm grouped IP address resource configuration Let the Linux cluster manage one IP address and move that address along with the HANA primary master nameserver. An auxiliary resource is needed for specific public cloud purpose.

    primitive rsc_ip_SLE_HDB00 IPAddr2 \
     op monitor interval=10s timeout=20s \
     params ip=192.168.178.188 cidr_netmask=32
    primitive rsc_lb_SLE_HDB00 azure-lb \
     params port=62502
    group grp_ip_SLE_HDB00 rsc_lb_SLE_HDB00 rsc_ip_SLE_HDB00 \
     meta resource-stickiness=1
    colocation col_ip_with_SLE_HDB00 \
     8000: grp_ip_SLE_HDB00:Started msl_SAPHanaCon_SLE_HDB00:Master

* crm NFS check resource configuration In case of NFS failure, HANA might stop working but the Linux cluster might not take action. To solve this, a dummy filesystem resource could be added. If this filesystem reports monitor failures, the node gets fenced and a take-over is initiated.
Note: Understand the impact before implementing.

    primitive rsc_fs_check_SLE_HDB00 Filesystem \
     params device="/hana/shared/SLE/check/" \
     directory="/hana/shared/check/" fstype=nfs4 \
    options="bind,defaults,rw,hard,proto=tcp,intr,noatime,vers=4,lock" \
     op monitor interval=120 timeout=120 on-fail=fence \
     op_params OCF_CHECK_LEVEL=20 \
     op start interval=0 timeout=120 \
     op stop interval=0 timeout=120
    clone cln_fs_check_SLE_HDB00 rsc_fs_check_SLE_HDB00 \
     meta clone-node-max=1 interleave=true
    location fs_check_not_on_majority_maker \
     cln_fs_check_SLE_HDB00 -inf: vm-majority

* check how resource stickiness affects promotion scoring SAPHanaSR uses an internal scoring table. The promotion scores for HANA primary and secondary master are in a certain range. The scores used by the Linux cluster should be in the same range.

    # SAPHanaSR-showAttr | grep master.:master
    # crm_simulate -Ls | grep promotion

* cleanup SDB stonith resource after write failure In rare cases the SBD stonith resource failes writing to the block device. After the root cause has been found and fixed, the failure message can be cleaned.

    # stonith_admin --cleanup --history=<originator_node>

BUGS
In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.

SEE ALSO
ocf_suse_SAPHanaTopology(7) , ocf_suse_SAPHanaController(7) , ocf_heartbeat_IPAddr2(7) , ocf_heartbeat_Filesystem(7) , sbd(8) , stonith_sbd(7) , stonith_admin(8) , crm_no_quorum_policy(7) , crm(8) , crm_simulate(8) , SAPHanaSR-ScaleOut(7) , SAPHanaSR-showAttr(7) , corosync.conf(5) , votequorum(5) , nfs(5) , mount(8) , ha_related_suse_tids(7) ,
https://documentation.suse.com/sbp/all/?context=sles-sap ,
https://documentation.suse.com/sles-sap/ ,
https://www.suse.com/support/kb/ ,
https://www.clusterlabs.org

AUTHORS
F.Herschel, L.Pinne.

COPYRIGHT
(c) 2018 SUSE Linux GmbH, Germany.
(c) 2019-2021 SUSE LLC
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html