SAPHanaSR_maintenance_examples(7)

NAME
SAPHanaSR_maintenance_examples - maintenance examples for SAPHana and SAPHanaController.

DESCRIPTION

Maintenance examples for SAPHana and SAPHanaController. Please see ocf_suse_SAPHana(7) respectively ocf_suse_SAPHanaController(7), SAPHanaSR.py(7), SAPHanaSrMultiTarget.py(7), SAPHanaSR-manageAttr(8) for more examples and read the REQUIREMENTS section below.

EXAMPLES

* Check status of Linux cluster and HANA system replication pair. This steps should be performed before doing anything with the cluster, and after something has been done. See also cs_show_saphanasr_status(8) and section REQUIREMENTS below.

    # cs_clusterstate -a
    # crm_mon -1r
    # crm configure show | grep cli-
    # SAPHanaSR-showAttr
    # cs_clusterstate -i

* Watch status of HANA cluster resources and system replication. This might be conveniant when performing administrative actions or cluster tests. It does not replace the afore mentioned checks. See also cs_show_saphanasr_status(8).

    # watch -n 9 "(crm_mon -1r;SAPHanaSR-showAttr;cs_clusterstate -i)|egrep -v'(^$|configured|###)'"

* Initiate an administrative take-over of the HANA primary from one node to the other by using the Linux cluster. This procedure does not work for scale-out. If the cluster should also register the former primary as secondary, AUTOMATED_REGISTER="true" is needed. Before the take-over will be initiated, the status of the Linux cluster and the HANA system replication has to be checked. The take-over should be initiated as forced migration of the multi-state SAPHana resource.
Not working: Regular migration, migration of IP address, migration of primitive SAPHana resource, setting primary node standby.
After take-over of the primary has been finished, the migration rule has to be deleted. If AUTOMATED_REGISTER="true" is set, finally the former primary will be registered as secondary, once the migration rule has been deleted.

    # crm_mon -1r
    # SAPHanaSR-showAttr
    # crm configure show | grep cli-
    # cs_clusterstate -i
    # crm resource move msl_SAPHana_SLE_HDB10 force
    # cs_clusterstate -i
    # SAPHanaSR-showAttr
    # crm resource clear msl_SAPHana_SLE_HDB10
    # SAPHanaSR-showAttr
    # cs_clusterstate -i

Note: Former versions of the Linux cluster used "migrate" instead of "move" and "unmigrate" instead of "clear".

* Perform an SAP HANA take-over by using SAP tools. The procedures is described here for scale-out. It works for scale-up as well. The status of HANA databases, system replication and Linux cluster has to be checked. The SAP HANA resources are set into maintenance, an sr_takeover is performed, the old primary is registered as new secondary. Therefor the correct secondary site name has to be used, see later example. Finally the SAP HANA resources are given back to the Linux cluster. See also section REQUIREMENTS below and later example on determing the correct site name.

    1. On either node
    # crm_mon -1r
    # SAPHanaSR-showAttr
    # crm configure show | grep cli-
    # cs_clusterstate -i
    If everything looks fine, proceed.
    # crm resource maintenance msl_SAPHana_SLE_HDB10
    # crm_mon -1r
    2. On the HANA primary master nameserver (e.g. node11)
    # su - sleadm
    ~> sapcontrol -nr 10 -function StopSystem HDB
    ~> sapcontrol -nr 10 -function GetSystemInstanceList

    Only proceed after you made sure the HANA primary is down!

    3. On the HANA secondary master nameserver (e.g. node21)
    # su - sleadm
    ~> hdbnsutil -sr_takeover
    ~> HDBsettings.sh systemReplicationStatus.py; echo RC:$?
    ~> HDBsettings.sh landscapeHostConfiguration.py; echo RC:$?
    If everything looks fine, proceed.
    4. On the former HANA primary master nameserver, now future secondary master nameserver (e.g. node11)
    ~> hdbnsutil -sr_register --remoteHost=node21 --remoteInstance=10 --replicationMode=sync --name=site2 --operationMode=logreplay
    ~> sapcontrol -nr 10 -function StartSystem HDB
    ~> exit
    5. On the new HANA primary master nameserver (e.g. node21)
    ~> HDBsettings.sh systemReplicationStatus.py; echo RC:$?
    ~> HDBsettings.sh ./landscapeConfigurationStatus.py; echo RC:$?
    ~> exit
    If everything looks fine, proceed.
    6. On either node
    # cs_clusterstate -i
    # crm resource refresh msl_SAPHana_SLE_HDB10
    # crm resource maintenance msl_SAPHana_SLE_HDB10 off
    # SAPHanaSR-showAttr
    # crm_mon -1r
    # cs_clusterstate -i

* Check the two site names that are known to the Linux cluster. This is useful in case AUTOMATED_REGISTER is not yet set. In that case a former primary needs to be registered manually with the former site name as new secondary. The point is finding the site name that already is in use by the Linux cluster. That exact site name has to be used for registration of the new secondary. See also REQUIREMENTS of SAPHanaSR(7) and SAPHanaSR-ScaleOut(7).
In this example, node is suse11 on the future secondary site to be registered. Remote HANA Master Name Server is suse21 on current primary site. Lowercase-SID is ha1.

    # crm configure show suse11 suse21
    # crm configure show SAPHanaSR | grep hana_ha1_site_mns
    # ssh suse21
    # su - ha1adm -c "hdbnsutil -sr_state; echo rc: $?"
    # exit

* Manually start the HANA primary if only one site is available. This might be necessary in case the cluster can not detect the status of both sites. This is an advanced task.

Before doing this, make sure HANA is not primary on the other site!

    1. Start the cluster on remaining nodes.
    2. Wait and check for cluster is running, and in status idle.
    3. Become sidadm, and start HANA manually.
    4. Wait and check for HANA is running.
    5. In case the cluster does not promote the HANA to primary, instruct the cluster to migrate the IP address to that node.
    6. Wait and check for HANA has been promoted to primary by the cluster.
    7. Remove the migration rule from the IP address.
    8. Check if cluster is in status idle.
    9. You are done, for now.
    10. Please bring back the other node and register that HANA as soon as possible. If the HANA primary stays alone for too long, the log area will fill up.

* Overview on maintenance procedure for HANA, the Linux cluster remains running, on pacemaker-1.0. See also section REQUIREMENTS below.

    1. Check if everything looks fine, see above.
    2. Set the Linux cluster into maintenance mode.
    # crm configure property maintenance-mode=true
    3. Perform the HANA maintenance, e.g. update to latest SPS.
    4. Set the SAPHanaController m/s resource to unmanaged.
    # crm resource unmanage <m/s-resource>
    5. Set the Linux cluster back into ready mode.
    # crm configure property maintenance-mode=false
    6. Cleanup the SAPHanaController m/s resource.
    # crm resource cleanup <m/s-resource> node <node>
    7. Set the SAPHanaController m/s resource back to managed.
    # crm resource manage <m/s-resource>
    8. Check if everything looks fine, see above.

Note: The YaST module hana_updater does something similar, combined with an administrative take-over.

On pacemaker-2.0 respectively do the following.

    1. Check if everything looks fine, see above.
    2. Set the SAPHanaController multi-state resource into maintenance mode.
    # crm resource maintenance msl_SAPHanaCon_SLE_HDB10 on
    3. Perform the HANA maintenance, e.g. update to latest SPS.
    4. Tell the cluster to forget about HANA status and to reprobe the resources.
    # crm resource refresh msl_SAPHanaCon_SLE_HDB10
    5. Set the SAPHanaController multi-state resource back to managed.
    # crm resource maintenance msl_SAPHanaCon_SLE_HDB10 off
    6. Remove the meta attribute from CIB, optional.
    # crm resource meta msl_SAPHanaCon_SLE_HDB10 delete maintenance
    7. Check if everything looks fine, see above.

The two procedures must not be mixed. If the procedure for pacemaker-1.0 has been used, left-over maintenance attribute have to be removed from the CIB before proceeding with the new procedure for pacemaker-2.0.

* Overview on maintenance procedure for Linux, HANA remains running, on pacemaker-2.0. See also section REQUIREMENTS below and example on checking status of HANA and cluster above.

    1. Check status of Linux cluster and HANA, see above.
    2. Set the Linux cluster into maintenance mode, on either node.
    # crm maintenance on
    3. Stop Linux Cluster on all nodes. Make sure to do that on all nodes.
    # systemctl stop pacemaker
    4. Perform Linux maintenance.
    5. Start Linux cluster on all nodes. Make sure to do that on all nodes.
    # systemctl start pacemaker
    6. Let Linux cluster detect status of HANA resource, on either node.
    # crm resource refresh cln_...
    # crm resource refresh msl_...
    7. Set cluster ready for operations, on either node.
    # crm maintenance off
    8. Check status of Linux cluster and HANA, see above.

* Overview on update procedure for the SAPHanaSR and SAPHanaSR-ScaleOut package. This procedure can be used to update RAs, HANA HADR provider hook scripts and related tools while HANA and Linux cluster stay online. See also SAPHanaSR-manageAttr(8) for details on reloading the HANA HADR provider.

    1. Check status of Linux cluster and HANA, see above.
    2. Set resources SAPHana (or SAPHanaController) and SAPHanaTopology to maintenance.
    3. Update RPM on all cluster nodes.
    4. Reload HANA HADR provider hook script on both sites.
    5. Refresh resources SAPHana (or SAPHanaController) and SAPHanaTopology.
    6. Set resources SAPHana (or SAPHanaController) and SAPHanaTopology from maintenance to managed.
    7. Check status of Linux cluster and HANA, see above.

* Remove left-over maintenance attribute from overall Linux cluster. This could be done to avoid confusion caused by different maintenance procedures. See above overview on maintenance procedures whith running Linux cluster. Before doing so, check for cluster attribute maintenance-mode="false".

    # SAPHanaSR-showAttr
    # crm_attribute --query -t crm_config -n maintenance-mode
    # crm_attribute --delete -t crm_config -n maintenance-mode
    # SAPHanaSR-showAttr

* Remove left-over standby attribute from Linux cluster nodes. This could be done to avoid confusion caused by different maintenance procedures. See above overview on maintenance procedures whith running Linux cluster. Before doing so for all nodes, check for node attribute standby="off" on all nodes.

    # SAPHanaSR-showAttr
    # crm_attribute --query -t node -N node1 -n standby
    # crm_attribute --delete -t node -N node1 -n standby
    # SAPHanaSR-showAttr

* Remove left-over maintenance attribute from resource. This should usually not be needed. See above overview on maintenance procedures whith running Linux cluster.

    # SAPHanaSR-showAttr
    # crm_resource --resource cln_SAPHanaTop_HA1_HDB00 --delete-parameter maintenance --meta
    # SAPHanaSR-showAttr

* Manually update global site attribute. In rare cases the global site attribute hana_<sid>_glob_prim or hana_<sid>_glob_sec is not updated automatically after successful take-over, while all other attributes are updated correctly. The global site attribute stays outdated even after the cluster has been idle for a while. In this case, that site attribute could be updated manually. Make sure everything else is fine and just the global site attribute has not been updated. Updating hana_<sid>_glob_sec for SID HA1 with site name VOLKACH:

    # crm configure show SAPHanaSR
    # crm_attribute --type crm_config --name hana_ha1_glob_sec --update=VOLKACH
    # crm configure show SAPHanaSR

* Upgrade scale-out srHook attribute from old-style to multi-target. As final result of this upgrade, the RAs and hook script are upgraded from old-style to multi-target. Further the Linux cluster's old-style global srHook attribute hana_${sid}_glob_srHook is replaced by site-aware attributes hana_${sid}_site_srHook_${SITE}. New auxiliary attributes are introduced. The complete procedure and related requirements are described in detail in manual page SAPHanaSR-manageAttr(8).
The procedure at a glance:

    a. Initially check if everything looks fine.
    b. Set Linux cluster resources SAPHanaController and SAPHanaTopology into maintenance.
    c. Install multi-target aware SAPHanaSR-ScaleOut package on all nodes.
    d. Adapt sudoers permission on all nodes.
    e. Replace HANA HADR provider configuration on both sites.
    f. Reload HANA HADR provider hook script on both sites.
    g. Check Linux cluster and HANA HADR provider for matching defined upgrade entry state.
    h. Migrate srHook attribute from old-style to multi-target.
    i. Check Linux cluster for matching defined upgrade target state.
    j. Set Linux cluster resources SAPHanaController and SAPHanaTopology from maintenance to managed.
    k. Optionally connect third HANA site via system replication outside of the Linux cluster.
    l. Finally check if everything looks fine.

FILES

REQUIREMENTS
* For the current version of the resource agents that come with the software packages SAPHanaSR and SAPHanaSR-ScaleOut, the support is limited to the scenarios and parameters described in the respective manual pages SAPHanaSR(7) and SAPHanaSR-ScaleOut(7).

* Be patient. For detecting the overall HANA status, the Linux cluster needs a certain amount of time, depending on the HANA and the configured intervalls and timeouts.

* Before doing anything, always check for the Linux cluster's idle status, left-over migration constraints, and resource failures as well as the HANA landscape status, and the HANA SR status.

* Maintenance attributes for cluster, nodes and resources must not be mixed.

* The Linux cluster needs to be up and running to allow HA/DR provider events being written into CIB attributes. The current HANA SR status might differ from CIB srHook attribute after Linux cluster maintenance.

* Manually activating an HANA primary, like start of HANA primary or take-over outside the cluster creates risk of a duplicate-primary situation. The user is responsible for data integrity, particularly when activating an HANA primary.

BUGS
In case of any problem, please use your favourite SAP support process to open a request for the component BC-OP-LNX-SUSE. Please report any other feedback and suggestions to feedback@suse.com.

SEE ALSO
ocf_suse_SAPHanaTopology(7) , ocf_suse_SAPHana(7) , ocf_suse_SAPHanaController(7) , SAPHanaSR-monitor(8) , SAPHanaSR-showAttr(8) , SAPHanaSR(7) , SAPHanaSR-ScaleOut(7) , SAPHanaSR-manageAttr(8) , cs_clusterstate(8) , cs_show_saphanasr_status(8) , cs_wait_for_idle(8) , crm(8) , crm_simulate(8) , crm_mon(8) , crm_attribute(8)
https://www.suse.com/products/sles-for-sap/resource-library/sap-best-practices.html ,
https://www.suse.com/support/kb/doc/?id=000019253 ,
https://www.suse.com/support/kb/doc/?id=000019207 ,
https://www.suse.com/support/kb/doc/?id=000019142 ,
https://www.suse.com/c/how-to-upgrade-your-suse-sap-hana-cluster-in-an-easy-way/
https://help.sap.com/doc/eb75509ab0fd1014a2c6ba9b6d252832/1.0.12/en-US/SAP_HANA_Administration_Guide_en.pdf

AUTHORS
F.Herschel, L.Pinne.

COPYRIGHT
(c) 2017-2018 SUSE Linux GmbH, Germany.
(c) 2019-2021 SUSE LLC
This maintenance examples are coming with ABSOLUTELY NO WARRANTY.
For details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html