= Configure STONITH =

////
We prefer [[ch-stonith]], but older versions of asciidoc dont deal well
with that construct for chapter headings
////
anchor:ch-stonith[Chapter 13, STONITH]
indexterm:[STONITH, Configuration]

== What Is STONITH ==

STONITH is an acronym for Shoot-The-Other-Node-In-The-Head and it
protects your data from being corrupted by rogue nodes or concurrent
access.

Just because a node is unresponsive, this doesn't mean it isn't
accessing your data. The only way to be 100% sure that your data is
safe, is to use STONITH so we can be certain that the node is truly
offline, before allowing the data to be accessed from another node.


STONITH also has a role to play in the event that a clustered service
cannot be stopped. In this case, the cluster uses STONITH to force the
whole node offline, thereby making it safe to start the service
elsewhere.

== What STONITH Device Should You Use ==

It is crucial that the STONITH device can allow the cluster to
differentiate between a node failure and a network one.

The biggest mistake people make in choosing a STONITH device is to
use remote power switch (such as many on-board IMPI controllers) that
shares power with the node it controls. In such cases, the cluster
cannot be sure if the node is really offline, or active and suffering
from a network fault.

Likewise, any device that relies on the machine being active (such as
SSH-based "devices" used during testing) are inappropriate.

== Differences of STONITH Resources ==

Stonith resources are somewhat special in Pacemaker.

In previous versions, only "running" resources could be used by
Pacemaker for fencing.  This requirement has been relaxed to allow
other parts of the cluster (such as resources like DRBD) to reliably
initiate fencing. footnote:[Fencing a node while Pacemaker was moving
stonith resources around would otherwise fail]

Now all nodes have access to their definitions and instantiate them
on-the-fly when needed, however preference is given to 'verified'
instances which are the ones the cluster has explicitly started.

In the case of a cluster split, the partition with a verified instance
will have a slight advantage as stonith-ng in the other partition will
have to hear from all its current peers before choosing a node to
perform the fencing.

[NOTE]
===========
To disable a fencing device/resource, 'target-role' can be set as you would for a normal resource.
===========

[NOTE]
===========
To prevent a specific node from using a fencing device, location constraints will work as expected.
===========

[IMPORTANT]
===========

Currently there is a limitation that fencing resources may only have a
one set of meta-attributes and one set of instance-attributes.  This
can be revisited if it becomes a significant limitation for people.

===========

.Properties of Fencing Devices
[width="95%",cols="1m,1m,1m,5<",options="header",align="center"]
|=========================================================

|Field
|Type
|Default
|Description

|stonith-timeout
|time
|60s
|How long to wait for the STONITH action to complete per a stonith device.
 Overrides the stonith-timeout cluster property
 indexterm:[stonith-timeout,Fencing]
 indexterm:[Fencing,Property,stonith-timeout]

|priority
|integer
|0
|The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
 indexterm:[priority,Fencing]
 indexterm:[Fencing,Property,priority]

|pcmk_host_argument
|string
|port
|Advanced use only: An alternate parameter to supply instead of 'port'
 Some devices do not support the standard 'port' parameter or may provide additional ones. Use this to specify an alternate, device-specific, parameter that should indicate the machine to be fenced. A value of 'none' can be used to tell the cluster not to supply any additional parameters.
 indexterm:[pcmk_host_argument,Fencing]
 indexterm:[Fencing,Property,pcmk_host_argument]

|pcmk_host_map
|string
|
|A mapping of host names to ports numbers for devices that do not support host names.
 Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2
 indexterm:[pcmk_host_map,Fencing]
 indexterm:[Fencing,Property,pcmk_host_map]

|pcmk_host_list
|string
|
|A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
 indexterm:[pcmk_host_list,Fencing]
 indexterm:[Fencing,Property,pcmk_host_list]

|pcmk_host_check
|string
|dynamic-list
|How to determin which machines are controlled by the device.
 Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine)
 indexterm:[pcmk_host_check,Fencing]
 indexterm:[Fencing,Property,pcmk_host_check]

|pcmk_reboot_action
|string
|reboot
|Advanced use only: An alternate command to run instead of 'reboot'
 Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'reboot' action.
 indexterm:[pcmk_reboot_action,Fencing]
 indexterm:[Fencing,Property,pcmk_reboot_action]

|pcmk_reboot_timeout
|time
|60s
|Advanced use only: Specify an alternate timeout to use for reboot actions instead of stonith-timeout
 Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'reboot' actions.
 indexterm:[pcmk_reboot_timeout,Fencing]
 indexterm:[Fencing,Property,pcmk_reboot_timeout]

|pcmk_reboot_retries
|integer
|2
|Advanced use only: The maximum number of times to retry the 'reboot' command within the timeout period
 Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'reboot' actions before giving up.
 indexterm:[pcmk_reboot_retries,Fencing]
 indexterm:[Fencing,Property,pcmk_reboot_retries]

|pcmk_off_action
|string
|off
|Advanced use only: An alternate command to run instead of 'off'
 Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'off' action.
 indexterm:[pcmk_off_action,Fencing]
 indexterm:[Fencing,Property,pcmk_off_action]

|pcmk_off_timeout
|time
|60s
|Advanced use only: Specify an alternate timeout to use for off actions instead of stonith-timeout
 Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'off' actions.
 indexterm:[pcmk_off_timeout,Fencing]
 indexterm:[Fencing,Property,pcmk_off_timeout]

|pcmk_off_retries
|integer
|2
|Advanced use only: The maximum number of times to retry the 'off' command within the timeout period
 Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'off' actions before giving up.
 indexterm:[pcmk_off_retries,Fencing]
 indexterm:[Fencing,Property,pcmk_off_retries]

|pcmk_list_action
|string
|list
|Advanced use only: An alternate command to run instead of 'list'
 Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'list' action.
 indexterm:[pcmk_list_action,Fencing]
 indexterm:[Fencing,Property,pcmk_list_action]

|pcmk_list_timeout
|time
|60s
|Advanced use only: Specify an alternate timeout to use for list actions instead of stonith-timeout
 Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'list' actions.
 indexterm:[pcmk_list_timeout,Fencing]
 indexterm:[Fencing,Property,pcmk_list_timeout]

|pcmk_list_retries
|integer
|2
|Advanced use only: The maximum number of times to retry the 'list' command within the timeout period
 Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'list' actions before giving up.
 indexterm:[pcmk_list_retries,Fencing]
 indexterm:[Fencing,Property,pcmk_list_retries]

|pcmk_monitor_action
|string
|monitor
|Advanced use only: An alternate command to run instead of 'monitor'
 Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'monitor' action.
 indexterm:[pcmk_monitor_action,Fencing]
 indexterm:[Fencing,Property,pcmk_monitor_action]

|pcmk_monitor_timeout
|time
|60s
|Advanced use only: Specify an alternate timeout to use for monitor actions instead of stonith-timeout
 Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'monitor' actions.
 indexterm:[pcmk_monitor_timeout,Fencing]
 indexterm:[Fencing,Property,pcmk_monitor_timeout]

|pcmk_monitor_retries
|integer
|2
|Advanced use only: The maximum number of times to retry the 'monitor' command within the timeout period
 Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'monitor' actions before giving up.
 indexterm:[pcmk_monitor_retries,Fencing]
 indexterm:[Fencing,Property,pcmk_monitor_retries]

|pcmk_status_action
|string
|status
|Advanced use only: An alternate command to run instead of 'status'
 Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'status' action.
 indexterm:[pcmk_status_action,Fencing]
 indexterm:[Fencing,Property,pcmk_status_action]

|pcmk_status_timeout
|time
|60s
|Advanced use only: Specify an alternate timeout to use for status actions instead of stonith-timeout
 Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'status' actions.
 indexterm:[pcmk_status_timeout,Fencing]
 indexterm:[Fencing,Property,pcmk_status_timeout]

|pcmk_status_retries
|integer
|2
|Advanced use only: The maximum number of times to retry the 'status' command within the timeout period
 Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'status' actions before giving up.
 indexterm:[pcmk_status_retries,Fencing]
 indexterm:[Fencing,Property,pcmk_status_retries]

|=========================================================

== Configuring STONITH ==

[NOTE]
===========

Both configuration shells include functionality to simplify the
process below, particularly the step for deciding which parameters are
required.  However since this document deals only with core
components, you should refer to the Stonith chapter of +Clusters from
Scratch+ for those details.

===========

. Find the correct driver: +stonith_admin --list-installed+

. Find the required parameters associated with the device: +stonith_admin --metadata --agent <agent name>+

. Create a file called +stonith.xml+ containing a primitive resource
  with a class of 'stonith', a type of <agent name> and a parameter
  for each of the values returned in step 2.

. If the device does not know how to fence nodes based on their uname,
  you may also need to set the special +pcmk_host_map+ parameter.  See
  +man stonithd+ for details.

. If the device does not support the list command, you may also need
  to set the special +pcmk_host_list+ and/or +pcmk_host_check+
  parameters.  See +man stonithd+ for details.

. If the device does not expect the victim to be specified with the
  port parameter, you may also need to set the special
  +pcmk_host_argument+ parameter. See +man stonithd+ for details.

. Upload it into the CIB using cibadmin: +cibadmin -C -o resources --xml-file stonith.xml+

. Set stonith-enabled to true. +crm_attribute -t crm_config -n stonith-enabled -v true+

. Once the stonith resource is running, you can test it by executing:
  +stonith_admin --reboot nodename+. Although you might want to stop the
  cluster on that machine first.

=== Example ===

Assuming we have an chassis containing four nodes and an IPMI device
active on 10.0.0.1, then we would chose the fence_ipmilan driver in step
2 and obtain the following list of parameters

.Obtaining a list of STONITH Parameters

[source,C]
----
# stonith_admin --metadata -a fence_ipmilan
----

[source,XML]
----
<?xml version="1.0" ?>
<resource-agent name="fence_ipmilan" shortdesc="Fence agent for IPMI over LAN">
<longdesc>
fence_ipmilan is an I/O Fencing agent which can be used with machines controlled by IPMI. This agent calls support software using ipmitool (http://ipmitool.sf.net/).

To use fence_ipmilan with HP iLO 3 you have to enable lanplus option (lanplus / -P) and increase wait after operation to 4 seconds (power_wait=4 / -T 4)</longdesc>
<parameters>
        <parameter name="auth" unique="1">
                <getopt mixed="-A" />
                <content type="string" />
                <shortdesc lang="en">IPMI Lan Auth type (md5, password, or none)</shortdesc>
        </parameter>
        <parameter name="ipaddr" unique="1">
                <getopt mixed="-a" />
                <content type="string" />
                <shortdesc lang="en">IPMI Lan IP to talk to</shortdesc>
        </parameter>
        <parameter name="passwd" unique="1">
                <getopt mixed="-p" />
                <content type="string" />
                <shortdesc lang="en">Password (if required) to control power on IPMI device</shortdesc>
        </parameter>
        <parameter name="passwd_script" unique="1">
                <getopt mixed="-S" />
                <content type="string" />
                <shortdesc lang="en">Script to retrieve password (if required)</shortdesc>
        </parameter>
        <parameter name="lanplus" unique="1">
                <getopt mixed="-P" />
                <content type="boolean" />
                <shortdesc lang="en">Use Lanplus</shortdesc>
        </parameter>
        <parameter name="login" unique="1">
                <getopt mixed="-l" />
                <content type="string" />
                <shortdesc lang="en">Username/Login (if required) to control power on IPMI device</shortdesc>
        </parameter>
        <parameter name="action" unique="1">
                <getopt mixed="-o" />
                <content type="string" default="reboot"/>
                <shortdesc lang="en">Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata</shortdesc>
        </parameter>
        <parameter name="timeout" unique="1">
                <getopt mixed="-t" />
                <content type="string" />
                <shortdesc lang="en">Timeout (sec) for IPMI operation</shortdesc>
        </parameter>
        <parameter name="cipher" unique="1">
                <getopt mixed="-C" />
                <content type="string" />
                <shortdesc lang="en">Ciphersuite to use (same as ipmitool -C parameter)</shortdesc>
        </parameter>
        <parameter name="method" unique="1">
                <getopt mixed="-M" />
                <content type="string" default="onoff"/>
                <shortdesc lang="en">Method to fence (onoff or cycle)</shortdesc>
        </parameter>
        <parameter name="power_wait" unique="1">
                <getopt mixed="-T" />
                <content type="string" default="2"/>
                <shortdesc lang="en">Wait X seconds after on/off operation</shortdesc>
        </parameter>
        <parameter name="delay" unique="1">
                <getopt mixed="-f" />
                <content type="string" />
                <shortdesc lang="en">Wait X seconds before fencing is started</shortdesc>
        </parameter>
        <parameter name="verbose" unique="1">
                <getopt mixed="-v" />
                <content type="boolean" />
                <shortdesc lang="en">Verbose mode</shortdesc>
        </parameter>
</parameters>
<actions>
        <action name="on" />
        <action name="off" />
        <action name="reboot" />
        <action name="status" />
        <action name="diag" />
        <action name="list" />
        <action name="monitor" />
        <action name="metadata" />
</actions>
</resource-agent>
----

from which we would create a STONITH resource fragment that might look
like this:

.Sample STONITH Resource
[source,XML]
----
<primitive id="Fencing" class="stonith" type="fence_ipmilan" >
  <instance_attributes id="Fencing-params" >
    <nvpair id="Fencing-passwd" name="passwd" value="testuser" />
    <nvpair id="Fencing-login" name="login" value="abc123" />
    <nvpair id="Fencing-ipaddr" name="ipaddr" value="10.0.0.1" />
    <nvpair id="Fencing-pcmk_host_list" name="pcmk_host_list" value="pcmk-1 pcmk-2" />
  </instance_attributes>
  <operations >
    <op id="Fencing-monitor-10m" interval="10m" name="monitor" timeout="300s" />
  </operations>
</primitive>
----

And finally, since we disabled it earlier, we need to re-enable STONITH.

[source,Bash]
----
# crm_attribute -t crm_config -n stonith-enabled -v true
----

== Advanced Fencing Configurations ==

Some people consider that having one fencing device is a single point
of failure footnote:[Not true, since a node or resource must fail
before fencing even has a chance to], others prefer removing the node
from the storage and network instead of turning it off.

Whatever the reason, Pacemaker supports fencing nodes with multiple
devices through a feature called fencing topologies.

Simply create the individual devices as you normally would and then
define one or more fencing levels in the fencing-topology section in
the configuration.

* Each level is attempted in +ascending index+ order
* If a device fails, +processing terminates+ for the current level.
  No further devices in that level are exercised and the next level is attempted instead.
* If the operation succeeds for all the listed devices in a level, the level is deemed to have passed
* The operation is finished +when a level has passed+ (success), or all levels have been attempted (failed)
* If the operation failed, the next step is determined by the Policy Engine and/or crmd.

Some possible uses of topologies include:

* try poison-pill and fail back to power
* try disk and network, and fall back to power if either fails
* initiate a kdump and then poweroff the node

.Properties of Fencing Levels
[width="95%",cols="1m,6<",options="header",align="center"]
|=========================================================

|Field
|Description

|id
|Your name for the level
 indexterm:[id,fencing-level]
 indexterm:[Fencing,fencing-level,id]

|target
|The node to which this level applies
 indexterm:[target,fencing-level]
 indexterm:[Fencing,fencing-level,target]

|index
|The order in which to attempt the levels.
 Levels are attempted in +ascending index+ order +until one succeeds+.
 indexterm:[index,fencing-level]
 indexterm:[Fencing,fencing-level,index]

|devices
|A comma separated list of devices for which the
 indexterm:[devices,fencing-level]
 indexterm:[Fencing,fencing-level,devices]

|=========================================================

=== Example use of Fencing Topologies ===
[source,XML]
----
 <cib crm_feature_set="3.0.6" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0">
  <configuration>
    ...
    <fencing-topology>
      <!-- For pcmk-1, try poison-pill and fail back to power -->
      <fencing-level id="f-p1.1" target="pcmk-1" index="1" devices="poison-pill"/>
      <fencing-level id="f-p1.2" target="pcmk-1" index="2" devices="power"/>

      <!-- For pcmk-2, try disk and network, and fail back to power -->
      <fencing-level id="f-p2.1" target="pcmk-2" index="1" devices="disk,network"/>
      <fencing-level id="f-p2.2" target="pcmk-2" index="2" devices="power"/>
    </fencing-topology>
    ...
  <configuration>
  <status/>
</cib>
----

=== Example use of advanced Fencing Topologies: dual layer and dual devices ===

The following example illustrates an advanced use of +fencing_topology+ in a cluster with the following properties:

* 3 nodes (2 active prod-mysql nodes, 1 prod_mysql-rep in standby for quorum purposes)
* the active nodes have an IPMI-controlled power board reached at 10.10.10.1 and 10.10.10.2
* the active nodes also have two independant PSUs (Power Supplu Units) connected to two independant PDUs (Power Distribution Unit) reached at 10.20.1.1 (port 10 and port 11) and 10.20.2.1 (port 10 and port 11)
* the first fencing method uses the +fence_ipmi+ agent
* the second fencing method uses the +fence_apc_snmp+ agent targetting 2 fencing devices (one per PSU, either port 10 or 11)
* fencing is only implemented for the active nodes and has location constraints
* fencing topology is set to try IPMI fencing first then default to a "sure-kill" dual PDU fencing

In a normal failure scenario, STONITH will first select +fence_ipmi+ to try and kill the faulty node.
Using a +fencing_topology+, if that first method fails, STONITH will then move on to selecting +fence_apc_snmp+ twice:

* once for the first PDU 
* again for the second PDU 

The fence action is considered successful only if both PDUs report the required status. If any of them fails, STONITH loops back to the first fencing method, +fence_ipmi+, and so on until the node is fenced or fencing action is cancelled.

.First fencing method: single IPMI device

Each cluster node has it own dedicated IPMI channel that can be called for fencing using the following primitives:
[source,XML]
----
      <primitive class="stonith" id="fence_prod-mysql1_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_prod-mysql1_ipmi-instance_attributes">
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-ipaddr" name="ipaddr" value="10.10.10.1"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-verbose" name="verbose" value="true"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_prod-mysql2_ipmi-instance_attributes">
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-ipaddr" name="ipaddr" value="10.10.10.2"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-verbose" name="verbose" value="true"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>
        </instance_attributes>
      </primitive>
----

.Second fencing method: dual PDU devices

Each cluster node also has two distinct power channels controlled by two distinct PDUs. That means a total of 4 fencing devices configured as follows:

- Node 1, PDU 1, PSU 1 @ port 10
- Node 1, PDU 2, PSU 2 @ port 10
- Node 2, PDU 1, PSU 1 @ port 11
- Node 2, PDU 2, PSU 2 @ port 11

The matching fencing agents are configured as follows:
[source,XML]
----
      <primitive class="stonith" id="fence_prod-mysql1_apc1" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql1_apc1-instance_attributes">
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-ipaddr" name="ipaddr" value="10.20.1.1"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-port" name="port" value="10"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql1_apc2" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql1_apc2-instance_attributes">
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-ipaddr" name="ipaddr" value="10.20.2.1"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-port" name="port" value="10"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_apc1" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql2_apc1-instance_attributes">
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-ipaddr" name="ipaddr" value="10.20.1.1"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-port" name="port" value="11"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_apc2" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql2_apc2-instance_attributes">
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-ipaddr" name="ipaddr" value="10.20.2.1"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-port" name="port" value="11"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
        </instance_attributes>
      </primitive>

----

.Location Constraints 

To prevent STONITH from running a fencing agent on the very same node it is supposed to fence, constraints are placed on all the fencing primitives:
[source,XML]
----
    <constraints>
      <rsc_location id="l_fence_prod-mysql1_ipmi" node="prod-mysql1" rsc="fence_prod-mysql1_ipmi" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_ipmi" node="prod-mysql2" rsc="fence_prod-mysql2_ipmi" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql1_apc2" node="prod-mysql1" rsc="fence_prod-mysql1_apc2" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql1_apc1" node="prod-mysql1" rsc="fence_prod-mysql1_apc1" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_apc1" node="prod-mysql2" rsc="fence_prod-mysql2_apc1" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_apc2" node="prod-mysql2" rsc="fence_prod-mysql2_apc2" score="-INFINITY"/>
    </constraints>
----

.Fencing topology

Now that all the fencing resources are defined, it's time to create the right topology. 
We want to first fence using IPMI and if that does not work, fence both PDUs to effectively and surely kill the node.
[source,XML]
----
    <fencing-topology>
      <fencing-level devices="fence_prod-mysql1_ipmi" id="fencing-2" index="1" target="prod-mysql1"/>
      <fencing-level devices="fence_prod-mysql1_apc1,fence_prod-mysql1_apc2" id="fencing-3" index="2" target="prod-mysql1"/>
      <fencing-level devices="fence_prod-mysql2_ipmi" id="fencing-0" index="1" target="prod-mysql2"/>
      <fencing-level devices="fence_prod-mysql2_apc1,fence_prod-mysql2_apc2" id="fencing-1" index="2" target="prod-mysql2"/>
    </fencing-topology>
----
Please note, in +fencing_topology+, the lower index value determines the priority of the first fencing method. 

.Final configuration

Put together, the configuration looks like this:
[source,XML]
----
<cib admin_epoch="0" crm_feature_set="3.0.7" epoch="292" have-quorum="1" num_updates="29" validate-with="pacemaker-1.2">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/>
        <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="off"/>
        <nvpair id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes" value="3"/>
       ...
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="prod-mysql1" uname="prod-mysql1">
      <node id="prod-mysql2" uname="prod-mysql2"/>
      <node id="prod-mysql-rep1" uname="prod-mysql-rep1"/>
        <instance_attributes id="prod-mysql-rep1">
          <nvpair id="prod-mysql-rep1-standby" name="standby" value="on"/>
        </instance_attributes>
      </node>
    </nodes>
    <resources>
      <primitive class="stonith" id="fence_prod-mysql1_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_prod-mysql1_ipmi-instance_attributes">
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-ipaddr" name="ipaddr" value="10.10.10.1"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-verbose" name="verbose" value="true"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
          <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_ipmi" type="fence_ipmilan">
        <instance_attributes id="fence_prod-mysql2_ipmi-instance_attributes">
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-ipaddr" name="ipaddr" value="10.10.10.2"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-verbose" name="verbose" value="true"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
          <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql1_apc1" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql1_apc1-instance_attributes">
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-ipaddr" name="ipaddr" value="10.20.1.1"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-port" name="port" value="10"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql1_apc2" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql1_apc2-instance_attributes">
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-ipaddr" name="ipaddr" value="10.20.2.1"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-port" name="port" value="10"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql1_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_apc1" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql2_apc1-instance_attributes">
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-ipaddr" name="ipaddr" value="10.20.1.1"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-port" name="port" value="11"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
        </instance_attributes>
      </primitive>
      <primitive class="stonith" id="fence_prod-mysql2_apc2" type="fence_apc_snmp">
        <instance_attributes id="fence_prod-mysql2_apc2-instance_attributes">
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-ipaddr" name="ipaddr" value="10.20.2.1"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-action" name="action" value="off"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-port" name="port" value="11"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-login" name="login" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-passwd" name="passwd" value="fencing"/>
          <nvpair id="fence_prod-mysql2_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/>
        </instance_attributes>
      </primitive>
   </resources>
    <constraints>
      <rsc_location id="l_fence_prod-mysql1_ipmi" node="prod-mysql1" rsc="fence_prod-mysql1_ipmi" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_ipmi" node="prod-mysql2" rsc="fence_prod-mysql2_ipmi" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql1_apc2" node="prod-mysql1" rsc="fence_prod-mysql1_apc2" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql1_apc1" node="prod-mysql1" rsc="fence_prod-mysql1_apc1" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_apc1" node="prod-mysql2" rsc="fence_prod-mysql2_apc1" score="-INFINITY"/>
      <rsc_location id="l_fence_prod-mysql2_apc2" node="prod-mysql2" rsc="fence_prod-mysql2_apc2" score="-INFINITY"/>
    </constraints>
    <fencing-topology>
      <fencing-level devices="fence_prod-mysql1_ipmi" id="fencing-2" index="1" target="prod-mysql1"/>
      <fencing-level devices="fence_prod-mysql1_apc1,fence_prod-mysql1_apc2" id="fencing-3" index="2" target="prod-mysql1"/>
      <fencing-level devices="fence_prod-mysql2_ipmi" id="fencing-0" index="1" target="prod-mysql2"/>
      <fencing-level devices="fence_prod-mysql2_apc1,fence_prod-mysql2_apc2" id="fencing-1" index="2" target="prod-mysql2"/>
    </fencing-topology>
   ...
  </configuration>
</cib>
----
