=  Resource Constraints =

indexterm:[Resource,Constraints]

== Scores ==

Scores of all kinds are integral to how the cluster works.
Practically everything from moving a resource to deciding which
resource to stop in a degraded cluster is achieved by manipulating
scores in some way.

Scores are calculated on a per-resource basis and any node with a
negative score for a resource can't run that resource.  After
calculating the scores for a resource, the cluster then chooses the
node with the highest one.

=== Infinity Math ===

+INFINITY+ is currently defined as 1,000,000 and addition/subtraction
with it follows these three basic rules:

* Any value + +INFINITY+ = +INFINITY+
* Any value - +INFINITY+ = -+INFINITY+
* +INFINITY+ - +INFINITY+ = -+INFINITY+

== Deciding Which Nodes a Resource Can Run On ==

indexterm:[Location Constraints]
indexterm:[Resource,Constraints,Location]
There are two alternative strategies for specifying which nodes a
resources can run on.  One way is to say that by default they can run
anywhere and then create location constraints for nodes that are not
allowed.  The other option is to have nodes "opt-in"...  to start with
nothing able to run anywhere and selectively enable allowed nodes.
      
=== Options ===

.Options for Simple Location Constraints
[width="95%",cols="2m,5<",options="header",align="center"]
|=========================================================

|Field
|Description

|id
|A unique name for the constraint
indexterm:[id,Location Constraints]
indexterm:[Constraints,Location,id]

|rsc
|A resource name
indexterm:[rsc,Location Constraints]
indexterm:[Constraints,Location,rsc]

|node
|A node's name
indexterm:[node,Location Constraints]
indexterm:[Constraints,Location,node]

|score
|Positive values indicate the resource should run on this
 node. Negative values indicate the resource should not run on this
 node.

 Values of \+/- +INFINITY+ change "should"/"should not" to
 "must"/"must not".
indexterm:[score,Location Constraints]
indexterm:[Constraints,Location,score]

|=========================================================

=== Asymmetrical "Opt-In" Clusters ===
indexterm:[Asymmetrical Opt-In Clusters]
indexterm:[Cluster Type,Asymmetrical Opt-In]

To create an opt-in cluster, start by preventing resources from
running anywhere by default:

[source,C]
# crm_attribute --attr-name symmetric-cluster --attr-value false

Then start enabling nodes.  The following fragment says that the web
server prefers +sles-1+, the database prefers +sles-2+ and both can
fail over to +sles-3+ if their most preferred node fails.

.Example set of opt-in location constraints 
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-3" score="0"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-2" score="200"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-3" score="0"/>
</constraints>
-------
======

=== Symmetrical "Opt-Out" Clusters ===
indexterm:[Symmetrical Opt-Out Clusters]
indexterm:[Cluster Type,Symmetrical Opt-Out]

To create an opt-out cluster, start by allowing resources to run
anywhere by default:

[source,C]
# crm_attribute --attr-name symmetric-cluster --attr-value true

Then start disabling nodes.  The following fragment is the equivalent
of the above opt-in configuration.
       
.Example set of opt-out location constraints 
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2-dont-run" rsc="Webserver" node="sles-2" score="-INFINITY"/>
    <rsc_location id="loc-3-dont-run" rsc="Database" node="sles-1" score="-INFINITY"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="200"/>
</constraints>
-------
======

Whether you should choose opt-in or opt-out depends both on your
personal preference and the make-up of your cluster.  If most of your
resources can run on most of the nodes, then an opt-out arrangement is
likely to result in a simpler configuration.  On the other-hand, if
most resources can only run on a small subset of nodes an opt-in
configuration might be simpler.

[[node-score-equal]]
=== What if Two Nodes Have the Same Score ===

If two nodes have the same score, then the cluster will choose one.
This choice may seem random and may not be what was intended, however
the cluster was not given enough information to know any better.

.Example of two resources that prefer two nodes equally 
======
[source,XML]
-------
<constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="INFINITY"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-2" score="INFINITY"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-1" score="500"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="300"/>
    <rsc_location id="loc-5" rsc="Database" node="sles-2" score="200"/>
</constraints>
-------
======

In the example above, assuming no other constraints and an inactive
cluster, Webserver would probably be placed on sles-1 and Database on
sles-2.  It would likely have placed Webserver based on the node's
uname and Database based on the desire to spread the resource load
evenly across the cluster.  However other factors can also be involved
in more complex configurations.

[[s-resource-ordering]]
== Specifying in which Order Resources Should Start/Stop ==

indexterm:[Resource,Constraints,Ordering]
indexterm:[Resource,Start Order]
indexterm:[Ordering Constraints]
The way to specify the order in which resources should start is by
creating +rsc_order+ constraints.

.Properties of an Ordering Constraint
[width="95%",cols="2m,5<",options="header",align="center"]
|=========================================================
            
|Field
|Description

|id
|A unique name for the constraint
indexterm:[id,Ordering Constraints]
indexterm:[Constraints,Ordering,id]

|first
|The name of a resource that must be started before the +then+
 resource is allowed to.
indexterm:[first,Ordering Constraints]
indexterm:[Constraints,Ordering,first]

|then
|The name of a resource. This resource will start after the +first+ resource.
indexterm:[then,Ordering Constraints]
indexterm:[Constraints,Ordering,then]

|kind
|How to enforce the constraint. ('Since 1.1.2')

* Optional - Just a suggestion.  Only applies if both resources are
  starting/stopping.

* Mandatory - Always.  If 'first' is stopping or cannot be started,
  'then' must be stopped.

* Serialize - Ensure that no two stop/start actions occur concurrently
  for a set of resources.

indexterm:[kind,Ordering Constraints]
indexterm:[Constraints,Ordering,kind]

|symmetrical
|If true, which is the default, stop the resources in the reverse
 order. Default value: _true_
indexterm:[symmetrical,Ordering Constraints]
indexterm:[Ordering Constraints,symmetrical]

|=========================================================

=== Mandatory Ordering ===

When the +then+ resource cannot run without the +first+ resource being
active, one should use mandatory constraints.  To specify a constraint
is mandatory, use scores greater than zero.  This will ensure that the
then resource will react when the first resource changes state.

* If the +first+ resource was running and is stopped, the +then+
  resource will also be stopped (if it is running).
* If the +first+ resource was not running and cannot be started, the
  +then+ resource will be stopped (if it is running).
* If the +first+ resource is (re)started while the +then+ resource is
  running, the +then+ resource will be stopped and restarted.

=== Advisory Ordering ===

On the other hand, when +score="0"+ is specified for a constraint, the
constraint is considered optional and only has an effect when both
resources are stopping and/or starting.  Any change in state by the
+first+ resource will have no effect on the +then+ resource.

.Example of an optional and mandatory ordering constraint     
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1" first="Database" then="Webserver" />
    <rsc_order id="order-2" first="IP" then="Webserver" score="0"/>
</constraints>
-------
======

Some additional information on ordering constraints can be found in
the document http://clusterlabs.org/doc/Ordering_Explained.pdf[Ordering Explained].

[[s-resource-colocation]]
== Placing Resources Relative to other Resources ==

indexterm:[Resource,Constraints,Colocation]
indexterm:[Resource,Location Relative to other Resources]
When the location of one resource depends on the location of another
one, we call this colocation.

There is an important side-effect of creating a colocation constraint
between two resources: it affects the order in which resources are
assigned to a node.  If you think about it, it's somewhat obvious.
You can't place A relative to B unless you know where B is.
footnote:[
While the human brain is sophisticated enough to read the constraint
in any order and choose the correct one depending on the situation,
the cluster is not quite so smart. Yet.
]

So when you are creating colocation constraints, it is important to
consider whether you should colocate A with B or B with A.

Another thing to keep in mind is that, assuming A is collocated with
B, the cluster will also take into account A's preferences when
deciding which node to choose for B.

For a detailed look at exactly how this occurs, see the
http://www.clusterlabs.org/mediawiki/images/6/61/Colocation_Explained.pdf[Colocation
Explained] document.

=== Options ===

.Properties of a Collocation Constraint
[width="95%",cols="2m,5<",options="header",align="center"]
|=========================================================

|Field
|Description

|id
|A unique name for the constraint.
indexterm:[id,Colocation Constraints]
indexterm:[Constraints,Colocation,id]

|rsc
|The colocation source. If the constraint cannot be satisfied, the
 cluster may decide not to allow the resource to run at all.
indexterm:[rsc,Colocation Constraints]
indexterm:[Constraints,Colocation,rsc]

|with-rsc
|The colocation target. The cluster will decide where to put this
 resource first and then decide where to put the resource in the +rsc+
 field.
 indexterm:[with-rsc,Colocation Constraints]
 indexterm:[Constraints,Colocation,with-rsc]

|score
|Positive values indicate the resource should run on the same
 node. Negative values indicate the resources should not run on the
 same node. Values of \+/- +INFINITY+ change "should" to "must".
 indexterm:[score,Colocation Constraints]
 indexterm:[Constraints,Colocation,score]

|=========================================================

=== Mandatory Placement ===

Mandatory placement occurs any time the constraint's score is
++INFINITY+ or +-INFINITY+.  In such cases, if the constraint can't be
satisfied, then the +rsc+ resource is not permitted to run.  For
+score=INFINITY+, this includes cases where the +with-rsc+ resource is
not active.

If you need +resource1+ to always run on the same machine as
+resource2+, you would add the following constraint:

.An example colocation constraint
[source,XML]
<rsc_colocation id="colocate" rsc="resource1" with-rsc="resource2" score="INFINITY"/>

Remember, because +INFINITY+ was used, if +resource2+ can't run on any
of the cluster nodes (for whatever reason) then +resource1+ will not
be allowed to run.

Alternatively, you may want the opposite...  that +resource1+ cannot
run on the same machine as +resource2+.  In this case use
+score="-INFINITY"+
        
.An example anti-colocation constraint
[source,XML]
<rsc_colocation id="anti-colocate" rsc="resource1" with-rsc="resource2" score="-INFINITY"/>

Again, by specifying +-INFINTY+, the constraint is binding.  So if the
only place left to run is where +resource2+ already is, then
+resource1+ may not run anywhere.

=== Advisory Placement ===

If mandatory placement is about "must" and "must not", then advisory
placement is the "I'd prefer if" alternative.  For constraints with
scores greater than +-INFINITY+ and less than +INFINITY+, the cluster
will try and accommodate your wishes but may ignore them if the
alternative is to stop some of the cluster resources.
        

Like in life, where if enough people prefer something it effectively
becomes mandatory, advisory colocation constraints can combine with
other elements of the configuration to behave as if they were
mandatory.

.An example advisory-only colocation constraint
[source,XML]
<rsc_colocation id="colocate-maybe" rsc="resource1" with-rsc="resource2" score="500"/>

[[s-resource-sets-ordering]]
== Ordering Sets of Resources ==

A common situation is for an administrator to create a chain of
ordered resources, such as:

.A chain of ordered resources
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1" first="A" then="B" />
    <rsc_order id="order-2" first="B" then="C" />
    <rsc_order id="order-3" first="C" then="D" />
</constraints>
-------
======

.Visual representation of the four resources' start order for the above constraints
image::images/resource-set.png["Ordered set",width="16cm",height="2.5cm",align="center"]
        
=== Ordered Set ===

To simplify this situation, there is an alternate format for ordering
constraints:

.A chain of ordered resources expressed as a set
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======

[WARNING]
=========
Always pay attention to how your tools expose this functionality.
In some tools +create set A B+ is *NOT* equivalent to +create A then B+.
=========

While the set-based format is not less verbose, it is significantly
easier to get right and maintain.  It can also be expanded to allow
ordered sets of (un)ordered resources.  In the example below, +rscA+
and +rscB+ can both start in parallel, as can +rscC+ and +rscD+,
however +rscC+ and +rscD+ can only start once _both_ +rscA+ _and_ 
 +rscB+ are active.

.Ordered sets of unordered resources
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
  </constraints>
-------
======
        
.Visual representation of the start order for two ordered sets of unordered resources
image::images/two-sets.png["Two ordered sets",width="13cm",height="7.5cm",align="center"]

Of course either set -- or both sets -- of resources can also be
internally ordered (by setting +sequential="true"+) and there is no
limit to the number of sets that can be specified.

.Advanced use of set ordering - Three ordered sets, two of which are internally unordered
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="true">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
      <resource_set id="ordered-set-3" sequential="false">
        <resource_ref id="E"/>
        <resource_ref id="F"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======

.Visual representation of the start order for the three sets defined above
image::images/three-sets.png["Three ordered sets",width="16cm",height="7.5cm",align="center"]


=== Resource Set OR Logic ===

The unordered set logic discussed so far has all been "AND" logic.
To illustrate this take the 3 resource set figure in the previous section.
Those sets can be expressed, +(A and B) then (C) then (D) then (E and F)+

Say for example we want change the first set, (A and B), to use "OR" logic
so the sets look like this, +(A or B) then (C) then (D) then (E and F)+.
This functionality can be achieved through the use of the +require-all+
option.  By default this option is 'require-all=true' which is why the
"AND" logic is used by default.  Changing +require-all=false+ means only one
resource in the set needs to be started before continuing on to the next set.

Note that the 'require-all=false' option only makes sense to use in conjunction
with unordered sets, 'sequential=false'.  Think of it like this, 'sequential=false'
modifies the set to be an unordered set that uses "AND" logic by default, by adding
'require-all=false' the unordered set's "AND" logic is flipped to "OR" logic.

.Resource Set "OR" logic. Three ordered sets, where the first set is internally unordered with "OR" logic.
======
[source,XML]
-------
<constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false" require-all="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="true">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
      <resource_set id="ordered-set-3" sequential="false">
        <resource_ref id="E"/>
        <resource_ref id="F"/>
      </resource_set>
    </rsc_order>
</constraints>
-------
======


[[s-resource-sets-colocation]]
== Collocating Sets of Resources ==

Another common situation is for an administrator to create a set of
collocated resources.  Previously this was possible either by defining
a resource group (See <<group-resources>>) which could not always
accurately express the design; or by defining each relationship as an
individual constraint, causing a constraint explosion as the number of
resources and combinations grew.

.A chain of collocated resources
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" rsc="B" with-rsc="A" score="INFINITY"/>
    <rsc_colocation id="coloc-2" rsc="C" with-rsc="B" score="INFINITY"/>
    <rsc_colocation id="coloc-3" rsc="D" with-rsc="C" score="INFINITY"/>
</constraints>
-------
======

To make things easier, we allow an alternate form of colocation
constraints using +resource_sets+.  Just like the expanded version, a
resource that can't be active also prevents any resource that must be
collocated with it from being active.  For example, if +B+ was not
able to run, then both +C+ (and by inference +D+) must also remain
stopped.

.The equivalent colocation chain expressed using +resource_sets+   
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======

[WARNING]
=========
Always pay attention to how your tools expose this functionality.
In some tools +create set A B+ is *NOT* equivalent to +create A with B+.
=========

.A group resource with the equivalent colocation rules
[source,XML]
-------
<group id="dummy">
    <primitive id="A" .../>
    <primitive id="B" .../>
    <primitive id="C" .../>
    <primitive id="D" .../>
</group>
-------

This notation can also be used in this context to tell the cluster
that a set of resources must all be located with a common peer, but
have no dependencies on each other.  In this scenario, unlike the
previous, +B would+ be allowed to remain active even if +A or+ +C+ (or
both) were inactive.

.Using colocation sets to specify a common peer.    
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="true">
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======

Of course there is no limit to the number and size of the sets used.
The only thing that matters is that in order for any member of set N
to be active, all the members of set N+1 must also be active (and
naturally on the same node); and if a set has +sequential="true"+,
then in order for member M to be active, member M+1 must also be
active.  You can even specify the role in which the members of a set
must be in using the set's role attribute.
      
.A colocation chain where the members of the middle set have no inter-dependencies and the last has master status.
======
[source,XML]
-------
<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-1" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
        <resource_ref id="E"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="true" role="Master">
        <resource_ref id="F"/>
        <resource_ref id="G"/>
      </resource_set>
    </rsc_colocation>
</constraints>
-------
======
        
.Visual representation of a colocation chain where the members of the middle set have no inter-dependencies
image::images/three-sets-complex.png["Colocation chain",width="16cm",height="9cm",align="center"]
