
                        PACEMAKER
                 CLUSTER TEST SUITE (CTS)


Purpose
-------

CTS thoroughly exercises a pacemaker test cluster by running a randomized
series of predefined tests on the cluster. CTS can be run against a
pre-existing cluster configuration or (more typically) overwrite the existing
configuration with a test configuration.


Requirements
------------

* Three or more machines (one test exerciser and two or more test cluster
  machines).

* The test cluster machines should be on the same subnet and have journalling
  filesystems (ext3, ext4, xfs, etc.) for all of their filesystems other than
  /boot. You also need a number of free IP addresses on that subnet if you
  intend to test mutual IP address takeover.

* The test exerciser machine doesn't need to be on the same subnet as the test
  cluster machines.  Minimal demands are made on the exerciser machine - it
  just has to stay up during the tests.

* It helps a lot in tracking problems if all machines' clocks are closely
  synchronized. NTP does this automatically, but you can do it by hand if you
  want.

* The exerciser needs to be able to ssh over to the cluster nodes as root
  without a password challenge. Configure ssh accordingly (see the Mini-HOWTO
  at the end of this document for more details).

* The exerciser needs to be able to resolve the machine names of the
  test cluster - either by DNS or by /etc/hosts.

	
Preparation
-----------

Install Pacemaker (including CTS) on all machines. These scripts are
coordinated with particular versions of Pacemaker, so you need the same version
of CTS as the rest of Pacemaker, and you need the same version of
pacemaker and CTS on both the test exerciser and the test cluster machines.

Configure cluster communications (Corosync, CMAN or Heartbeat) on the
cluster machines and verify everything works.

NOTE: Do not run the cluster on the test exerciser machine.

NOTE: Wherever machine names are mentioned in these configuration files,
they must match the machines' `uname -n` name.  This may or may not match
the machines' FQDN (fully qualified domain name) - it depends on how
you (and your OS) have named the machines. 


Run CTS
-------

Now assuming you did all this, what you need to do is run CTSlab.py:

    python ./CTSlab.py [options] number-of-tests-to-run

You must specify which nodes are part of the cluster with --nodes, e.g.:

   --node "pcmk-1 pcmk-2 pcmk-3"

Most people will want to save the output with --outputfile, e.g.:

   --outputfile ~/cts.log

Unless you want to test your pre-existing cluster configuration, you also want:

   --clobber-cib
   --populate-resources
   --test-ip-base $IP    # e.g. --test-ip-base 192.168.9.100

and configure some sort of fencing:

   --stonith $TYPE  # e.g. "--stonith rhcs" to use fence_xvm or "--stonith lha" to use external/ssh

A complete command line might look like:
  
  python ./CTSlab.py --nodes "pcmk-1 pcmk-2 pcmk-3" --outputfile ~/cts.log \
	 --clobber-cib --populate-resources --test-ip-base 192.168.9.100   \
	 --stonith rhcs 50

For more options, use the --help option.

To extract the result of a particular test, run:

  crm_report -T $test


Mini-HOWTO: Allow passwordless remote SSH connections
-----------------------------------------------------

The CTS scripts run "ssh -l root" so you don't have to do any of your testing
logged in as root on the test machine. Here is how to allow such connections
without requiring a password to be entered each time:

* On your test exerciser, create an SSH key if you do not already have one.
  Most commonly, SSH keys will be in your ~/.ssh directory, with the
  private key file not having an extension, and the public key file
  named the same with the extension ".pub" (for example, ~/.ssh/id_dsa.pub).

  If you don't already have a key, you can create one with:

		ssh-keygen -t dsa

* From your test exerciser, authorize your SSH public key for root on all test
  machines (both the exerciser and the cluster test machines):

	ssh-copy-id -i ~/.ssh/id_dsa.pub root@$MACHINE 

  You will probably have to provide your password, and possibly say
  "yes" to some questions about accepting the identity of the test machines.

  The above assumes you have a DSA SSH key in the specified location;
  if you have some other type of key (RSA, ECDSA, etc.), use its file name
  in the -i option above.

  If you have an old version of SSH that doesn't have ssh-copy-id,
  you can take the single line out of your public key file
  (e.g. ~/.ssh/identity.pub or ~/.ssh/id_dsa.pub) and manually add it to
  root's ~/.ssh/authorized_keys file on each test machine.

* To test, try this command from the exerciser machine for each
  of your cluster machines, and for the exerciser machine itself.

	ssh -l root $MACHINE

  If this works without prompting for a password, you're in business.
  If not, look at the documentation for your version of ssh.
