Greetings! :-)

This file is a continuation of the CentOS wiki page regarding writing tests for the QA process. For background information on writing tests for the t_functional QA process, please refer to:

	http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional

As a newcomer, you should read this document from start to finish. Questions/comments/suggestions should be voiced in the #centos-devel channel on Freenode IRC, or via email on the centos-devel@centos.org mailing list.

=== Introduction ===

What are the QA scripts?

------------------------

Small, self-contained test scripts that provide "component testing" of CentOS RPMs. These scripts verify packages install correctly via yum and ensure that whatever the package installs, works as expected.

What are they used for?

-----------------------

Quality assurance - making sure that each package installs and functions correctly on a given architecture. The CentOS QA process directly benefits from having a set of repeatable, automated tests to run against each distinct build and package as and when it's created.

When do they get used?

----------------------

As part of the QA process for every CentOS build prior to its testing/release and every time a CentOS supplied package is updated.

Where are they stored?

----------------------

In a publically available repository hosted on gitorious.org

What's in the repository?

-------------------------

Here's a breakdown:

tests/	:	contains all test scripts

tests/0_lib/	:	contains all the common functions and shared code for the tests all files in that directory are 'sourced' 

					before any of the tests are run, which also means it can only contain bash code ( no subdir allowed )

tests/0_common/	:	contain's tests that are run before any other test, and immediately after the 0_lib/ code is sourced.

					These should be tests that check system sanity and environment. These tests should also not leave behind any state or content 

					residue that would impact package and role specific tests that are run after

tests/p_<name>/	:	Each of the p_<name> directories would contain tests for that specific package. The <name> needs to be - rpm --qf "%{name}\n" 

					for the srpm.

tests/r_<role>	:	Each of the r_<role> directories should contain the tests specific to a role. eg: 'lamp'. The test harness looks at a file 

					called 'package_deps' inside each of the role directories and runs the role tests if any package listed in that file has been 

					changed / built etc.

					Role tests can be run with specific kickstarts. At the moment each role can have 1 kickstart file. It must be called 

					ks_<role>.cfg and it must be in the tests/r_<role>/ directory

What language are tests written in?

-----------------------------------

As of June 2011, all of test scripts are written in Bash. You're free to write test scripts in any language that's installable via yum - Python/Perl/Ruby etc. The only proviso is that, as a first step, you make a call (using a simple Bash script) to:

	t_InstallPackage <package_name> 

to install whatever package(s) need to be available for your subsequent, non-Bash test scripts to execute against. In short, at least some part of your test scripts will need to be in Bash.

What's t_installPackage?

------------------------

To promote a manageable level of consistency across the test suite, a handful of useful functions and variables have been consolidated into a small (but expanding) Bash library. Standard test script tasks such as logging, service control, package installation etc should (ideally) all be performed via calls to functions provided by the helper library. Unless there's a sound (and documented) reason for not doing so, use of the library should be preferred at all times.

Is the Bash library documented anywhere?

----------------------------------------

All of the functions available in the Bash library are fully documented using comments contained within the library itself, there's nothing particularly complicated or cryptic in the implementations, so you should be able to work out what's going on fairly easily. If the usage of anything in the library isn't obviously self-evident, please let us know. The library itself is by no means comprehensive and simply serves as the basis for writing consistent test scripts.

How do I use the library?

-------------------------

You should include the following statement in your test script:

	source ../0_lib/functions.sh

What return value/exit status should I return?

----------------------------------------------

Your scripts should exit with a status code of 0 to indicate success, and 1 to indicate failure. 

What environment is best for writing tests?

-------------------------------------------

Something with a working copy of git, a text editor to write tests in, and (preferably) a virtual machine environment so you can run tests/roll back/run tests/roll back etc etc.

NOTE:

Make sure that SELinux is enabled in VM and your test does not fail 

because of it.

What should I be testing for?

-----------------------------

First a foremost, that the packages you've chosen to test, install correctly. This should be the first thing your script does via a call to t_InstallPackage. t_InstallPackage will (as the name suggests) attempt to install the requested packages from the local QA repository via a call to yum. If the yum install process fails, t_InstallPackage will exit with a fail status code, and the test harness halts.

Assuming the package(s) install correctly, your scripts should then exercise the packages binaries in however way you see fit - start with something simple; perhaps just calling the binary with the --help switch and checking the exit status is correct (or grep'ing for expected words). Once you're comfortable with how the test scripts work, try something more inventive/advanced :-)

Is there an execution order?

----------------------------

Tests are executed in alphabetical order. Any files named 'readme' (case insensitive) or starting with '_' are ignored. On that basis, if you have any shared variables or config values that need a home, you could put them in a file named (for example) '_config' and refer to it from within your test scripts. You are of course free to keep everything inside a single file, but if it's a common value shared amongst your test scripts for a given package, it might make sense to separate things into a stand-alone file; whatever you believe is the most managable arrangement.

How much debugging output should I provide?

-------------------------------------------

You're free to produce as much debugging output as you feel is necessary to convey the actions your script is performing. If your script returns an exit status indicating failure, it's (obviously) a lot easier to decipher what went wrong if your script is emitting clear and concise messages. As a first step, each of your test scripts should make a call to t_Log (or similar, if you're not using Bash), including the name of your script and a short description of what you're testing for. For example 

	t_Log "Running $0 - checking procinfo runs and returns non-zero exit status."

What should I name my tests?

----------------------------

Scripts are processed in alphabetical order and grouped together into folders on a per-package basis. Package test folders should be named p_XXX where XXX matches the output of:

	rpm -q --qf "%{name}\n" <package_name>

Following the same approach, files within each package test folder are processed in alphabetical order. So (for example), tests that start with '0_' are processed before those starting with '5_', which are processed before those starting with '10_' etc. You should install any packages that your test requires in low-numbered scripts and then test against that package in incrementally higher scripts. If that makes no sense, see the "Hello World" example at the end of this document for a practical example. 

How do I test my tests?

-----------------------

In order to test your scripts in "stand-alone" mode, you'll need to perform the following command (assuming you're in the t_functional directory):

	source tests/0_lib/functions.sh

You can try executing the runtests.sh script found in t_functional, but some of the tests in 0_common will fail owing to the repo.centos.qa hosts being unreachable outside of the CentOS QA environment. You're welcome to remove the execute permissions from '00_qa_repo_config.sh` and '30_dns_works.sh` if you want to run the entire test suite.

What comments should I include in my tests?

-------------------------------------------

Start your tests with a comment block which includes your name and e-mail address. After that, make a call to t_Log, passing in $0 (or the non-Bash equivalent for your script's file name). Something like:

	t_Log "Running $0 - Postfix SMTP socket connect + 220 banner response test."

Make the comment relevant to what you are really testing. Some good examples are :

- php: testing mysql connection

- mysql: create database, load sample schema, drop database

- sshd: local connections over ssh work

- sshd: testing key based access is functional

Examples of not so good comments:

- php test

- does this work

- another test

If you are writing a test to satisfy or check for a regression or issue reported at bugs.centos.org, make sure you include reference to that issue number. One way to achieve that is in the test comment above. eg: to test if the issue reported in bugs.centos.org/view.php?id=4955 is being satisfied :

 t_Log "Running $0 - NTP should use the CentOS pool (#4955)"

=== A Practical Example ===

We'll now assemble all of the above information into a practical example, to help get you started. For the purposes of this example, we're going to stick to Bash - adapt as required based on your language of choice.

Firstly, get yourself a copy of the current testing repository. This is available via 

	https://gitorious.org/testautomation/t_functional

If you're not familiar with how git works, spend some time searching around the web for a couple of git tutorials to help you get comfortable with the concepts, terminology and execution.

Once you've got a working tree, it's time to pick a package. For the purposes of this example, we'll use [http://aide.sourceforge.net/ AIDE] - the Advanced Intrustion Detection Engine. 

First thing to do is create a folder for your package. Using the standard Linux `mkdir':

	cd t_functional/tests

	mkdir p_aide

Now that we have a home for our tests, we need to set about getting the package installed. Repeating the advice provided earlier, all test scripts are executed in alphabetical order so we'll put a call to t_InstallPackage in a file named '0-install-aide.sh'. Using your preferred editor:

	#!/bin/bash

	t_Log "$0 - installing AIDE"

	t_InstallPackage aide

That's all we need. Breaking this down, we start our script with a logging statement via t_Log. Nothing particularly special/complex going on there. Following on, we get our package installed via a call to a library provided function - t_InstallPackage. You don't need to check the return values from either of these functions. The t_InstallPackage function evaluates the exit status from yum and if there's a problem, will abort the test run.

Now to write a (very) simple test script to exercise the AIDE binary. Back to your editor, create a new file called 5-aide-basic-test.sh

	#!/bin/bash

	t_Log "$0 - basic AIDE initialisation test"

	AIDE=`which aide`

	[ -x "$AIDE" ] || { t_Log "FAIL: AIDE binary doesn't exist or isn't executable"; exit $FAIL; }

	# Perform an initialisation of the AIDE database

	$AIDE --init

	# Check for a 0 exit status

	t_CheckExitStatus $?

Again, nothing particularly complex here. The only thing probably worth explaining is the call to `t_CheckExitStatus', which is just a convenience wrapper around an evaluation of $? with 0. Using t_CheckExitStatus is the preferred means of evaluating exit codes from previously called functions. If the exit status isn't 0, a failure message is logged and the test harness halts.

That's it! :-)