Blame doc/WritingTests

Grip Firmly 44e7f7
Greetings! :-)
Grip Firmly 44e7f7
Grip Firmly 44e7f7
This file is a continuation of the CentOS wiki page regarding writing tests for the QA process. For background information on writing tests for the t_functional QA process, please refer to:
Grip Firmly 44e7f7
Christoph Galuschka 3bd55b
	http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional
Grip Firmly 44e7f7
Grip Firmly 44e7f7
As a newcomer, you should read this document from start to finish. Questions/comments/suggestions should be voiced in the #centos-devel channel on Freenode IRC, or via email on the centos-devel@centos.org mailing list.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
=== Introduction ===
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What are the QA scripts?
Grip Firmly 44e7f7
------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Small, self-contained test scripts that provide "component testing" of CentOS RPMs. These scripts verify packages install correctly via yum and ensure that whatever the package installs, works as expected.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What are they used for?
Grip Firmly 44e7f7
-----------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Quality assurance - making sure that each package installs and functions correctly on a given architecture. The CentOS QA process directly benefits from having a set of repeatable, automated tests to run against each distinct build and package as and when it's created.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
When do they get used?
Grip Firmly 44e7f7
----------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
As part of the QA process for every CentOS build prior to its testing/release and every time a CentOS supplied package is updated.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Where are they stored?
Grip Firmly 44e7f7
----------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
In a publically available repository hosted on gitorious.org
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What's in the repository?
Grip Firmly 44e7f7
-------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Here's a breakdown:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
tests/	:	contains all test scripts
Grip Firmly 44e7f7
Grip Firmly 44e7f7
tests/0_lib/	:	contains all the common functions and shared code for the tests all files in that directory are 'sourced' 
Grip Firmly 44e7f7
					before any of the tests are run, which also means it can only contain bash code ( no subdir allowed )
Grip Firmly 44e7f7
Grip Firmly 44e7f7
tests/0_common/	:	contain's tests that are run before any other test, and immediately after the 0_lib/ code is sourced.
Grip Firmly 44e7f7
					These should be tests that check system sanity and environment. These tests should also not leave behind any state or content 
Grip Firmly 44e7f7
					residue that would impact package and role specific tests that are run after
Grip Firmly 44e7f7
Grip Firmly 44e7f7
tests/p_<name>/	:	Each of the p_<name> directories would contain tests for that specific package. The <name> needs to be - rpm --qf "%{name}\n" 
Grip Firmly 44e7f7
					for the srpm.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
tests/r_<role>	:	Each of the r_<role> directories should contain the tests specific to a role. eg: 'lamp'. The test harness looks at a file 
Grip Firmly 44e7f7
					called 'package_deps' inside each of the role directories and runs the role tests if any package listed in that file has been 
Grip Firmly 44e7f7
					changed / built etc.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
					Role tests can be run with specific kickstarts. At the moment each role can have 1 kickstart file. It must be called 
Grip Firmly 44e7f7
					ks_<role>.cfg and it must be in the tests/r_<role>/ directory
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What language are tests written in?
Grip Firmly 44e7f7
-----------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
As of June 2011, all of test scripts are written in Bash. You're free to write test scripts in any language that's installable via yum - Python/Perl/Ruby etc. The only proviso is that, as a first step, you make a call (using a simple Bash script) to:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	t_InstallPackage <package_name> 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
to install whatever package(s) need to be available for your subsequent, non-Bash test scripts to execute against. In short, at least some part of your test scripts will need to be in Bash.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What's t_installPackage?
Grip Firmly 44e7f7
------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
To promote a manageable level of consistency across the test suite, a handful of useful functions and variables have been consolidated into a small (but expanding) Bash library. Standard test script tasks such as logging, service control, package installation etc should (ideally) all be performed via calls to functions provided by the helper library. Unless there's a sound (and documented) reason for not doing so, use of the library should be preferred at all times.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Is the Bash library documented anywhere?
Grip Firmly 44e7f7
----------------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
All of the functions available in the Bash library are fully documented using comments contained within the library itself, there's nothing particularly complicated or cryptic in the implementations, so you should be able to work out what's going on fairly easily. If the usage of anything in the library isn't obviously self-evident, please let us know. The library itself is by no means comprehensive and simply serves as the basis for writing consistent test scripts.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
How do I use the library?
Grip Firmly 44e7f7
-------------------------
Grip Firmly 44e7f7
Christoph Galuschka 619e64
You should include the following statement in your test script:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	source ../0_lib/functions.sh
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What return value/exit status should I return?
Grip Firmly 44e7f7
----------------------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Your scripts should exit with a status code of 0 to indicate success, and 1 to indicate failure. 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What environment is best for writing tests?
Grip Firmly 44e7f7
-------------------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Something with a working copy of git, a text editor to write tests in, and (preferably) a virtual machine environment so you can run tests/roll back/run tests/roll back etc etc.
Grip Firmly 44e7f7
Athmane Madjoudj d63384
NOTE:
Athmane Madjoudj d63384
Make sure that SELinux is enabled in VM and your test does not fail 
Athmane Madjoudj d63384
because of it.
Athmane Madjoudj d63384
Grip Firmly 44e7f7
What should I be testing for?
Grip Firmly 44e7f7
-----------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
First a foremost, that the packages you've chosen to test, install correctly. This should be the first thing your script does via a call to t_InstallPackage. t_InstallPackage will (as the name suggests) attempt to install the requested packages from the local QA repository via a call to yum. If the yum install process fails, t_InstallPackage will exit with a fail status code, and the test harness halts.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Assuming the package(s) install correctly, your scripts should then exercise the packages binaries in however way you see fit - start with something simple; perhaps just calling the binary with the --help switch and checking the exit status is correct (or grep'ing for expected words). Once you're comfortable with how the test scripts work, try something more inventive/advanced :-)
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Is there an execution order?
Grip Firmly 44e7f7
----------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Tests are executed in alphabetical order. Any files named 'readme' (case insensitive) or starting with '_' are ignored. On that basis, if you have any shared variables or config values that need a home, you could put them in a file named (for example) '_config' and refer to it from within your test scripts. You are of course free to keep everything inside a single file, but if it's a common value shared amongst your test scripts for a given package, it might make sense to separate things into a stand-alone file; whatever you believe is the most managable arrangement.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
How much debugging output should I provide?
Grip Firmly 44e7f7
-------------------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
You're free to produce as much debugging output as you feel is necessary to convey the actions your script is performing. If your script returns an exit status indicating failure, it's (obviously) a lot easier to decipher what went wrong if your script is emitting clear and concise messages. As a first step, each of your test scripts should make a call to t_Log (or similar, if you're not using Bash), including the name of your script and a short description of what you're testing for. For example 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	t_Log "Running $0 - checking procinfo runs and returns non-zero exit status."
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What should I name my tests?
Grip Firmly 44e7f7
----------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Scripts are processed in alphabetical order and grouped together into folders on a per-package basis. Package test folders should be named p_XXX where XXX matches the output of:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	rpm -q --qf "%{name}\n" <package_name>
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Following the same approach, files within each package test folder are processed in alphabetical order. So (for example), tests that start with '0_' are processed before those starting with '5_', which are processed before those starting with '10_' etc. You should install any packages that your test requires in low-numbered scripts and then test against that package in incrementally higher scripts. If that makes no sense, see the "Hello World" example at the end of this document for a practical example. 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
How do I test my tests?
Grip Firmly 44e7f7
-----------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
In order to test your scripts in "stand-alone" mode, you'll need to perform the following command (assuming you're in the t_functional directory):
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	source tests/0_lib/functions.sh
Grip Firmly 44e7f7
Grip Firmly 44e7f7
You can try executing the runtests.sh script found in t_functional, but some of the tests in 0_common will fail owing to the repo.centos.qa hosts being unreachable outside of the CentOS QA environment. You're welcome to remove the execute permissions from '00_qa_repo_config.sh` and '30_dns_works.sh` if you want to run the entire test suite.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
What comments should I include in my tests?
Grip Firmly 44e7f7
-------------------------------------------
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Start your tests with a comment block which includes your name and e-mail address. After that, make a call to t_Log, passing in $0 (or the non-Bash equivalent for your script's file name). Something like:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	t_Log "Running $0 - Postfix SMTP socket connect + 220 banner response test."
Grip Firmly 44e7f7
Karanbir Singh a19219
Make the comment relevant to what you are really testing. Some good examples are :
Karanbir Singh a19219
- php: testing mysql connection
Karanbir Singh a19219
- mysql: create database, load sample schema, drop database
Karanbir Singh a19219
- sshd: local connections over ssh work
Karanbir Singh a19219
- sshd: testing key based access is functional
Karanbir Singh a19219
Karanbir Singh a19219
Examples of not so good comments:
Karanbir Singh a19219
- php test
Karanbir Singh a19219
- does this work
Karanbir Singh a19219
- another test
Karanbir Singh a19219
Karanbir Singh a19219
If you are writing a test to satisfy or check for a regression or issue reported at bugs.centos.org, make sure you include reference to that issue number. One way to achieve that is in the test comment above. eg: to test if the issue reported in bugs.centos.org/view.php?id=4955 is being satisfied :
Karanbir Singh a19219
Karanbir Singh a19219
 t_Log "Running $0 - NTP should use the CentOS pool (#4955)"
Grip Firmly 44e7f7
Grip Firmly 44e7f7
=== A Practical Example ===
Grip Firmly 44e7f7
Grip Firmly 44e7f7
We'll now assemble all of the above information into a practical example, to help get you started. For the purposes of this example, we're going to stick to Bash - adapt as required based on your language of choice.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Firstly, get yourself a copy of the current testing repository. This is available via 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	https://gitorious.org/testautomation/t_functional
Grip Firmly 44e7f7
Grip Firmly 44e7f7
If you're not familiar with how git works, spend some time searching around the web for a couple of git tutorials to help you get comfortable with the concepts, terminology and execution.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Once you've got a working tree, it's time to pick a package. For the purposes of this example, we'll use [http://aide.sourceforge.net/ AIDE] - the Advanced Intrustion Detection Engine. 
Grip Firmly 44e7f7
Grip Firmly 44e7f7
First thing to do is create a folder for your package. Using the standard Linux `mkdir':
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	cd t_functional/tests
Grip Firmly 44e7f7
	mkdir p_aide
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Now that we have a home for our tests, we need to set about getting the package installed. Repeating the advice provided earlier, all test scripts are executed in alphabetical order so we'll put a call to t_InstallPackage in a file named '0-install-aide.sh'. Using your preferred editor:
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	#!/bin/bash
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	t_Log "$0 - installing AIDE"
Grip Firmly 44e7f7
	t_InstallPackage aide
Grip Firmly 44e7f7
Grip Firmly 44e7f7
That's all we need. Breaking this down, we start our script with a logging statement via t_Log. Nothing particularly special/complex going on there. Following on, we get our package installed via a call to a library provided function - t_InstallPackage. You don't need to check the return values from either of these functions. The t_InstallPackage function evaluates the exit status from yum and if there's a problem, will abort the test run.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Now to write a (very) simple test script to exercise the AIDE binary. Back to your editor, create a new file called 5-aide-basic-test.sh
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	#!/bin/bash
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	t_Log "$0 - basic AIDE initialisation test"
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	AIDE=`which aide`
Grip Firmly 44e7f7
	[ -x "$AIDE" ] || { t_Log "FAIL: AIDE binary doesn't exist or isn't executable"; exit $FAIL; }
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	# Perform an initialisation of the AIDE database
Grip Firmly 44e7f7
	$AIDE --init
Grip Firmly 44e7f7
Grip Firmly 44e7f7
	# Check for a 0 exit status
Grip Firmly 44e7f7
	t_CheckExitStatus $?
Grip Firmly 44e7f7
Grip Firmly 44e7f7
Again, nothing particularly complex here. The only thing probably worth explaining is the call to `t_CheckExitStatus', which is just a convenience wrapper around an evaluation of $? with 0. Using t_CheckExitStatus is the preferred means of evaluating exit codes from previously called functions. If the exit status isn't 0, a failure message is logged and the test harness halts.
Grip Firmly 44e7f7
Grip Firmly 44e7f7
That's it! :-)
Grip Firmly 44e7f7