diff --git a/doc/WritingTests b/doc/WritingTests index a07d376..86974a2 100644 --- a/doc/WritingTests +++ b/doc/WritingTests @@ -1,198 +1,7 @@ Greetings! :-) -This file is a continuation of the CentOS wiki page regarding writing tests for the QA process. For background information on writing tests for the t_functional QA process, please refer to: +This document has been moved to: http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional As a newcomer, you should read this document from start to finish. Questions/comments/suggestions should be voiced in the #centos-devel channel on Freenode IRC, or via email on the centos-devel@centos.org mailing list. - -=== Introduction === - -What are the QA scripts? ------------------------- - -Small, self-contained test scripts that provide "component testing" of CentOS RPMs. These scripts verify packages install correctly via yum and ensure that whatever the package installs, works as expected. - -What are they used for? ------------------------ - -Quality assurance - making sure that each package installs and functions correctly on a given architecture. The CentOS QA process directly benefits from having a set of repeatable, automated tests to run against each distinct build and package as and when it's created. - -When do they get used? ----------------------- - -As part of the QA process for every CentOS build prior to its testing/release and every time a CentOS supplied package is updated. - -Where are they stored? ----------------------- - -In a publically available repository hosted on gitorious.org - -What's in the repository? -------------------------- - -Here's a breakdown: - -tests/ : contains all test scripts - -tests/0_lib/ : contains all the common functions and shared code for the tests all files in that directory are 'sourced' - before any of the tests are run, which also means it can only contain bash code ( no subdir allowed ) - -tests/0_common/ : contain's tests that are run before any other test, and immediately after the 0_lib/ code is sourced. - These should be tests that check system sanity and environment. These tests should also not leave behind any state or content - residue that would impact package and role specific tests that are run after - -tests/p_/ : Each of the p_ directories would contain tests for that specific package. The needs to be - rpm --qf "%{name}\n" - for the srpm. - -tests/r_ : Each of the r_ directories should contain the tests specific to a role. eg: 'lamp'. The test harness looks at a file - called 'package_deps' inside each of the role directories and runs the role tests if any package listed in that file has been - changed / built etc. - - Role tests can be run with specific kickstarts. At the moment each role can have 1 kickstart file. It must be called - ks_.cfg and it must be in the tests/r_/ directory - -What language are tests written in? ------------------------------------ - -As of June 2011, all of test scripts are written in Bash. You're free to write test scripts in any language that's installable via yum - Python/Perl/Ruby etc. The only proviso is that, as a first step, you make a call (using a simple Bash script) to: - - t_InstallPackage - -to install whatever package(s) need to be available for your subsequent, non-Bash test scripts to execute against. In short, at least some part of your test scripts will need to be in Bash. - -What's t_installPackage? ------------------------- - -To promote a manageable level of consistency across the test suite, a handful of useful functions and variables have been consolidated into a small (but expanding) Bash library. Standard test script tasks such as logging, service control, package installation etc should (ideally) all be performed via calls to functions provided by the helper library. Unless there's a sound (and documented) reason for not doing so, use of the library should be preferred at all times. - -Is the Bash library documented anywhere? ----------------------------------------- - -All of the functions available in the Bash library are fully documented using comments contained within the library itself, there's nothing particularly complicated or cryptic in the implementations, so you should be able to work out what's going on fairly easily. If the usage of anything in the library isn't obviously self-evident, please let us know. The library itself is by no means comprehensive and simply serves as the basis for writing consistent test scripts. - -How do I use the library? -------------------------- - -You should include the following statement in your test script: - - source ../0_lib/functions.sh - -What return value/exit status should I return? ----------------------------------------------- - -Your scripts should exit with a status code of 0 to indicate success, and 1 to indicate failure. - -What environment is best for writing tests? -------------------------------------------- - -Something with a working copy of git, a text editor to write tests in, and (preferably) a virtual machine environment so you can run tests/roll back/run tests/roll back etc etc. - -NOTE: -Make sure that SELinux is enabled in VM and your test does not fail -because of it. - -What should I be testing for? ------------------------------ - -First a foremost, that the packages you've chosen to test, install correctly. This should be the first thing your script does via a call to t_InstallPackage. t_InstallPackage will (as the name suggests) attempt to install the requested packages from the local QA repository via a call to yum. If the yum install process fails, t_InstallPackage will exit with a fail status code, and the test harness halts. - -Assuming the package(s) install correctly, your scripts should then exercise the packages binaries in however way you see fit - start with something simple; perhaps just calling the binary with the --help switch and checking the exit status is correct (or grep'ing for expected words). Once you're comfortable with how the test scripts work, try something more inventive/advanced :-) - -Is there an execution order? ----------------------------- - -Tests are executed in alphabetical order. Any files named 'readme' (case insensitive) or starting with '_' are ignored. On that basis, if you have any shared variables or config values that need a home, you could put them in a file named (for example) '_config' and refer to it from within your test scripts. You are of course free to keep everything inside a single file, but if it's a common value shared amongst your test scripts for a given package, it might make sense to separate things into a stand-alone file; whatever you believe is the most managable arrangement. - -How much debugging output should I provide? -------------------------------------------- - -You're free to produce as much debugging output as you feel is necessary to convey the actions your script is performing. If your script returns an exit status indicating failure, it's (obviously) a lot easier to decipher what went wrong if your script is emitting clear and concise messages. As a first step, each of your test scripts should make a call to t_Log (or similar, if you're not using Bash), including the name of your script and a short description of what you're testing for. For example - - t_Log "Running $0 - checking procinfo runs and returns non-zero exit status." - -What should I name my tests? ----------------------------- - -Scripts are processed in alphabetical order and grouped together into folders on a per-package basis. Package test folders should be named p_XXX where XXX matches the output of: - - rpm -q --qf "%{name}\n" - -Following the same approach, files within each package test folder are processed in alphabetical order. So (for example), tests that start with '0_' are processed before those starting with '5_', which are processed before those starting with '10_' etc. You should install any packages that your test requires in low-numbered scripts and then test against that package in incrementally higher scripts. If that makes no sense, see the "Hello World" example at the end of this document for a practical example. - -How do I test my tests? ------------------------ - -In order to test your scripts in "stand-alone" mode, you'll need to perform the following command (assuming you're in the t_functional directory): - - source tests/0_lib/functions.sh - -You can try executing the runtests.sh script found in t_functional, but some of the tests in 0_common will fail owing to the repo.centos.qa hosts being unreachable outside of the CentOS QA environment. You're welcome to remove the execute permissions from '00_qa_repo_config.sh` and '30_dns_works.sh` if you want to run the entire test suite. - -What comments should I include in my tests? -------------------------------------------- - -Start your tests with a comment block which includes your name and e-mail address. After that, make a call to t_Log, passing in $0 (or the non-Bash equivalent for your script's file name). Something like: - - t_Log "Running $0 - Postfix SMTP socket connect + 220 banner response test." - -Make the comment relevant to what you are really testing. Some good examples are : -- php: testing mysql connection -- mysql: create database, load sample schema, drop database -- sshd: local connections over ssh work -- sshd: testing key based access is functional - -Examples of not so good comments: -- php test -- does this work -- another test - -If you are writing a test to satisfy or check for a regression or issue reported at bugs.centos.org, make sure you include reference to that issue number. One way to achieve that is in the test comment above. eg: to test if the issue reported in bugs.centos.org/view.php?id=4955 is being satisfied : - - t_Log "Running $0 - NTP should use the CentOS pool (#4955)" - -=== A Practical Example === - -We'll now assemble all of the above information into a practical example, to help get you started. For the purposes of this example, we're going to stick to Bash - adapt as required based on your language of choice. - -Firstly, get yourself a copy of the current testing repository. This is available via - - https://gitorious.org/testautomation/t_functional - -If you're not familiar with how git works, spend some time searching around the web for a couple of git tutorials to help you get comfortable with the concepts, terminology and execution. - -Once you've got a working tree, it's time to pick a package. For the purposes of this example, we'll use [http://aide.sourceforge.net/ AIDE] - the Advanced Intrustion Detection Engine. - -First thing to do is create a folder for your package. Using the standard Linux `mkdir': - - cd t_functional/tests - mkdir p_aide - -Now that we have a home for our tests, we need to set about getting the package installed. Repeating the advice provided earlier, all test scripts are executed in alphabetical order so we'll put a call to t_InstallPackage in a file named '0-install-aide.sh'. Using your preferred editor: - - #!/bin/bash - - t_Log "$0 - installing AIDE" - t_InstallPackage aide - -That's all we need. Breaking this down, we start our script with a logging statement via t_Log. Nothing particularly special/complex going on there. Following on, we get our package installed via a call to a library provided function - t_InstallPackage. You don't need to check the return values from either of these functions. The t_InstallPackage function evaluates the exit status from yum and if there's a problem, will abort the test run. - -Now to write a (very) simple test script to exercise the AIDE binary. Back to your editor, create a new file called 5-aide-basic-test.sh - - #!/bin/bash - - t_Log "$0 - basic AIDE initialisation test" - - AIDE=`which aide` - [ -x "$AIDE" ] || { t_Log "FAIL: AIDE binary doesn't exist or isn't executable"; exit $FAIL; } - - # Perform an initialisation of the AIDE database - $AIDE --init - - # Check for a 0 exit status - t_CheckExitStatus $? - -Again, nothing particularly complex here. The only thing probably worth explaining is the call to `t_CheckExitStatus', which is just a convenience wrapper around an evaluation of $? with 0. Using t_CheckExitStatus is the preferred means of evaluating exit codes from previously called functions. If the exit status isn't 0, a failure message is logged and the test harness halts. - -That's it! :-) - diff --git a/doc/first_steps_with_git b/doc/first_steps_with_git index 3bacd48..8ade006 100644 --- a/doc/first_steps_with_git +++ b/doc/first_steps_with_git @@ -1,112 +1,3 @@ -First Steps with Gitorious and git +This document has been moved to this wiki page: -This file is intended for beginners with git. - -A few details can be found at -http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional - -Hopefully this file will answer the remaining questions - -Git ---- -Probably all available options on git can be found with "man git". Specific help on an option ie "git fetch" can be found with "git fetch --help" - -Getting started with gitorious ------------------------------- - -First you need an account with gitorious: -https://gitorious.org/users/new -After registration you will receive an eMail to verify your eMail-Account. - -Afterwards you can login to gitorious - -SSH-Keys --------- -If you intend to use SSH with git, you have to register your public key with gitorious. Go to "Manage SSH keys" and then "Add SSH key". Copy the content of your key (home-directory/.ssh/id_rsa.pub or home-directory/.ssh/id_dsa.pub into the window and "Save". - -Cloning t_functional --------------------- -At https://gitorious.org/testautomation/t_functional you will find the button "Clone repository". Next you will be asked for a name for your clone - pick a name at your descretion or use the suggestion. Klick "Clone repository" and you have your own copy of t_functional with an SSH-URL that looks something like this: -git@gitorious.org:<>/testautomation/<>.git. -The HTTPS URL looks something like this: -https://gitorious.org/<>/testautomation/<> - -To work -------- -To work with your copy, you have to create a local clone on your linux box. -"yum -y install git" should install all you need to use git. - -Cloning: -"git clone <> <>" will create a clone of your gitorious-repo on your local box. You will find your clone at <>. -You should set at least two global settings: - -git config --global user.name "your username" -git config --global user.email "your email" - -All git operations work best if executed within your <>. - -Within the destination-dir you will find the folder .git. Go there and edit the config-file. Change your user-credentials and add: -[remote "upstream"] - url = https://git.gitorious.org/testautomation/t_functional.git - fetch = +refs/heads/*:refs/remotes/upstream/* -This allows you to update your clone from the t_functional-master in later steps. - -Now to work with the test scripts :) -Lots of good hints to start with can be found ind doc/WritingTests and in the allready available scripts. - -After you are satisfied with your work, it has to be commited. -New files will have to be added to git with "git add <>". You can also use "git add <>". -A final "git commit --all" will open a vi-like editor. Please enter some comments regarding your changes. After exiting the vi your changes are commit. -Use "git push" to push theses changes to gitorious. - -Merge request -------------- -Your push (or pushes) to git need to be merged with the t_functional master. This is done with the web-interface. On the webpage of your clone - something like https://gitorious.org/<>/testautomation/<> - press the button "Request merge". Add some comments/notes, scroll down and select your commits up to the point you want to merge (you can keep the dropdown menues unchanged). "Create merge request" will do just that. On https://gitorious.org/testautomation/t_functional you will now find the number of "Merge requests" being increased by 1. Clicking on "Merge Requests" will show you all current and previous requests (possible states are Open, In Review and Closed). - -Keeping in sync ---------------- -With time your gitorious clone and your local copy will drift from the t_functional master (you work on your copy and others work on their copies/master, pushes/commits and merge request happen and so on). The best way to deal with this is to do a local merge of the current master and your local copy: -"git fetch upstream" will fetch the current t_functional master from gitorious. -"git merge upstream/master" will now merge your local copy with the previously fetched master. This effectively writes all master changes to your local copy. - -If you work on two different machines - assuming you commit your work after it is done - you can do something similar to keep your second machine in sync with your work and not always have to delete your local copy and do a "git clone ..." again: -"git fetch origin" will fetch the current master of YOUR t_functional-clone from gitorious (not the t_functional master). -"git merge origin/master" will now update your local copy to the latest commit you pushed to gitorious. - -During these operations you will also see the "version" numbers of the various commits. i.e.: ->> -me@centos:~/t_functional>git fetch origin -remote: Counting objects: 17, done. -remote: Compressing objects: 100% (9/9), done. -remote: Total 9 (delta 6), reused 0 (delta 0) -Unpacking objects: 100% (9/9), done. -From gitorious.org:~<>/testautomation/<>s-t_functional - 4a5d56d..33951fe master -> origin/master ->> -The fetch operation downloaded the differences from 4a5d56d to 33951fe and put them in origin/master. - ->> -me@centos:~/t_functional>git merge origin/master -Updating 4a5d56d..33951fe -Fast-forward - tests/p_freeradius/0-install_freeradius.sh | 4 ++-- - tests/p_gcc/test_gcc.sh | 11 ++--------- - tests/p_php/10-php-test.sh | 3 +-- - 3 files changed, 5 insertions(+), 13 deletions(-) ->> -Now my local copy should match the data available online. - -At the time of writing this 33951fe is also the latest version available on the t_functional - master so: ->> -me@centos:~/t_functional> git merge upstream/master -Already up-to-date. ->> -There is no difference between the t_functional clone and the t_functiomal master so 'Already up-to-date' (as the last push to the master was the authors last commit to his clone and a subsequent merge request). - -Last ressort ------------- - -If - for any reason - you are unable to sync your repos correctly, you can always delete your clone at gitorious and -create a new one. - -Happy git'ing and testing! +http://wiki.centos.org/QaWiki/AutomatedTests/GettingStartedWithGitorious diff --git a/tests/0_common/01_dist_release_check.sh b/tests/0_common/01_dist_release_check.sh new file mode 100644 index 0000000..71a514c --- /dev/null +++ b/tests/0_common/01_dist_release_check.sh @@ -0,0 +1,5 @@ +#!/bin/bash +# Just a check to determine full version (example 5.8) or just dist (example 6) + +export qa_dist=$(rpm -q --queryformat '%{version}\n' centos-release) +export qa_releasever=$(rpm -q --queryformat '%{version}.' centos-release ; rpm -q --queryformat '%{release}\n' centos-release|cut -f 1 -d '.') diff --git a/tests/p_acl/0-install_acl.sh b/tests/p_acl/0-install_acl.sh index 91953a3..be914c2 100755 --- a/tests/p_acl/0-install_acl.sh +++ b/tests/p_acl/0-install_acl.sh @@ -4,3 +4,6 @@ t_Log "$0 - installing acl" t_InstallPackage acl +t_Log "Remount root fs with acl support" +mount -o remount,acl / +sleep 2 diff --git a/tests/p_bind/0-install_bind.sh b/tests/p_bind/0-install_bind.sh index 2554262..028daf1 100755 --- a/tests/p_bind/0-install_bind.sh +++ b/tests/p_bind/0-install_bind.sh @@ -31,7 +31,7 @@ EOF cp /usr/share/doc/bind-*/sample/etc/named.rfc1912.zones /etc/ cp -R /usr/share/doc/bind-*/sample/var/named/local* /var/named/ cp -R /usr/share/doc/bind-*/sample/var/named/named.* /var/named/ - chown -R named:named /var/named/* /etc/named.* + chown -R root:named /var/named/* /etc/named.* fi t_ServiceControl named start diff --git a/tests/p_busybox/00_install_busybox.sh b/tests/p_busybox/00_install_busybox.sh new file mode 100755 index 0000000..b2cae54 --- /dev/null +++ b/tests/p_busybox/00_install_busybox.sh @@ -0,0 +1,6 @@ +#!/bin/sh +# Author: Christoph Galuschka + +t_Log "Running $0 - attempting to install busybox." +t_InstallPackage busybox + diff --git a/tests/p_busybox/10_test_busybox.sh b/tests/p_busybox/10_test_busybox.sh new file mode 100755 index 0000000..6e59516 --- /dev/null +++ b/tests/p_busybox/10_test_busybox.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# Author: Athmane Madjoudj +# Author: Christoph Galuschka + +t_Log "Running $0 - busybox test: busybox lists available functions." + +busybox | grep -q 'Currently defined functions' +busy_ok=$? +if [ $busy_ok = 1 ] + then + t_Log 'busybox does not seem to list available functions' +fi +t_CheckExitStatus $busy_ok diff --git a/tests/p_busybox/20_functiontest_busybox.sh b/tests/p_busybox/20_functiontest_busybox.sh new file mode 100755 index 0000000..171b1de --- /dev/null +++ b/tests/p_busybox/20_functiontest_busybox.sh @@ -0,0 +1,44 @@ +#!/bin/sh +# Author: Athmane Madjoudj +# Author: Christoph Galuschka + +t_Log "Running $0 - busybox functional tests: busybox provided functions are working." + + +ret_val=0 +busybox | grep -q 'ls,' +if [ $? == 0 ] + then + t_Log "busybox provides 'ls'; testing it" + touch /var/tmp/busybox + busybox ls /var/tmp/ |grep -q busybox + if [ $? == 1 ] + then + t_Log "busybox provides 'ls' but it does not seem to work" + ret_val=1 + else + t_Log "'ls' works" + fi + #cleaning up + /bin/rm /var/tmp/busybox +fi + +busybox | grep -q 'touch,' +if [ $? == 0 ] + then + t_Log "busybox provides 'touch'; testing it" + busybox touch /var/tmp/busybox + ls /var/tmp/ |grep -q busybox + if [ $? == 1 ] + then + t_Log "busybox provides 'touch' but it does not seem to work" + ret_val=1 + else + t_Log "'touch' works" + fi + #cleaning up + /bin/rm /var/tmp/busybox +fi + +t_CheckExitStatus $ret_val + diff --git a/tests/p_tftp-server/20-tftp-server_put-test.sh b/tests/p_tftp-server/20-tftp-server_put-test.sh index 29a4136..d726e5b 100755 --- a/tests/p_tftp-server/20-tftp-server_put-test.sh +++ b/tests/p_tftp-server/20-tftp-server_put-test.sh @@ -1,25 +1,31 @@ #!/bin/sh # Author: Athmane Madjoudj -# Christoph Galuschka +# Christoph Galuschka +# Fabian Arrotin t_Log "Running $0 - tftp-server put file test." -if (t_GetPkgRel basesystem | grep -q el6) -then - TFTP_DIR=/var/lib/tftpboot +dist=$(t_DistCheck) + +if [ "$dist" = "6" ]; then + setsebool tftp_anon_write 1 + TFTP_DIR=/var/lib/tftpboot +elif [ "$dist" = "5" ]; then + setsebool allow_tftp_anon_write 1 + chcon -R -t tftpdir_rw_t /tftpboot/ + TFTP_DIR=/tftpboot else - TFTP_DIR=/tftpboot + TFTP_DIR=/tftpboot fi + chmod 777 $TFTP_DIR echo "t_functional_test" > put_test touch $TFTP_DIR/put_test chmod 666 $TFTP_DIR/put_test -# Fix selinux bool -setsebool tftp_anon_write 1 -tftp 127.0.0.1 -c put put_test +tftp 127.0.0.1 -c put put_test cat $TFTP_DIR/put_test | grep -q 't_functional_test' ret_val=$? diff --git a/utils/cleanup_vm.sh b/utils/cleanup_vm.sh index 4832fe3..3c529a6 100644 --- a/utils/cleanup_vm.sh +++ b/utils/cleanup_vm.sh @@ -32,6 +32,10 @@ done # Clean pgsql data dir (rpm -q postgresql | grep -q el6) && /bin/rm -rf /var/lib/pgsql/ +# Clean tmp +rm -rf /tmp/* +rm -rf /var/tmp/* + # Keep yum cache sed -i 's/keepcache=0/keepcache=1/' /etc/yum.conf