diff --git a/README.md b/README.md index 801fcfa..5236a69 100644 --- a/README.md +++ b/README.md @@ -11,3 +11,24 @@ See the doc directory for additional information on test writing. | CentOS 6 - i386 | [![Build Status](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c6-32/badge/icon)](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c6-32/) | | CentOS 5 - x86_64 | [![Build Status](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c5-64/badge/icon)](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c5-64/) | | CentOS 5 - i386 | [![Build Status](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c5-32/badge/icon)](https://ci.centos.org/job/CentOS-Core-QA-t_functional-c5-32/) | + +## Running tests + +To run these tests on your local machine : +``` +./runtests.sh +``` + +To only run a specific test ( eg. p_openssh ) : +``` +./runtests.sh p_openssh +``` +## Writing tests + +There is a dedicated [wiki page](http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional) covering that. As a newcomer, you should read this document from start to finish. +Questions/comments/suggestions should be voiced in the #centos-devel channel on Freenode IRC, or via email on the centos-devel@centos.org mailing list. + +## Disabling tests + +While it's a very bad idea, sometimes , during major.minor release, our scripts really find issues that are then repored upstream. +For the time being, one can add tests to be skipped by our QA harness setup (validating all new installable trees)See the [skipped-tests.list](skipped-tests.list) file. diff --git a/doc/RunningTests b/doc/RunningTests deleted file mode 100644 index f873730..0000000 --- a/doc/RunningTests +++ /dev/null @@ -1,5 +0,0 @@ -To run these tests on your local machine : -./runtests.sh - -To only run a specific test ( eg. p_openssh ) : -./runtests.sh p_openssh diff --git a/doc/TODO b/doc/TODO deleted file mode 100644 index f74e860..0000000 --- a/doc/TODO +++ /dev/null @@ -1,11 +0,0 @@ - -- We also need a way to dictate if a role_ should be run on realiron, - virtual machine host, virtual machine ( and maybe even the type of virt - being used! ) - -- Find a solution on how we test desktop applications such as using: - 1. GNU LDTP (Linux Desktop Testing Project) - 2. Dogtail - -- Make a list of the top 200 most often downloaded apps from mirror.centos.org - and see if we can get 100% coverage for them; then expand to top 500 diff --git a/doc/WritingTests b/doc/WritingTests deleted file mode 100644 index 86974a2..0000000 --- a/doc/WritingTests +++ /dev/null @@ -1,7 +0,0 @@ -Greetings! :-) - -This document has been moved to: - - http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional - -As a newcomer, you should read this document from start to finish. Questions/comments/suggestions should be voiced in the #centos-devel channel on Freenode IRC, or via email on the centos-devel@centos.org mailing list. diff --git a/runtests.sh b/runtests.sh index 257d2aa..38fa79d 100755 --- a/runtests.sh +++ b/runtests.sh @@ -29,6 +29,16 @@ set -e # exit on undefined variables set -u +# Searching for tests to disable +if [ -e skipped-tests.list ] ;then + t_Log "QA Harness : searching for tests to disable with valid reason" + egrep ^${centos_ver} skipped-tests.list | while read line; + do test=$(echo $line|cut -f 2 -d '|') + t_Log "Disabling QA harness test ${test}" + chmod -x ${test} + done +fi + # process our test scripts if [ $# -gt 0 ]; then t_Process <(/usr/bin/find ./tests/0_*/ -type f|sort -t'/' ) @@ -41,5 +51,14 @@ else fi # and, we're done. +if [ -e skipped-tests.list ] ;then + t_Log "QA Harness : Searching for disabled tests (skipped-tests.list)" + egrep ^${centos_ver} skipped-tests.list | while read line; + do test=$(echo $line|cut -f 2 -d '|') + reason=$(echo $line|cut -f 3 -d '|') + t_Log " =WARNING= : Disabled test : ${test} (${reason})" + done +fi + t_Log "QA t_functional tests finished." exit 0 diff --git a/skipped-tests.list b/skipped-tests.list new file mode 100644 index 0000000..ce3167e --- /dev/null +++ b/skipped-tests.list @@ -0,0 +1,10 @@ +# This file contains list of tests we need/want to skip +# Reason is when there is upstream bug that we're aware of +# So this file should contain: +# - centos version (using $centos_ver) +# - test to skip (tests/p_${name}/test.sh) +# - reason why it's actually skipped (url to upstream BZ, or bug report) +# Separated by | +8|tests/p_gzip/30-gzexe-test|https://apps.centos.org/kanboard/project/23/task/833 +8|tests/p_diffutils/10-cmp-tests|https://bugzilla.redhat.com/show_bug.cgi?id=1732960 +