For CentOS 7/8, we had sig-core-t_functional as part of the CentOS Automated QA test process. We were able to submit tests and they'd be run before packages were pushed to the public repos, which is nice.
For CentOS Stream 9, nothing like this currently exists (AFAIK). I know that package-specific tests are now supposed to live in Gitlab next to the package sources, which makes sense. However, I don't see where those tests are actually run. For example, I can't see any curl tests being run in Gitlab. I've also heard mentions of Zuul, but either I have the wrong URL or it doesn't run any tests either.
Besides tests for individual packages, it would be great to have somewhere to run more complete integration tests such as "are all package groups installable?" or even "can a basic VM be installed?". Community members could then contribute tests to be run before packages are released to the public repos, which will hopefully reduce the likelihood of shipping broken packages.
I'm aware that these sorts of tests are currently run behind closed doors on the Red Hat side, but that's not very useful for the CentOS Stream community. These tests should be run publicly and there should be a way for community members to collaborate and submit tests.
I would like to know what the CentOS Project Board's position is on this, if it is a priority and what the timelines are for implementing full public testing for CentOS Stream.
Tagging @bookwar
Let me clarify the current state of the MR testing:
1) dist-git integration tests on MRs
We had STI/TMT support for a while, but it was an opt-in. Only recently we have added the configuration to enable the pipeline for TMT/STI tests for all centos stream packages:
https://gitlab.com/redhat/centos-stream/ci-cd/zuul/jobs-config/-/merge_requests/57
From that moment CentOS Stream Zuul will run mock-build, rpminspect but also additionally integration tests for every Merge Request to CentOS stream, if those integration tests are described in dist-git in TMT or STI format, the same way as they work in Fedora:
https://docs.fedoraproject.org/en-US/ci/tmt/
See example in https://gitlab.com/redhat/centos-stream/rpms/osbuild-composer/-/merge_requests/47
Since the tests are located in dist-git we encourage contribution to them the same way as people would contribute changes to the packages themselves. In the ideal scenario any change/bugfix to a package spec files should come with a test in the same MR to cover new functionality it adds.
2) One can also introduce new generic tests to Zuul or tweak current ones. Its configuration is open and all contributions are welcome:
https://gitlab.com/redhat/centos-stream/ci-cd/zuul/jobs/-/blob/master/zuul.d/jobs.yaml
RHEL OSCI team is discussing the possibility to add test-compose or rpmdeplint pipelines but it is yet a work in progress.
3) third-party CI
OpenStack/Triple-O folks have contributed additional Zuul pipeline which is called third-party-check which can be additionally triggered to run external test on CentOS MRs.
third-party-check
The idea is that if a certain third-party would like to run additional tests on their own infrastructure, for example custom hardware lab, they can hook these tests into the third-party-check pipeline, which will be non-blocking but still reporting test results for maintainers. This is an MVP and if there is an interest to use it, please reach out and we can work together to set it up.
I'm not familiar with TMT, but I don't see the tests there? (I'm looking for .fmf files, I guess that's the wrong thing to look for?)
In any case, where do those tests run and where can I see their output?
2) One can also introduce new generic tests to Zuul or tweak current ones. Its configuration is open and all contributions are welcome: https://gitlab.com/redhat/centos-stream/ci-cd/zuul/jobs/-/blob/master/zuul.d/jobs.yaml RHEL OSCI team is discussing the possibility to add test-compose or rpmdeplint pipelines but it is yet a work in progress.
This is all for per-package tests, right? How could we run a test to validate that the repository metadata is correct before it's distributed to mirrors? How could we try to install a VM with the packages about to be released to see if the basic stuff works?
This is interesting, but I think we first need some public, common and blocking integration testing to filter basic regressions and bugs.
I'm not familiar with TMT, but I don't see the tests there? (I'm looking for .fmf files, I guess that's the wrong thing to look for?) In any case, where do those tests run and where can I see their output?
The tests are in ./tests in the repo:
https://gitlab.com/redhat/centos-stream/rpms/osbuild-composer/-/tree/c9s/tests
There is a comment from Zuul on the MR https://gitlab.com/redhat/centos-stream/rpms/osbuild-composer/-/merge_requests/47#note_1157378229 which says: rpm-tmt-test https://centos.softwarefactory-project.io/zuul/t/centos/build/bd9e3c78cb654785a1c01393604b66cb : SUCCESS in 23m 57s (non-voting)
rpm-tmt-test https://centos.softwarefactory-project.io/zuul/t/centos/build/bd9e3c78cb654785a1c01393604b66cb : SUCCESS in 23m 57s (non-voting)
if you go by that link you get to Zuul interface and in Artifacts tab you can find link to the Testing Farm logs:
https://centos.softwarefactory-project.io/zuul/t/centos/build/bd9e3c78cb654785a1c01393604b66cb/artifacts
The above indeed works on Merge Requests per component. It still counts as integration testing though, because to run a test we create a VM with latest CentOS Stream state, install the updated rpm on it and then run a testsuite on the installed system.
But I agree that we additionally need a separate public and extendable test interface for composes (snapshots of the repositories and images). I think the request to the CentOS Board is valid.
I asked similar question time ago https://bugzilla.redhat.com/show_bug.cgi?id=2023303
From Cloud SIG, the most interesting part would be to contribute tests somehow to validate composes, more that in specific MRs on distgits.
https://testing.stream.centos.org/ is live now and based on t_functional. The long term plan is still in a bit of flux @bstinson anything you'd like to add to this update?
https://github.com/CentOS/sig-core-t_functional/ seems to be the place to start
Documentation needed to complete this issue.
Metadata Update from @spotz: - Issue assigned to ngompa
https://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional
Metadata Update from @spotz: - Issue assigned to bstinson (was: ngompa)
Loop in the Integration SiG this might be something for them to lead
This can be closed, Integration SiG has taken this issue over
Metadata Update from @spotz: - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.