release notes for v6.5
we announced several plugins which will fetch test names and execution results
from your automated test suite.
Plugins can be controlled via environment variables which will affect how
test results are recorded in the Kiwi TCMS database! This blog post is an introduction
how that works and what you can do with it! For this purpose I will be using the
plugin-demo repository, which simulates
real development work.
Full documentation and list of available plugins is available in chapter
Automation Frameworks Plugins!
Always create new TestRun by default
The default behavior is always to create a new TestRun if controlling variables
are not overridden! Product name, version and build will receive default values
if tests are running in Travis CI or Jenkins. For example
Travis Build #2 for
creates TR-12 and
Next we convert the README file from Markdown to reStructuredText which triggers
Travis Build #3 for
This build again creates a new TestRun and new TestPlan for it. Respectively
Important: we can see that version is different which will affect how artifacts
are organized in Kiwi TCMS, possibly affect how you will report status to stakeholders!
Override ENV variables for more control
Let's say the QA team has decided that all test results must be reported under the
same TestPlan. This means version must be the same between various builds in Travis CI!
To control this we export
TCMS_PRODUCT_VERSION=master in CI before executing
the TAP plugin! Checkout the commit on GitHub to see how it is done!
Travis Build #4 for
Because this is the first time where Version == master the plugin creates
TP-14 and reports the results under
Right after that I realized we can make this configuration a bit more generic
because our team is planning to support multiple versions of the product and
development will be done in separate branches!
Travis Build #5 for
still ends up with Version == master because we are still working on the
branch! That is to say, the product is in active mode of development.
Results are reported in
TR-18 which is again part of
Travis Build #6 for
adds the new functionality README badges and reports test results in
TR-20 which is again part of
More ENV overrides
While giving status reports back to stakeholders and developers the information that
we have in the TestRun is Build number! This follows the numbering scheme in Travis CI
(or Jenkins job number) and isn't very useful.
TCMS_BUILD to be the first 7 characters of the commit hash! When QA
tells devel that something isn't working and redirects them to the TestRun they can
immediately use the Build information and git checkout the offending variant of the product
This results in
Travis Build #7 for
Report results in pre-existing TestRun
There are many reasons you may want to do this:
- include both manual and automation tests for the same build;
- mix results from multiple test suites for the same build,
e.g. unit, functional, performance
- mix results from multiple but similar platforms in the same build,
e.g. cross-platform application for iOS and Android
To do so I've created TR-24 beforehand and
TCMS_RUN_ID=24 in my CI environment. TR-24 also contains
TC-57: Verify we can report results from several plugins into the same TR.
this was created and added via the web interface.
Note: mixing additional test cases can be done either before or after
automation results are reported with the plugin!
Important: when reporting results to an existing TestRun Kiwi TCMS plugins
don't care in which TestPlan this TR is! In theory it is possible to report
the results for
Product=kiwitcms/plugin-demo into any TP/TR pair! However we
don't want to do this crazy thing and instead I've created TR-24 inside the already
Note: because I don't know what is the git commit beforehand I've configured
Build=unspecified. If I wanted I could update this with the correct value
once I know the commit hash for the related changes I am testing.
Important: the plugin-demo repository uses both kiwitcms-tap-plugin and
kiwitcms-junit.xml-plugin at the same time! They differ in the way test names
are compiled! Both are reported in TR-24. See TC-57 text for information how to
distinguish between the two.
Travis Build #8 for
Also check-out comments in TR executions to see when and who had executed the test case.
So far there have been some tests which were failing (although Travis reports PASS) so
I decided to fix them.
Travis Build #9 for
is still configured with
TCMS_RUN_ID=24. This means results will be reported in
TR-24, effectively overwriting previous results.
Check-out Change Log under each individual execution and you will see several
time stamps when status was updated! In other words, as long as
pointing to an existing TestRun this TR will keep the results of the last test suite execution!
Continue development, restore ENV configuration
Travis Build #10 for
TCMS_RUN_ID! Results are reported in
Branch out for an LTS version
Our team has decided to make the first LTS release of the product. We branch out into
lts-v0 branch in git and cut the first NVR. This results in
Travis Build #11 for
Because this is the first time we are running tests for this product version
we end up with the newly created TP-16!
Note: see how version was populated with the correct value for the git branch!
This is because my CI config uses
TCMS_PRODUCT_VERSION=$TRAVIS_BRANCH and no further
changes were required! I made this change back in Travis Build #5 anticipating what
will come in the future!
The product is now live and customers have reported critical bugs for it:
URLs for the badges in README are wrong! I fix those and add more tests of course, see:
Travis Build #12 for
TR-29 contains the new TC-58!
cherry-pick between branches
lts-v0 branch customers have identified a serious issue. It doesn't exist on
but the test is still valid so I cherry-pick it! In git you can backport or forwardport very
easily. Regardless of the direction Kiwi TCMS plugins will record the test results for you.
Travis Build #13 for
We can see that TC-58, which was originally implemented on the
is now present!
This is a very basic example of how you can organize collection and reporting for
your automation test suite results with Kiwi TCMS. The links here only refer to
artifacts created by
kiwitcms-tap-plugin while in the DB we keep others as well.
There are feature requests for more ENV variables which will allow you to control
TestPlan creation and child/parent relationship between test plans. See
https://github.com/kiwitcms/tap-plugin/issues and vote with a
to help us plan for these features.
Kiwi TCMS automation framework plugins are nothing more than result parsers
which talk back to a database. It is up to you to decide how to organize the
collection of test results and how to report on them later, when the need arises.
Future installments in this post series will take a look at workflows with
feature branches and pull requests and discuss possible organization scenarios.
You are welcome to share your ideas in the comments below.