How to Write Tests

Why

You most probably know why it's a good idea to write tests, but let's name some of the reasons here:

  • Automatic tests can run 24/7, which is way way more effective than testing code manually
  • Tests can capture what a code is expected to do and, if well done, it does not let us introduce regressions
  • Each code will eventually break, but you may not find out without tests
  • People might be afraid of a change if they can't guarantee all the code still does what it's expected to do - tests give you the confidence for change
  • Bad and ugly test makes you write a good API
  • Test output (RSpec) can help with documentation

How

YaST team uses RSpec for unit testing and a mix of solutions, depending on each particular case, for integration testing.

Unit tests with RSpec

Here are some basic rules that all YaST projects should follow regarding RSpec tests:

  • Test must be readable - everyone has to easily understand what the test actually tests
  • It's easy-to-maintain - no hacks, no magic
  • Needs to cover the whole code, otherwise you would feel a false sense of security
  • A test suite should be comprehensive but every individual test on it should be specific
  • Test all aspects of a method (depending on context, and input), see Examples
  • Uses "allow" for queries, but "expect" commands, see Examples
  • Expectation in "it" block - must be explicit, see Examples
  • Description of a test should describe the behavior - you should get the idea just by reading the "it" / "context", see Examples

Parallel Tests

Adding more and more unit tests obviously means that the time needed for running the complete test suite also increases. And at some point it becomes annoying.

Normally the RSpec executes the tests in sequence, one by one. However, using the parallel_tests Ruby gem it is possible to run the tests in parallel.

The rake unit:test includes support for running parallel tests, just make sure you have installed the ruby<version>-rubygem-parallel_tests package otherwise the task will use the standard (sequential) RSpec. Add the gem to BuildRequires dependency if you want to use it also during the RPM package build.

To activate the parallel tests just create a .rspec_parallel file in the Git root directory. The suggested content is

--format progress --format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

This will use the "dot" progress output and save the runtime statistics for each test file to the specified log file. Next time it will be used to split the tests into more balanced test groups. That means the very first run will be less effective and probably it will take a bit more time.

Obviously there must not be any dependencies or race conditions between the tests otherwise it will fail.

You can force running parallel tests using PARALLEL_TESTS=1 rake test:unit, that is great for checking if the current tests can be run in parallel. Or force classic sequential RSpec with PARALLEL_TESTS=0 rake test:unit to find whether a test failure might be caused by some race condition. See the test:unit description for more details.

You can check a full example in the yast2-storage-ng package.

Integration tests with openQA

Integration tests for all the openSUSE distributions are performed by the openSUSE openQA instance. When possible, is desirable to add tests for new features or bug fixes in YaST to the opensuse distri, so they can be integrated in the continuous testing process of the distribution.

See the links section for more information about how to develop openQA tests for YaST.

AutoYaST integration tests

AutoYaST has its own framework for running integration tests by using Veewee, Vagrant and Pennyworth. See the corresponding Github page for more information.

Tests for the command line with the Test Anything Protocol

Apart from the usual user interface, some YaST modules also offer a non-interactive command line interface. In order to test that CLI, simple TAP-compliant scripts are used. TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness.

The CLI testing scripts are placed in a t directory at the root of each YaST module and are executed using the command prove. This command is a runner for the Test Anything Protocol which has a stdio interface and thus is well suited for command line tests. The program is conveniently part of a base openSUSE system, in perl5.rpm.

The scripts operate directly on the running system, which means that, although most of them try to clean up after themselves, they can reconfigure the system. As a consequence, they must be executed in a scratch virtual machine. For that purpose, a specific openQA test module called yast2_cmdline.pm is used to run the scripts in a safe way. All modules with CLI integration tests must be explicitly included in yast2_cmdline.pm.

Links