How to Write Tests
Why
You most probably know why it's a good idea to write tests, but let's name some of the reasons here:
- Automatic tests can run 24/7, which is way way more effective than testing code manually
- Tests can capture what a code is expected to do and, if well done, it does not let us introduce regressions
- Each code will eventually break, but you may not find out without tests
- People might be afraid of a change if they can't guarantee all the code still does what it's expected to do - tests give you the confidence for change
- Bad and ugly test makes you write a good API
- Test output (RSpec) can help with documentation
How
YaST team uses RSpec for unit testing and a mix of solutions, depending on each particular case, for integration testing.
Unit tests with RSpec
Here are some basic rules that all YaST projects should follow regarding RSpec tests:
- Test must be readable - everyone has to easily understand what the test actually tests
- It's easy-to-maintain - no hacks, no magic
- Needs to cover the whole code, otherwise you would feel a false sense of security
- A test suite should be comprehensive but every individual test on it should be specific
- Test all aspects of a method (depending on context, and input), see Examples
- Uses "allow" for queries, but "expect" commands, see Examples
- Expectation in "it" block - must be explicit, see Examples
- Description of a test should describe the behavior - you should get the idea just by reading the "it" / "context", see Examples
Parallel Tests
Adding more and more unit tests obviously means that the time needed for running the complete test suite also increases. And at some point it becomes annoying.
Normally the RSpec executes the tests in sequence, one by one. However, using the parallel_tests Ruby gem it is possible to run the tests in parallel.
The rake unit:test
includes support for running parallel tests, just make sure
you have installed the ruby<version>-rubygem-parallel_tests
package otherwise
the task will use the standard (sequential) RSpec. Add the gem to
BuildRequires
dependency if you want to use it also during the RPM package build.
To activate the parallel tests just create a .rspec_parallel
file in the Git
root directory. The suggested content is
--format progress --format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log
This will use the "dot" progress output and save the runtime statistics for each test file to the specified log file. Next time it will be used to split the tests into more balanced test groups. That means the very first run will be less effective and probably it will take a bit more time.
Obviously there must not be any dependencies or race conditions between the tests otherwise it will fail.
You can force running parallel tests using PARALLEL_TESTS=1 rake test:unit
,
that is great for checking if the current tests can be run in parallel.
Or force classic sequential RSpec with PARALLEL_TESTS=0 rake test:unit
to find whether a test failure might be caused by some race condition.
See the test:unit description
for more details.
You can check a full example in the yast2-storage-ng package.
Integration tests with openQA
Integration tests for all the openSUSE distributions are performed by the openSUSE openQA instance. When possible, is desirable to add tests for new features or bug fixes in YaST to the opensuse distri, so they can be integrated in the continuous testing process of the distribution.
See the links section for more information about how to develop openQA tests for YaST.
AutoYaST integration tests
AutoYaST has its own framework for running integration tests by using Veewee, Vagrant and Pennyworth. See the corresponding Github page for more information.
Tests for the command line with the Test Anything Protocol
Apart from the usual user interface, some YaST modules also offer a non-interactive command line interface. In order to test that CLI, simple TAP-compliant scripts are used. TAP, the Test Anything Protocol, is a simple text-based interface between testing modules in a test harness.
The CLI testing scripts are placed in a t
directory at the root of each
YaST module and are executed using the command prove
. This command is a
runner for the Test Anything Protocol which has a stdio interface and thus
is well suited for command line tests. The program is conveniently part of
a base openSUSE system, in perl5.rpm
.
The scripts operate directly on the running system, which means that, although
most of them try to clean up after themselves, they can reconfigure the system.
As a consequence, they must be executed in a scratch virtual machine. For that
purpose, a specific openQA test module called
yast2_cmdline.pm
is used to run the scripts in a safe way. All modules with CLI integration tests
must be explicitly included in yast2_cmdline.pm
.
Links
- RSpec
- Better Specs - rspec guidelines with Ruby
- The RSpec Style Guide
- YaST RSpec helpers
- openQA