Question

How do I run parallel tests with Selenium testing framework Serenity?

Answer 1

Sometimes projects have a lot of tests, and executing of them takes a lot of time. Testing can be sped up significantly by running different tests in parallel. However, this is often harder to implement than it sounds.

Some build automation tools have builtin parallel test execution, but this not so good for huge amount of tests and heavy tests. For example web tests are as a rule much slower than other types of tests, it make them good candidates for concurrent testing, in theory at least, but the implementation can be tricky. For example, although it is easy enough to configure running tests in parallel, on the other hand running several webdriver instances of Firefox/Chrome in parallel on the same display, tends to become unreliable.

The natural solution in this case is to split the web tests into smaller batches, and to run each batch on a different machine and/or on a different virtual display. When each batch has finished, the results can be retrieved and aggregated into the final test reports.

However splitting tests into manually batches by hand tends to be tedious and unreliable – it is easy to forget to add a new test to a batch, for example, or have unevenly-distributed batches.

Serenity provides mechanisms to automatically split your test suite into slices at runtime, removing the need for any manual structuring of source files or maintenance of scripts. In practice, you decide on the number of batches you require, then run a job associated with each batch number which will deterministically run only tests that belong to that batch. When all jobs have completed running, the test result output is aggregated into the final serenity test report.

Splitting a Serenity suite into batches

The following parameters affect Serenity’s batching behaviour and can be provided at runtime as system parameters or be added to a serenity.properties or serenity.conf file:

serenity.batch.count

Refers to the total number of separately/concurrently running jobs that the whole test suite will be divided up into at runtime. So for instance if you had 8 build slaves available to run your tests, you would set this property to 8. This value would be the same for all jobs running each batch, The property needs to be greater than 1 in order for serenity to trigger its batching behaviour.

serenity.batch.number

This parameter should be different for each batch, and should be a value between 1 and serenity.batch.count.

JUnit-specific configuration

serenity.batch.strategy

Optional parameter for choosing batch weighting strategy (Junit tests only). Possible values are:

• DIVIDE_EQUALLY: Test classes are allocated to batches on a 'round-robin' basis without regard for the number of test methods in each class. So for example if serenity.batch.count is 3 and we have classes T1, T2, T3, T4, T5, the first batch will contain T1, T4, the second will contain T2, T5, and the third will contain T3.DIVIDE_EQUALLY is the default behaviour when no batch strategy parameter is explicitly set.

• DIVIDE_BY_TEST_COUNT. Classes are allocated to batches based on the number of test methods they contain. So for example if serenity.batch.count is 2 and class T1 contains 4 tests, T2 contains 2 tests, T3 contains 2 tests, both batches will contain 4 tests each, the first batch containing T1, and the second one containing T3 and T2. DIVIDE_BY_TEST_COUNT tends to provide more evenly balanced batches than DIVIDE_EQUALLY.

src: https://serenity-bdd.github.io/theserenitybook/latest/serenity-parallel.html