Tuesday, April 26, 2011

Combinatorial tests in JUNIT

As a friend of mine (Hi Pigelvy, still talking to me ?) recently asked about the ability to create your own parameter suppliers for combinatory tests in JUNIT 4... something.
Shame on me, I answered him to check on Google, which is not the correct behaviour of a sharing craftsman apprentice.
So I checked by myself and realised it was not so easy to achieve and understood better why Pigelvy stopped talking to me.
Facts are that I do have the same needs, even for the very basics situations when I have to check boolean flag positionings in POJO classes for example.

When we came to work together with Pigelvy, we choosed to work with JUNIT theories, because working with Theory exposes code with better readeability than paramerized tests (as a reminder about Theories in JUNIT check http://blogs.sun.com/jacobc/entry/junit_theories)

To make it simple, combinatory testing is used in pair with the Theories.class runner, flagging each testing method with the Theory annotation. Then when I had to apply test suites involving boolean flags I used to create a

@DataPoints boolean[] FLAGS = new boolean[]{true, false};

Really this sucks when abused. It is ugly because cluttering the code, and repeating this kind of declaration in all your tests classes is clearly a DRY principal violation.
Moreover, in a test case manipulating multiple annoted Theory methods with the same signature, involves that implicitely all arrays of values will be shared by all methods during the process of combining the all the values into arrays.
Some tests may not need the same experimental set of values and this behavior can become a real pain.

JUNIT, as far as I can understand, offers an easy way to "supply" sets of parameters to combine. Suppose I need to play with a range of integers (yes, that' s a classic).
The test method would be self explanatory if it looked like:

public void number_ShouldAlwaysBePositive(final @IntegerInRange(from = 1, to = 10) int number) {
assertThat(number, is(greaterThan(0)));

Obviously, JUNIT is going to execute my method with all the numbers in the interval from 1 to 10. All I need is apecial annotation like the following:

public @interface IntegerInRange {

int from();

int to();


... and a supplier class. The supplier class is where the stuff(magic ? :)) of supplying parameters really happens.

What I think I understood is that as you expose the nice DSL into the annotation definition and deals with the plumbing into the parameter supplier.
A parameter supplier extends a ParameterSupplier class. This constrains you by contract to implement the

public List getValueSources(final ParameterSignature parameterSignature)


By fail and retry (as I did not find documentation), I analysed the ParameterSignature parameter, used the tool methods and built up a list of "potential assignement".

public List getValueSources(final ParameterSignature parameterSignature) {
final IntegerInRange rangeSet = parameterSignature.getAnnotation(IntegerInRange.class);
final int lowerBound = rangeSet.from();
final int upperBound = rangeSet.to();

final List list = new ArrayList(upperBound - lowerBound);
for (int i = lowerBound; i < upperBound; i++) {
list.add(PotentialAssignment.forValue(String.valueOf(i), i));
return list;

Blame me, I did not check the possible nul referencing of the rangeSet instance, but I hope the code is clear enough.
Using the getAnnotation tool method, I got access to the annotation instance
I then extract the boundaries using the boundaries I create a list of potential assignment for JUNIT to deal with.

One should remark that I preferred using the PotentialAssignment.forValue factory method in order to create the PotentialAssignment instances (As Joshua Kerievsky , I do prefer creation knowledge confined into some factory).

As I like clarity too, I decided I preferred creating a one to one relationship between an annotation and a supplier.
Back to my egotist simpler problem of the begining (set of boolean parameters) I immediatly created an annotation:

public @interface BooleanParameter {}

with the matching BooleanParametersSupplier supplier getValueSources method implementation:

public List getValueSources(final ParameterSignature parameterSignature) {
final List list = new LinkedList();
list.add(PotentialAssignment.forValue("true", TRUE));
list.add(PotentialAssignment.forValue("false", FALSE));
return list;

An example of use being:

public void check_WithFlag_ShouldBeBound(final @BooleanParameter boolean check) {...}

This set of combinatory classes seems to offer far more possibilities I did not explore yet. For example, I did not succeed in using the description field of the PotentialAssignment class even provoking a failure. I have a hung there are numerous other possibilities and will gladly welcome feed backs about different uses.


Jeremy John Reeder said...

I tend to prefer "Parameterized" JUnit tests over "Theory" JUnit tests, for a couple of reasons.

My first reason is that the results of a Parameterized test are reported for each test case so I can see how many of the test cases failed and which ones they are. (A Theory test stops at the first problem and reports the entire test as if it were a single test case. So it provides a lot less information.)

My second reason is that, while Theory tests test all possible combinations of the parameter values that I provide, Parameterized tests allow me to create my own combinations in any way I like (such as with the AllPairsParameterFactory of the JCombinatorial library).

Parameterized testing and JCombinatorial are documented, with examples, at http://code.google.com/p/jcombinatorial/wiki/ExampleTestSeries .

Globulon said...

I perfectly understand your point of view and the fact that Theories do belong to an experimental package proves they may not be fully achieved.
The Legacy code I was working one was specific enough to support to be stressed with very small range of values supplied by parameters.
I am certainly going to have a look to JCombinatorial. Thanks for the link

Henrik Kaipe said...

How about combining parameter values this way?

@ParameterizedValue boolean boolValue;
@ParameterizedValue SomeEnum enumValue;

The parameter-value for "boolValue" will be picked from {true,false} and "enumValue" will get a constant from "SomeEnum"! - No need to define any supplier-class for an enum, right?

This is what "CallbackParamsRunner" does! See http://callbackparams.org

CallbackParams would test all possible combinations of "boolValue" and "enumValue".
If you add one more parameterized field ...

@ParameterizedValue boolValue2

... then the default behaviour is to pick values for "boolValue", "enumValue" and "boolValue2" in accordance with AllPairsParameterFactory of JCombinatorial! - But CallbackParams has its own implementation for this: http://callbackparams.org/project/api-javadoc/org/callbackparams/combine/CombineAllPossible2Combinations.html


For an enum there is an implicit static method "values()" that supply an array with all enum-constants. CallbackParams has extended this pattern so that a non-enum class must have a static method "values()" to be used as a "@ParamerizedValue" field.

One idea behind CallbackParams is that enums are better parameters than strings and primitive types. It is because enums can contain logic on their own.

If you check the website http://callbackparams.org you will also be introduced to an alternative parameter abstraction ...

@ParameterizedCallback ACallbackInterface callback;

... that works as a composite for all parameters that implement the interface "ACallbackInterface". It might seem odd but it is an abstraction that greatly simplifies maintenance ...

Please have a look!

Post a Comment