XWiki SAS is part of an European research project named STAMP. As part of this project I've been able to experiment a bit with Descartes, a mutation engine for PIT.
What PIT does is mutate the code under test and check if the existing test suite is able to detect those mutations. In other words, it checks the quality of your test suite.
Descartes plugs into PIT by providing a set of specific mutators. For example one mutator will replace the output of methods by some fixed value (for example a method returning a boolean will always return true). Another will remove the content of void methods. It then generates a report.
Here's an example of running Descartes on a module of XWiki:
You can see both the test coverage score (computed automatically by PIT using Jacoco) and the Mutation score.
If we drill down to one class (MacroId.java) we can see for example the following report for the equals() method:
What's interesting to note is that the test coverage says that the following code has been tested:
(getId() == macroId.getId() || (getId() != null && getId().equals(macroId.getId())))
&& (getSyntax() == macroId.getSyntax() || (getSyntax() != null && getSyntax().equals(
macroId.getSyntax())));
However, the mutation testing is telling us a different story. It says that if you change the equals method code with negative conditions (i.e. testing for inequality), the test still reports success.
If we check the test code:
public void testEquality()
{
MacroId id1 = new MacroId("id", Syntax.XWIKI_2_0);
MacroId id2 = new MacroId("id", Syntax.XWIKI_2_0);
MacroId id3 = new MacroId("otherid", Syntax.XWIKI_2_0);
MacroId id4 = new MacroId("id", Syntax.XHTML_1_0);
MacroId id5 = new MacroId("otherid", Syntax.XHTML_1_0);
MacroId id6 = new MacroId("id");
MacroId id7 = new MacroId("id");
Assert.assertEquals(id2, id1);
// Equal objects must have equal hashcode
Assert.assertTrue(id1.hashCode() == id2.hashCode());
Assert.assertFalse(id3 == id1);
Assert.assertFalse(id4 == id1);
Assert.assertFalse(id5 == id3);
Assert.assertFalse(id6 == id1);
Assert.assertEquals(id7, id6);
// Equal objects must have equal hashcode
Assert.assertTrue(id6.hashCode() == id7.hashCode());
}
We can indeed see that the test doesn't test for inequality. Thus in practice if we replace the equals method by return true; then the test still pass.
That's interesting because that's something that test coverage didn't notice!
More generally the report provides a summary of all mutations it has done and whether they were killed or not by the tests. For example on this class:
