The XWiki project is using a strategy to try to ensure that quality goes in the upward direction.
In short we fail the build if the Jacoco-computed coverage is below a per-module threshold. Devs can only increase the threshold but are not supposed to lower it.
However, from time to time, it happens that dev reduce the threshold (for example, when fixing a bug and this removes some lines of code and the coverage is lowered and the dev doesn't have the time to improve existing tests, etc).
Since we've been following this strategy for a long time now (at least since 2013), I thought it would be interesting to check, for a small subset of XWiki, how we fared.
Module Name | TPC on Feb 2013 | TPC on May 2017 | Difference |
---|---|---|---|
xwiki-commons-tool-verification-resources | - | 46% | - |
xwiki-commons-test-simple | 0% | 22% | +22% |
xwiki-commons-text | 93.5% | 94% | +0.5% |
xwiki-commons-component-api | 22.7% | 45% | +22.3% |
xwiki-commons-classloader-api | 0% | - | - |
xwiki-commons-classloader-protocol-jar | 0% | - | - |
xwiki-commons-observation-api | 15.9% | 100% | +84.1% |
xwiki-commons-component-observation | 76.2% | 74% | -2.2% |
xwiki-commons-component-default | 74.6% | 71% | -3.6% |
xwiki-commons-context | 76.7% | 81% | +4.3% |
xwiki-commons-blame-api | - | 94% | - |
xwiki-commons-logging-api | - | 76% | - |
xwiki-commons-diff-api | - | 62% | - |
xwiki-commons-diff-display | - | 95% | - |
xwiki-commons-script | 0% | 27% | +27% |
xwiki-commons-cache-infinispan | - | 76% | - |
xwiki-commons-crypto-common | - | 62% | - |
xwiki-commons-crypto-cipher | - | 70% | - |
xwiki-commons-crypto-password | - | 65% | - |
xwiki-commons-crypto-signer | - | 71% | - |
xwiki-commons-crypto-pkix | - | 76% | - |
xwiki-commons-crypto-store-filesystem | - | 73% | - |
xwiki-commons-configuration-api | 0% | - | - |
xwiki-commons-test-component | 0% | - | - |
xwiki-commons-environment-api | -100% | - | - |
xwiki-commons-environment-common | 0% | - | - |
xwiki-commons-environment-standard | 67.3% | 65% | -2.3% |
xwiki-commons-environment-servlet | 84.6% | 85% | +0.4% |
xwiki-commons-properties | 76.6% | 79% | +2.4% |
xwiki-commons-logging-api | 29.5% | - | - |
xwiki-commons-observation-local | 90.8% | 89% | -1.8% |
xwiki-commons-job | 36.1% | 58% | +21.9% |
xwiki-commons-logging-logback | 91.8% | 93% | +1.2% |
xwiki-commons-extension-api | - | 68% | - |
xwiki-commons-extension-maven | - | 70% | - |
xwiki-commons-extension-handler-jar | - | 82% | - |
xwiki-commons-extension-repository-maven | - | 69% | - |
xwiki-commons-repository-api | - | 76% | - |
xwiki-commons-extension-repository-xwiki | - | 18% | - |
xwiki-commons-filter-api | - | 29% | - |
xwiki-commons-xml | - | 59% | - |
xwiki-commons-filter-xml | - | 54% | - |
xwiki-commons-filter-test | - | 3% | - |
xwiki-commons-groovy | - | 94% | - |
xwiki-commons-velocity | - | 71% | - |
xwiki-commons-tool-xar-plugin | - | 10% | - |
Note that - denotates some modules that do not exist at a given date or for which the coverage is empty (for example a module with only Java interfaces).
Conclusions:
- Coverage has not increased substantially in general. However this is computed on xwiki-commons and those modules are pretty stable and don't change much. It would be interesting to compute something similar for xwiki-platform.
- Out of 14 modules that have seen their TPC modified between 2013 and May 2017, 10 have seen their coverage increase (that's 71%). So 4 have seen their coverage be reduced by up to -3.6% max.
So while we could better, it's still not too bad and the strategy seems to be globally working.