Jan 11 2013

On the XWiki project we have a Jenkins Job that runs every night and checks that the quality of Maven modules have not decreased.

This is done by using the Maven Clover plugin and failing the build if the TPC is under a specified threshold for each module.

It's defined in a quality profile since it takes some time to execute (this is why we execute it once per night ATM).

Here's the setup:

   <!-- Profile for QA verifications that takes time -->
   <profile>
     <id>quality</id>
     <build>
       <plugins>
         <!-- Fail the build if the test coverage is below a given value. -->
         <plugin>
           <groupId>com.atlassian.maven.plugins</groupId>
           <artifactId>maven-clover2-plugin</artifactId>
           <configuration>
             <targetPercentage>${xwiki.clover.targetPercentage}</targetPercentage>
           </configuration>
           <executions>
             <execution>
               <id>clover-check</id>
               <phase>verify</phase>
               <goals>
                 <goal>instrument-test</goal>
                 <goal>check</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
     </build>
   </profile>

Then in each Maven, we have (for example):

...
 <properties>
   <xwiki.clover.targetPercentage>74.6%</xwiki.clover.targetPercentage>
 </properties>
...

Now that's fine and it allows to find out when someone adds code in an existing module and doesn't add enough unit tests to keep the TPC above the current threshold.

There remains an issue. Imagine that I add some code with unit tests. I also need to not forget to update the TPC value in the pom.xml file.

So here's a quick command line tip to find out the current TPC max threshold for all modules located under the directory when you run it:

mvn clean install -Pquality -Dxwiki.clover.targetPercentage=100%
-Dmaven.clover.failOnViolation=false 2>&1 |
awk '/Checking for coverage of/ { module = $9; }
/^Total/ { split(module, array, "/"); print array[length(array)-3],$4 }'

For example when run in xwiki-commons/xwiki-commons-core, it gives:

xwiki-commons-test-simple 0%
xwiki-commons-text 93.5%
xwiki-commons-component-api 22.7%
xwiki-commons-classloader-api 0%
xwiki-commons-classloader-protocol-jar 0%
xwiki-commons-observation-api 15.9%
xwiki-commons-component-observation 76.2%
xwiki-commons-component-default 74.6%
xwiki-commons-context 76.7%
xwiki-commons-script 0%
xwiki-commons-configuration-api 0%
xwiki-commons-test-component 0%
xwiki-commons-environment-api -100%
xwiki-commons-environment-common 0%
xwiki-commons-environment-standard 67.3%
xwiki-commons-environment-servlet 84.6%
xwiki-commons-properties 76.6%
xwiki-commons-logging-api 29.5%
xwiki-commons-observation-local 90.8%
xwiki-commons-job 36.1%
xwiki-commons-logging-logback 91.8%

Now the next step is to write a script that will automatically change the pom.xml files with the max TPC threshold values.

UPDATE 2013-01-31: To do the same with Jacoco you would use:

mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=100
-Djacoco.haltOnFailure=false 2>&1 |
awk '/jacoco-check/ { module = $6 } /Insufficient/ { print module, $7 }'

UPDATE 2013-07-09: Starting with Jacoco 0.6.4 we need to use:

mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=2.00
-Djacoco.haltOnFailure=false 2>&1 |
awk '/jacoco-check/ { module = $6 } /instructions covered ratio is/ { print module, $(NF-5) }'
Tags:
Created by Vincent Massol on 2013/01/11 14:06
This wiki is licensed under a Creative Commons 2.0 license