Full Automated Test Coverage with Jenkins and Clover

Version 10.1 by Vincent Massol on 2016/12/10 14:06

Dec 10 2016

This blog post is not published yet.

Generating test coverage reports for a single Maven project is simple. You can use the Clover maven plugin easily for that. For example:

mvn clean clover:setup install clover:clover

Generating a report for several modules in the same Maven reactor (same build) is also easy since that's supported out of the box. For example:

mvn clean clover:setup install clover:aggregate clover:clover

However, generating a full coverage report for a multi-reactor project is much harder. Let's take the example of the XWiki project which has 4 separate Github repositories and thus 4 builds:

So the question is: How do we generate a single test coverage report for those 4 maven reactor builds. For example we want that tests that execute in the xwiki-enterprise repository generate coverage for source code located, say, in xwiki-commons.

Here's what we want to get:

dashboard.png 

The way to do this manually is to tell the Maven Clover plugin to use a single location for generating its data. Manually this can be achieved like this (more details can be found on the XWiki Test page):

# In xwiki-commons:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db
...
# In xwiki-enterprise:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

# From xwiki-enterprise, generate the full Clover report:
mvn clover:clover -N -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

This is already pretty cool. However it's taking a lot of time and it would be nicer if it could be executed on the CI (on http://ci.xwiki.org in our case).

One important note is that Clover modifies the artifacts and thus you need to be careful to not push them into production or make sure they're not used in other builds (since they'd fail since they'd need to have the Clover runtime JAR at execution time).

So, I chose to use Jenkins 2 and the new Pipeline plugin and used the following script (see the XWiki Clover Job):

node() {
 def mvnHome
 def localRepository
 def cloverDir
 stage('Preparation') {
   def workspace = pwd()
   localRepository = "${workspace}/maven-repository"
   // Make sure that the special Maven local repository for Clover exists
   sh "mkdir -p ${localRepository}"
   // Remove all XWiki artifacts from it
   sh "rm -Rf ${localRepository}/org/xwiki"
   sh "rm -Rf ${localRepository}/com/xpn"
   // Make sure that the directory where clover will store its data exists in
   // the workspace and that it's clean
   cloverDir = "${workspace}/clover-data"
   sh "rm -Rf ${cloverDir}"
   sh "mkdir -p ${cloverDir}"
   // Get the Maven tool.
   // NOTE: Needs to be configured in the global configuration.           
   mvnHome = tool 'Maven'
  }
 // each() has problems in pipeline, thus using a standard for()
 // See https://issues.jenkins-ci.org/browse/JENKINS-26481
 for (String repoName : ["xwiki-commons", "xwiki-rendering", "xwiki-platform", "xwiki-enterprise"]) {
   stage("Cloverify ${repoName}") {
     dir (repoName) {
       git "https://github.com/xwiki/${repoName}.git"
       runCloverAndGenerateReport(mvnHome, localRepository, cloverDir)
      }  
    }      
  }
 stage("Publish Clover Reports") {
    ...
  }
}
def runCloverAndGenerateReport(def mvnHome, def localRepository, def cloverDir) {
 wrap([$class: 'Xvnc']) {
   withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx2048m']) {
     sh "mvn -Dmaven.repo.local='${localRepository}' clean clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true"
     sh "mvn -Dmaven.repo.local='${localRepository}' clover:clover -N -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db"
    }
  }
}

Note that we use the "Xvnc" Jenkins plugin because we run Selenium2 functional tests which require a display.

When this Jenkins job is executed is results in:

pipeline.png 

Over 5 hours of build time... Now you understand why we want to have this running on the CI agent and not on my local machine emoticon_wink

And the generated reports can be seen on xwiki.org.

Good news, we have an overall coverage of 73.2% for the full XWiki Java codebase, that's not too bad (I thought it would be lower emoticon_wink).

Next blog post will be about trying to achieve this with the Jacoco Maven plugin and the associated challenges and issues... Hint: it's harder than with the Clover Maven plugin.

Tags: STAMP
Created by Vincent Massol on 2016/12/10 13:56