On the XWiki project, we use Clover to compute our global test coverage. We do this over several Git repositories and include functional tests (and more generally the coverage brought by some modules into other modules).
Now I wanted to see the difference between 2 reports that were generated:
- Report from 2016-12-20
- Report from 2017-11-09
I was surprised to see a drop in the global TPC, from 73.2% down to 71.3%. So I took the time to understand the issue.
It appears that Clover classifies your code classes as Application Code and Test Code (I have no idea what strategy it uses to differentiate them) and even though we've used the same version of Clover (4.1.2) for both reports, the test classes were not categorized similarly. It also seems that the TPC value given in the HTML report is from Application Code.
Luckily we asked the Clover Maven plugin to generate not only HTML reports but also XML reports. Thus I was able to write the following Groovy script that I executed in a wiki page in XWiki. I aggregated Application Code and Test code together in order to be able to compare the reports and the global TPC value.
def saveMetrics(def packageName, def metricsElement, def map) {
def coveredconditionals = metricsElement.@coveredconditionals.toDouble()
def coveredstatements = metricsElement.@coveredstatements.toDouble()
def coveredmethods = metricsElement.@coveredmethods.toDouble()
def conditionals = metricsElement.@conditionals.toDouble()
def statements = metricsElement.@statements.toDouble()
def methods = metricsElement.@methods.toDouble()
def mapEntry = map.get(packageName)
if (mapEntry) {
coveredconditionals = coveredconditionals + mapEntry.get('coveredconditionals')
coveredstatements = coveredstatements + mapEntry.get('coveredstatements')
coveredmethods = coveredmethods + mapEntry.get('coveredmethods')
conditionals = conditionals + mapEntry.get('conditionals')
statements = statements + mapEntry.get('statements')
methods = methods + mapEntry.get('methods')
}
def metrics = [:]
metrics.put('coveredconditionals', coveredconditionals)
metrics.put('coveredstatements', coveredstatements)
metrics.put('coveredmethods', coveredmethods)
metrics.put('conditionals', conditionals)
metrics.put('statements', statements)
metrics.put('methods', methods)
map.put(packageName, metrics)
}
def scrapeData(url) {
def root = new XmlSlurper().parseText(url.toURL().text)
def map = [:]
root.project.package.each() { packageElement ->
def packageName = packageElement.@name
saveMetrics(packageName.text(), packageElement.metrics, map)
}
root.testproject.package.each() { packageElement ->
def packageName = packageElement.@name
saveMetrics(packageName.text(), packageElement.metrics, map)
}
return map
}
def computeTPC(def map) {
def tpcMap = [:]
def totalcoveredconditionals = 0
def totalcoveredstatements = 0
def totalcoveredmethods = 0
def totalconditionals = 0
def totalstatements = 0
def totalmethods = 0
map.each() { packageName, metrics ->
def coveredconditionals = metrics.get('coveredconditionals')
totalcoveredconditionals += coveredconditionals
def coveredstatements = metrics.get('coveredstatements')
totalcoveredstatements += coveredstatements
def coveredmethods = metrics.get('coveredmethods')
totalcoveredmethods += coveredmethods
def conditionals = metrics.get('conditionals')
totalconditionals += conditionals
def statements = metrics.get('statements')
totalstatements += statements
def methods = metrics.get('methods')
totalmethods += methods
def elementsCount = conditionals + statements + methods
def tpc
if (elementsCount == 0) {
tpc = 0
} else {
tpc = ((coveredconditionals + coveredstatements + coveredmethods)/(conditionals + statements + methods)).trunc(4) * 100
}
tpcMap.put(packageName, tpc)
}
tpcMap.put("ALL", ((totalcoveredconditionals + totalcoveredstatements + totalcoveredmethods)/(totalconditionals + totalstatements + totalmethods)).trunc(4) * 100)
return tpcMap
}
// map1 = old
def map1 = computeTPC(scrapeData('http://maven.xwiki.org/site/clover/20161220/clover-commons+rendering+platform+enterprise-20161220-2134/clover.xml')).sort()
// map2 = new
def map2 = computeTPC(scrapeData('http://maven.xwiki.org/site/clover/20171109/clover-commons+rendering+platform-20171109-1920/clover.xml')).sort()
println "= Added Packages"
println "|=Package|=TPC New"
map2.each() { packageName, tpc ->
if (!map1.containsKey(packageName)) {
println "|${packageName}|${tpc}"
}
}
println "= Differences"
println "|=Package|=TPC Old|=TPC New"
map2.each() { packageName, tpc ->
def oldtpc = map1.get(packageName)
if (oldtpc && tpc != oldtpc) {
def css = oldtpc > tpc ? '(% style="color:red;" %)' : '(% style="color:green;" %)'
println "|${packageName}|${oldtpc}|${css}${tpc}"
}
}
println "= Removed Packages"
println "|=Package|=TPC Old"
map1.each() { packageName, tpc ->
if (!map2.containsKey(packageName)) {
println "|${packageName}|${tpc}"
}
}
{{/groovy}}
And the result was quite different from what the HTML report was giving us!
We went from 74.07% in 2016-12-20 to 76.28% in 2017-11-09 (so quite different from the 73.2% to 71.3% figure given by the HTML report). Much nicer!
Note that one reason I wanted to compare the TPC values was to see if our strategy of failing the build if a module's TPC is below the current threshold was working or not (I had tried to assess it before but it wasn't very conclusive).
Now I know that we won 1.9% of TPC in a bit less than a year and that looks good
EDIT: I'm aware of the Historical feature of Clover but:
- We haven't set it up so it's too late to compare old reports
- I don't think it would help with the issue we faced with test code being counted as Application Code, and that being done differently depending on the generated reports.