Think Tank

On the XWiki project it's been years that we've been checking automatically for backward compatibilities in our APIs as part of our build (It's even documented here).

Recently, we've moved to Java 8 for XWiki 8.1 and we've discovered that the tool we were using, CLIRR, doesn't work anymore with Java 8. So I searched, without much hope, for an alternative, and I was surprised (and delighted) to find that there are 2 valid solutions nowadays that support Java 8:

• japicmp
• Revapi

I've tried both and they both had issues. However, I've discovered that the maintainers or these 2 solutions were really cool guys and they very quickly fixed any issue I've raised. Big thanks to Lukas Krejci and Martin Mois, you're awesome guys!

In the end the XWiki project has had to make a difficult choice and we chose Revapi (Some reasons mentioned here but honestly they're both valid choices).

So here's how we use it now:

• In our top level POM:
  ...
       <plugins>
  ...
         <!-- Used for checking backward compatibility (binary and source) -->
         <plugin>
           <groupId>org.revapi</groupId>
           <artifactId>revapi-maven-plugin</artifactId>
           <!-- Lock down plugin version for build reproducibility -->
           <version>0.4.5</version>
           <dependencies>
             <dependency>
               <groupId>org.revapi</groupId>
               <artifactId>revapi-java</artifactId>
               <version>0.9.0</version>
             </dependency>
           </dependencies>
           <executions>
             <execution>
               <id>revapi-check</id>
               <goals>
                 <goal>check</goal>
               </goals>
             </execution>
           </executions>
           <configuration>
             <oldVersion>${xwiki.compatibility.previous.version}</oldVersion>
             <skip>${xwiki.revapi.skip}</skip>
             <analysisConfiguration>
              {
                "revapi": {
                  "java": {
                    "filter": {
                      "packages": {
                        "regex": true,
                        "include": ["org\\.xwiki\\..*"],
                        "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                      }
                    }
                  }
                }
              }
             </analysisConfiguration>
           </configuration>
         </plugin>
  ...
• Then in specific module POMs we override the configuration to add excludes (for backward and source incompatibilities that we know about and have voluntarily decided to do). For example:
  ...
       <plugin>
         <groupId>org.revapi</groupId>
         <artifactId>revapi-maven-plugin</artifactId>
         <configuration>
           <analysisConfiguration><![CDATA[
              {
                "revapi": {
                  "java": {
                    "filter": {
                      "packages": {
                        "regex": true,
                        "include": ["org\\.xwiki\\..*"],
                        "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                      }
                    }
                  },
                  "ignore": [
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.List<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                      "new": "method java.util.List<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    },
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                      "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    },
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                      "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    }         
                  ]
                }
              }
            ]]></analysisConfiguration>
         </configuration>
       </plugin>
  ...
• Now the interesting part is in generating reports. We've chosen an original way of doing this: We dynamically generate the Revapi report on our release notes wiki pages using Groovy. The big advantage is that there's no work to be done for the Release Manager:
  • Here's an example from the XWiki 8.1M1 Release Notes

    

  • The Groovy script below does the following:
    • Use the GitHub REST API to get the content of the pom.xml containing the Revapi excludes.
    • Parse the XML using Groovy
    • Display a report out of it
  • And here's the Groovy script packaged as a Wiki Macro for the curious ones:
    {{groovy}}
    import groovy.json.*
    
    def getIgnores(def repo, def path)
    {
     def url = "https://api.github.com/repos/xwiki/${repo}/contents/${path}".toURL().text
     def result = new JsonSlurper().parseText(url)
     def xml = new String(result.content.decodeBase64())
      result = new XmlSlurper().parseText(xml)
     def revapi = result.build.plugins.plugin.'**'.find { node ->
        node.artifactId.text() == 'revapi-maven-plugin'
     }
     if (revapi) {
        result = new JsonSlurper().parseText(revapi.configuration.analysisConfiguration.text())
       return result.revapi.ignore
     } else {
       return ''
     }
    }
    
    def displayIgnores(def ignores)
    {
      result = new JsonSlurper().parseText(ignores)
      result.each() {
        it.each() {
          println "* {{{${it.justification}}}}"
          println "** Violation type: {{code}}${it.code}{{/code}}"
          println "** Old: {{code}}${it.old}{{/code}}"
         if (it.new) {
            println "** New: {{code}}${it.new}{{/code}}"
         }
       }
     }
    }
    
    def getViolations(def version)
    {
     def xobject = doc.getObject('ReleaseNotes.BackwardCompatibility')
     if (!xobject) {
        xobject = doc.newObject('ReleaseNotes.BackwardCompatibility')
       def commonsTag
       def renderingTag
       def platformTag
       if (version == 'master') {
          commonsTag = renderingTag = platformTag = 'master'
       } else {
          commonsTag = "xwiki-commons-${version}"
          renderingTag = "xwiki-rendering-${version}"
          platformTag = "xwiki-platform-${version}"
       }
       def jsonCommons = getIgnores('xwiki-commons', "xwiki-commons-core/pom.xml?ref=${commonsTag}")
       def jsonRendering = getIgnores('xwiki-rendering', "pom.xml?ref=${renderingTag}")
       def jsonPlatform = getIgnores('xwiki-platform', "xwiki-platform-core/pom.xml?ref=${platformTag}")
        xobject.set('violations', JsonOutput.prettyPrint(JsonOutput.toJson([jsonCommons, jsonRendering, jsonPlatform])))
        doc.save('Added backward-compatiblity violations data', true)
     }
     return xobject.getProperty('violations').value
    }
    
    displayIgnores(getViolations(xcontext.macro.params.version))
    {{/groovy}}

In conclusion we're very happy with the move, especially since we now have Java 8 support but also because Revapi provides more checks than what CLIRR was doing. ...

On the XWiki project we have enabled Jenkins's Incremental Build feature:

This seemed like a nice feature to speed up our CI when building our Maven jobs. Alas, it doesn't work!

The problem is that from time to time you'll get build failure such as:

You'd think it's not possible, especially as we use Maven's -U flag and thus, even if the artifact is not present in the local repository it should be downloaded from the Maven Remote Repository that we use (and it's available there!).

The reason is because of a Maven bug: MNG-5542. What happens is that the Incremental Build feature will use the -pl Maven parameter to list all the Maven projects to build and when this feature is used, artifacts declared in the <parent> section of your POMs are just ignored and not downloaded...

The consequence is that your build will fail from time to time when one artifact declared in one of your <parent> is not present in your local repository... If you have decided to have one Maven repository per Job in Jenkins - which would seem a good idea to isolate your jobs and to be able to use the Parallel build feature of Jenkins - then you'll hit the problem very frequently...

So in the end you have to choose between Parallel builds and Incremental builds but you cannot have both at the same time!

Note that even with Parallel builds turned off, you build will fail from time to time, just less frequently...

One solution for the XWiki project would be to break our big job that builds the Platform (over 100 modules located in the same Git repo that we release together under a single version) into 100 jobs. But doing this manually is a no go so we would need to script this or wait for some Jenkins dev to implement this idea...

Big pain!

I'd like to explain what we do on the XWiki open source project to preserve backward compatibility of our Java APIs. We've done several iterations and we now have a process that is working quite well, even though there are still some issues but they're unavoidable with Java for the time being.

So here it goes:

• We start by having a special package named internal and everything we put in there is considered implementation details and should never be used by users of our APIs. We're allowed to modify anything at any time in the internal package. For example: org.xwiki.instance.internal.*
• We also use a Component-based approach, which forces us to separate interfaces (which are public) from implementation (which go in the internal package).
• We have defined an advanced deprecation mechanism which allows us to move deprecated code that we no longer use to legacy maven module (using Aspects with AspectJ) that can be optionally installed at runtime, and which we never use at build time. This prevents us from using any deprecated legacy code and it allows us to push away the cruft under the carpet  (without breaking backward compatibility!)
• We have the notion of "Young API" (a.k.a Unstable APIs) and when a new API is introduced, the developer can (and should) use the @Unstable annotation to signify to users that this API is unstable for the moment and can change at any time. We only allow an API to remain unstable for a full development cycle. We've recently introduced a custom Checkstyle check that enforces this!
• Last, we use the Maven CLIRR plugin to make sure we never break an API unintentionally (it fails our build). This allows us to only carefully and intentionally break our APIs. It also allows to us to mention what we break in our Release Notes (example).

The important point IMO is that we have automated tools to ensure that our strategy is applied, namely:

• CLIRR
• Our Unstable Annotation Checker (a custom checkstyle rule)

This is working pretty well for us even though we do break backward compatibility from time to time, when we judge that the code touched is unlikely to be used and working around the breakage would be too complex and would take too much time (for example adding a method in an interface requires writing a new interface and modifying all code accepting that interface). Luckily this is going to be made somewhat simpler in Java in the future with Default Methods (introduced in Java 8). It won't fit all cases though.

Another example of backward compatibility aspect that we don't handle is when someone changes what an API returns or what a API does. A simple example is if a method returns an object of type X but the implementation changes to return another implementation of type X that behaves differently... This is a tough one to detect and prevent.

WDYT? Are you doing something else in your projects? (I'd be curious of ways to improve even further what we do!).

This is a call to Jenkins developers and experts. I'm looking for a solution to the following problem.

Statement of the problem:

• On the XWiki project we have a Git repository with a lot of top level Maven modules (about 85 as of today)
• Each of these top level modules have several sub modules including modules that run functional tests (long to execute)
• Right now we build the full platform in a single job on Jenkins and this takes too long: about 1 hour. EDIT 26/09/2017: it now takes over 2 hours+

Note that we do not want to create many Git repositories like one repo per top level module since that makes it a lot harder to release all modules at once (we release them together with the same version) and it would mean creating lots of jobs manually (85!).

The ideal solution I can think of would be:

• Give to Jenkins the ability to create "virtual" jobs (one per top level module)
• Each such job would automatically have its dependencies on other jobs defined based on the Maven dependencies so that jobs wait automatically for jobs that need to be run before them
• This would allow to dispatch the full build to all the agents available

Doing this manually would be a real pain since it would mean creating 85 jobs and recreating the Maven dependencies with job triggers. Of course it should be possible to use the Scriptler plugin to automate this but parsing the Maven POMs to create the matching job hierarchy is not something trivial to do...

Does anyone have a solution for this? Do you agree it's a legitimate use case?

Thanks!

On the XWiki project we have a Jenkins Job that runs every night and checks that the quality of Maven modules have not decreased.

This is done by using the Maven Clover plugin and failing the build if the TPC is under a specified threshold for each module.

It's defined in a quality profile since it takes some time to execute (this is why we execute it once per night ATM).

Here's the setup:

Then in each Maven, we have (for example):

Now that's fine and it allows to find out when someone adds code in an existing module and doesn't add enough unit tests to keep the TPC above the current threshold.

There remains an issue. Imagine that I add some code with unit tests. I also need to not forget to update the TPC value in the pom.xml file.

So here's a quick command line tip to find out the current TPC max threshold for all modules located under the directory when you run it:

For example when run in xwiki-commons/xwiki-commons-core, it gives:

Now the next step is to write a script that will automatically change the pom.xml files with the max TPC threshold values.

UPDATE 2013-01-31: To do the same with Jacoco you would use:

UPDATE 2013-07-09: Starting with Jacoco 0.6.4 we need to use:

Having a CI tool set up is great. However if your CI starts sending a lot of build failure emails and a portion of these are false positives (i.e. for example CI infrastructure issues or flickering tests) then what will happen is that the developers are going to trust less and less your CI and builds will stay failing more and more till your release day when you're going to spend several days trying to stabilize your build and your tests before you can release. Thus delaying the release and starting on a downward slope...

It's thus very important that your CI only sends real build failures to the developers.

I've been looking for a way to do with Jenkins and I've found a solution that works (although I would have liked something simpler - If Jenkins experts read this and there's a better please let me know! ).

So my solution needs 2 Jenkins plugins:

• The Mail Ext plugin
• The Groovy Postbuild plugin
• The Scriptler plugin, for automation.

Without further ado, here's the script that you can run in Scriptler and which will modify the email notification of all your jobs:

After you run this script, if you check a job's configuration you'll find a Groovy postbuild step with the following script, which will set up a variable to tell the Email Ext plugin whether to skip sending an email or not:

And the Email Ext plugin will also have been configured and it will contain a prerun script that verifies if it should send an email or not.

Of course you should adapt the messages the script is looking for in your build log and adapt it to your cases.

For our need we catch the following:

• Check if the JVM has crashed
• Check if VNC is down (we run Selenium tests)
• Check for Browser crash (we run Selenium tests)
• Check for Git connection issue
• Check for machine slowness (if XWiki cannot start under 2 minutes then it means the machine has some problems)
• Check if Nexus is up (we deploy our artifacts to a Nexus reporitory)

Enjoy! ...

I just did a POC for recording the screen when a Selenium2 test executes. It was easy thanks to the Monte Media Library. I also adapted my code from this blog post.

Since I run in Maven the first step was to download the JAR (version 0.7.5 at the time of writing) and to install it in my local repository:

Then add it as a Maven dependency in my project:

And then to make sure my test starts and stop the recording:

Just make sure that startRecording is called before your setup runs so that you can record it too. The default directory where the recording is saved depends on your OS. On Mac it's in Movies. You can control that programmatically of course.

For the record, with the settings above, recording a test that ran for 23 seconds took only 2.1MB. Not bad

Eirik Bjørsnøs is the creator of SVNSearch, a tool to analyse your source repository (it supports SVN, CVS, Perforce and Git) and display its content using different views.

I have known about SVNSearch for several years now and I've always found this tool one of the best of its kind (I think it's ahead of the competition especially in the domains of speed, scalability and easiness or use). However I don't think the tool has received the publicity it deserves so I'll add my stone to help promote it.

So let's review some of its features by looking at the SVNSearch XWiki repository.

Amazingly Fast

The search are real fast and SNSearch supports real time updates: Go to your repository page on SVNSearch and make a commit. You'll see it appear in almost real time without doing any refresh (thanks to Cometd)

Dashboard View

You get a Dashboard-like view of any part of your repository (you get the same view with the graphs even down to the file level) with commits and graphs. Note that there's even a JIRA integration since the commits mentioning JIRA are hyperlinked automatically.

• Main Dashoard view (truncated) showing that 3 persons have worked on this Macro.java file

\

• File-level Dashboard view (truncated)

Powerful Search

You get a really powerful search form with Ajax-style autocomplete on path, filename, author and JIRA issues.

• "Did you mean" suggestions when searching on log content. This is based on actual content, not a dictionary, so for example searching for "explicitly" can suggest "explicitely" even though "explicitly" is correct

\

• Autocompletion on repository path

\

• Autocompletion on file. Note the CamelCase autocomplete: "SMI" offers "SpaceManagerImpl.java"

\

• Autocompletion on Authors

\

• Autocompletion on file types

\

• Autocompletion on JIRA issues

Activity Graphs

Useful to know the health of your project and how it progresses.

• Activity by Path (you can drill down by clicking in it)

\

• Activity over time (you can drill down by clicking in it)

\

• Activity by year (you can drill down by clicking in it)

\

• Activity by author (you can drill down by clicking in it)

\

• Committers over time

\

• Activity by day

\

• Activity by hour

\

• Committers Turnover

The last Committers Turnover graph needs to be viewed dynamically to better understand it. Since it's a bit complex here's how Eirik explains it: 

It's an visualisation of the composition of the development team over time. The algorithm goes like this: * Partition the timeline (first commit to last commit date) into 10 equally large partitions. * For each partition, identify the 20 percent most active committers * For each partition, create a graph showing how the relative activity of that group's changes over time

So what I end up with is a set of ten line graphs, each showing the relative amount of commits done by the group of most active commiters from each partition.

To be honest it's difficult even to explain it, so I can understand that people don't understand it intuitively

An way to look at it is to say that horizontal lines indicate a stable core group, while diagonal lines means that the core group is changing. So it can be used as an indicator of project health.

If you mouse-over the chart you should be able to see the actual contribution by each developer in each 20% group.

It's all based on a paper called "Contributor Turnover in Libre Software Projects", by Gregorio Robles and Jesus M. Gonzalez-Barahona.

Timeline View

This is a very useful graph for viewing each committer activity over time.

Collaboration View

Displays committers relationship (i.e. files they've worked together on) using a dynamic graph (you can zoom in/out, move nodes). Redish colors means recent activity while Bluish colors means old activity.

Diffs

SVNSearch can show commit list for a given file and also the diffs for each commit.

Other Features

• RSS feed for any search. This means you can follow development in a subproject or branch using any RSS reader.
• The from and to fields can actually contain revision ranges as well as date ranges.
• Author plots are sortable by name, first commit, last commit and number of commits.
• You can select multiple authors for comparison by separating them with a space or a comma.

Last I heard Eirik was running a private beta and wanted to sell this tool at some point in the future while keeping a free version on svnsearch.org for open source projects.