Category: \T\h\i\n\k\ \T\a\n\k (11 posts) /xwiki/resources/icons/silk/feed.png

Feb 06 2017

Jenkins going the Gradle way

Just realized that with the new Jenkins Pipeline strategy, Jenkins is actually moving towards a strategy similar to Gradle.

Before Gradle we had Maven which is using a Build by Configuration strategy. The idea is for users to tell Maven how to configure the build but not what it should do.

Before Pipeline, Jenkins Jobs were exactly that: you configured each job to give Jenkins each plugin's config, similar to Maven.

With Pipeline you now code your job in Groovy, specifying the what the job should do.

So you gain a lot of power to more precisely configure your jobs and an easier solution to reuse actions/configs between jobs. But you loose some simplicity and the fact that you could go to any Jenkins instance and understand what each job was doing easily. You now need to read code to understand what it's doing and everyone is going to have a different way of coding their jobs.

FYI I'm currently working on XWiki's Jenkinsfile. It's still simple at the moment but it'll become more complex as time passes.

Future will tell us if it's good or bad. FTM, being a dev, I'm enjoying it! emoticon_smile I especially like the perks that come with it (but which could have been implemented with a declarative job configuration too):

  • Save the CI job in the SCM next to the code
  • Ability to automatically add or remove jobs for SCM branches

See also my blog post about Jenkins GitHub Organization Jobs.

Feb 02 2017

Jenkins GitHub Organization Jobs

The Jenkins Pipeline plugin includes a very nice feature which is the "GitHub Organization" job type. This job type will scan a given GitHub organization repositories for Jenkinsfile files and when found will automatically create a pipeline job for them.

This has some nice advantages:

  • You save your Jenkins job configuration in your SCM (git in our case, in the Jenkinsfile), next to your code. You can receive email diffs to show who made modifications to it, the reason and understand the changes.
  • It supports branches: when you create a branch it's automatically discovered by Jenkins and the build is executed on it. And if the branch gets removed, it's removed automatically from Jenkins too. This point is awesome for us since we used to have groovy script to execute to copy jobs when there were new branches and when branches were removed.

So we started exploring this for the XWiki project, starting with Contrib Extensions.

Here's a screenshot of our Github Organization job for XWiki Contrib:

github-organization-contrib.png 

And here's an example of a pipeline job executing:

pipeline.png 

Now if you implement this you'll quickly find that you want to share pipeline scripts between Jenkinsfile, in order to not have duplicates.

FYI here's what the Jenkinsfile for the syntax-markdown pipeline job shown above looks like:

xwikiModule {
    name = 'syntax-markdown'
}

Simple, isn't it? emoticon_smile The trick is that we've configured Jenkins to automatically load a Global Pipeline Library (implicit load). You can do that by saving libraries at the root of SCM repositories and configure Jenkins to load them from the SCM sources (see this Jenkins doc for more details).

So we've created this GitHub repository and we've coded a vars/xwikiModule.groovy file. At the moment of writing this is its content (I expect it to be improved a lot in the near future):

// Example usage:
//   xwikiModule {
//     name = 'application-faq'
//     goals = 'clean install' (default is 'clean deploy')
//     profiles = 'legacy,integration-tests,jetty,hsqldb,firefox' (default is 'quality,legacy,integration-tests')
//  }

def call(body) {
   // evaluate the body block, and collect configuration into the object
   def config = [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

   // Now build, based on the configuration provided, using the followong configuration:
   // - config.name: the name of the module in git, e.g. "syntax-markdown"

    node {
       def mvnHome
       stage('Preparation') {
           // Get the Maven tool.
           // NOTE: Needs to be configured in the global configuration.
           mvnHome = tool 'Maven'
       }
        stage('Build') {
            dir (config.name) {
                checkout scm
               // Execute the XVNC plugin (useful for integration-tests)
               wrap([$class: 'Xvnc']) {
                    withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx1024m']) {
                     try {
                         def goals = config.goals ?: 'clean deploy'
                         def profiles = config.profiles ?: 'quality,legacy,integration-tests'
                          sh "mvn ${goals} jacoco:report -P${profiles} -U -e -Dmaven.test.failure.ignore"
                          currentBuild.result = 'SUCCESS'
                     } catch (Exception err) {
                          currentBuild.result = 'FAILURE'
                          notifyByMail(currentBuild.result)
                         throw e
                     }
                  }
               }
           }
       }
        stage('Post Build') {
           // Archive the generated artifacts
           archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
           // Save the JUnit test report
           junit testResults: '**/target/surefire-reports/TEST-*.xml'
       }
   }
}

def notifyByMail(String buildStatus) {
    buildStatus =  buildStatus ?: 'SUCCESSFUL'
   def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
   def summary = "${subject} (${env.BUILD_URL})"
   def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
    <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""


   def to = emailextrecipients([
           [$class: 'CulpritsRecipientProvider'],
           [$class: 'DevelopersRecipientProvider'],
           [$class: 'RequesterRecipientProvider']
   ])
   if (to != null && !to.isEmpty()) {
        mail to: to, subject: subject, body: details
   }
}

Ideas of some next steps:

Right now there's one limitation I've found: It seems I need to manually click on "Re-scan Organization" in the Jenkins UI so that new Jenkinsfile added to repositories are taken into account. I hope that will get fixed soon. One workaround would be to add another Jenkins job to do that regularly but it's not perfect. Also note that you absolutely must authenticate against GitHub as otherwise you'll quickly reach the GitHub API request limit (when authenticated you are allowed 5000 requests per hour).

Anyway it's great and I love it.

Dec 10 2016

Full Automated Test Coverage with Jenkins and Clover

Generating test coverage reports for a single Maven project is simple. You can use the Clover maven plugin easily for that. For example:

mvn clean clover:setup install clover:clover

Generating a report for several modules in the same Maven reactor (same build) is also easy since that's supported out of the box. For example:

mvn clean clover:setup install clover:aggregate clover:clover

However, generating a full coverage report for a multi-reactor project is much harder. Let's take the example of the XWiki project which has 4 separate Github repositories and thus 4 builds:

So the question is: How do we generate a single test coverage report for those 4 maven reactor builds. For example we want that tests that execute in the xwiki-enterprise repository generate coverage for source code located, say, in xwiki-commons.

Here's what we want to get:

dashboard.png 

The way to do this manually is to tell the Maven Clover plugin to use a single location for generating its data. Manually this can be achieved like this (more details can be found on the XWiki Test page):

# In xwiki-commons:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db
...
# In xwiki-enterprise:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

# From xwiki-enterprise, generate the full Clover report:
mvn clover:clover -N -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

This is already pretty cool. However it's taking a lot of time and it would be nicer if it could be executed on the CI (on http://ci.xwiki.org in our case).

One important note is that Clover modifies the artifacts and thus you need to be careful to not push them into production or make sure they're not used in other builds (since they'd fail since they'd need to have the Clover runtime JAR at execution time).

So, I chose to use Jenkins 2 and the new Pipeline plugin and used the following script (see the XWiki Clover Job):

node() {
  def mvnHome
  def localRepository
  def cloverDir
  stage('Preparation') {
    def workspace = pwd()
    localRepository = "${workspace}/maven-repository"
    // Make sure that the special Maven local repository for Clover exists
    sh "mkdir -p ${localRepository}"
    // Remove all XWiki artifacts from it
    sh "rm -Rf ${localRepository}/org/xwiki"
    sh "rm -Rf ${localRepository}/com/xpn"
    // Make sure that the directory where clover will store its data exists in
    // the workspace and that it's clean
    cloverDir = "${workspace}/clover-data"
    sh "rm -Rf ${cloverDir}"
    sh "mkdir -p ${cloverDir}"
    // Get the Maven tool.
    // NOTE: Needs to be configured in the global configuration.           
    mvnHome = tool 'Maven'
  }
  // each() has problems in pipeline, thus using a standard for()
  // See https://issues.jenkins-ci.org/browse/JENKINS-26481
  for (String repoName : ["xwiki-commons", "xwiki-rendering", "xwiki-platform", "xwiki-enterprise"]) {
    stage("Cloverify ${repoName}") {
      dir (repoName) {
        git "https://github.com/xwiki/${repoName}.git"
        runCloverAndGenerateReport(mvnHome, localRepository, cloverDir)
      }  
    }      
  }
  stage("Publish Clover Reports") {
    ...
  }
}
def runCloverAndGenerateReport(def mvnHome, def localRepository, def cloverDir) {
  wrap([$class: 'Xvnc']) {
    withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx2048m']) {
      sh "mvn -Dmaven.repo.local='${localRepository}' clean clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true"
      sh "mvn -Dmaven.repo.local='${localRepository}' clover:clover -N -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db"
    }
  }
}

Note that we use the "Xvnc" Jenkins plugin because we run Selenium2 functional tests which require a display.

When this Jenkins job is executed is results in:

pipeline.png 

Over 5 hours of build time... Now you understand why we want to have this running on the CI agent and not on my local machine emoticon_wink

And the generated reports can be seen on xwiki.org.

Good news, we have an overall coverage of 73.2% for the full XWiki Java codebase, that's not too bad (I thought it would be lower emoticon_wink).

Next blog post will be about trying to achieve this with the Jacoco Maven plugin and the associated challenges and issues... Hint: it's harder than with the Clover Maven plugin.

May 03 2016

Bye Bye CLIRR, Welcome Revapi!

On the XWiki project it's been years that we've been checking automatically for backward compatibilities in our APIs as part of our build (It's even documented here).

Recently, we've moved to Java 8 for XWiki 8.1 and we've discovered that the tool we were using, CLIRR, doesn't work anymore with Java 8. So I searched, without much hope, for an alternative, and I was surprised (and delighted) to find that there are 2 valid solutions nowadays that support Java 8:

I've tried both and they both had issues. However, I've discovered that the maintainers or these 2 solutions were really cool guys and they very quickly fixed any issue I've raised. Big thanks to Lukas Krejci and Martin Mois, you're awesome guys! emoticon_smile

In the end the XWiki project has had to make a difficult choice and we chose Revapi (Some reasons mentioned here but honestly they're both valid choices).

So here's how we use it now:

  • In our top level POM:
    ...
         <plugins>
    ...
           <!-- Used for checking backward compatibility (binary and source) -->
           <plugin>
             <groupId>org.revapi</groupId>
             <artifactId>revapi-maven-plugin</artifactId>
             <!-- Lock down plugin version for build reproducibility -->
             <version>0.4.5</version>
             <dependencies>
               <dependency>
                 <groupId>org.revapi</groupId>
                 <artifactId>revapi-java</artifactId>
                 <version>0.9.0</version>
               </dependency>
             </dependencies>
             <executions>
               <execution>
                 <id>revapi-check</id>
                 <goals>
                   <goal>check</goal>
                 </goals>
               </execution>
             </executions>
             <configuration>
               <oldVersion>${xwiki.compatibility.previous.version}</oldVersion>
               <skip>${xwiki.revapi.skip}</skip>
               <analysisConfiguration>
                {
                  "revapi": {
                    "java": {
                      "filter": {
                        "packages": {
                          "regex": true,
                          "include": ["org\\.xwiki\\..*"],
                          "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                        }
                      }
                    }
                  }
                }
               </analysisConfiguration>
             </configuration>
           </plugin>
    ...
  • Then in specific module POMs we override the configuration to add excludes (for backward and source incompatibilities that we know about and have voluntarily decided to do). For example:
    ...
         <plugin>
           <groupId>org.revapi</groupId>
           <artifactId>revapi-maven-plugin</artifactId>
           <configuration>
             <analysisConfiguration><![CDATA[
                {
                  "revapi": {
                    "java": {
                      "filter": {
                        "packages": {
                          "regex": true,
                          "include": ["org\\.xwiki\\..*"],
                          "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                        }
                      }
                    },
                    "ignore": [
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.List<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                        "new": "method java.util.List<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      },
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                        "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      },
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                        "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      }         
                    ]
                  }
                }
              ]]>
    </analysisConfiguration>
           </configuration>
         </plugin>
    ...
  • Now the interesting part is in generating reports. We've chosen an original way of doing this: We dynamically generate the Revapi report on our release notes wiki pages using Groovy. The big advantage is that there's no work to be done for the Release Manager:
    • Here's an example from the XWiki 8.1M1 Release Notes

      releasenotes.png

    • The Groovy script below does the following:
      • Use the GitHub REST API to get the content of the pom.xml containing the Revapi excludes.
      • Parse the XML using Groovy
      • Display a report out of it
    • And here's the Groovy script packaged as a Wiki Macro for the curious ones:
      {{groovy}}
      import groovy.json.*

      def getIgnores(def repo, def path)
      {
       def url = "https://api.github.com/repos/xwiki/${repo}/contents/${path}".toURL().text
       def result = new JsonSlurper().parseText(url)
       def xml = new String(result.content.decodeBase64())
        result = new XmlSlurper().parseText(xml)
       def revapi = result.build.plugins.plugin.'**'.find { node ->
          node.artifactId.text() == 'revapi-maven-plugin'
       }
       if (revapi) {
          result = new JsonSlurper().parseText(revapi.configuration.analysisConfiguration.text())
         return result.revapi.ignore
       } else {
         return ''
       }
      }

      def displayIgnores(def ignores)
      {
        result = new JsonSlurper().parseText(ignores)
        result.each() {
          it.each() {
            println "* {{{${it.justification}}}}"
            println "** Violation type: {{code}}${it.code}{{/code}}"
            println "** Old: {{code}}${it.old}{{/code}}"
           if (it.new) {
              println "** New: {{code}}${it.new}{{/code}}"
           }
         }
       }
      }

      def getViolations(def version)
      {
       def xobject = doc.getObject('ReleaseNotes.BackwardCompatibility')
       if (!xobject) {
          xobject = doc.newObject('ReleaseNotes.BackwardCompatibility')
         def commonsTag
         def renderingTag
         def platformTag
         if (version == 'master') {
            commonsTag = renderingTag = platformTag = 'master'
         } else {
            commonsTag = "xwiki-commons-${version}"
            renderingTag = "xwiki-rendering-${version}"
            platformTag = "xwiki-platform-${version}"
         }
         def jsonCommons = getIgnores('xwiki-commons', "xwiki-commons-core/pom.xml?ref=${commonsTag}")
         def jsonRendering = getIgnores('xwiki-rendering', "pom.xml?ref=${renderingTag}")
         def jsonPlatform = getIgnores('xwiki-platform', "xwiki-platform-core/pom.xml?ref=${platformTag}")
          xobject.set('violations', JsonOutput.prettyPrint(JsonOutput.toJson([jsonCommons, jsonRendering, jsonPlatform])))
          doc.save('Added backward-compatiblity violations data', true)
       }
       return xobject.getProperty('violations').value
      }

      displayIgnores(getViolations(xcontext.macro.params.version))
      {{/groovy}}

In conclusion we're very happy with the move, especially since we now have Java 8 support but also because Revapi provides more checks than what CLIRR was doing.
...

Jun 05 2015

Why is Jenkins's Incremental Build feature not working

On the XWiki project we have enabled Jenkins's Incremental Build feature:

jenkins.png

This seemed like a nice feature to speed up our CI when building our Maven jobs. Alas, it doesn't work!

The problem is that from time to time you'll get build failure such as:

Caused by: org.sonatype.aether.transfer.ArtifactNotFoundException: Could not find artifact org.xwiki.platform:xwiki-platform-store:pom:6.4.4-SNAPSHOT in local.central (xxx)

You'd think it's not possible, especially as we use Maven's -U flag and thus, even if the artifact is not present in the local repository it should be downloaded from the Maven Remote Repository that we use (and it's available there!).

The reason is because of a Maven bug: MNG-5542. What happens is that the Incremental Build feature will use the -pl Maven parameter to list all the Maven projects to build and when this feature is used, artifacts declared in the <parent> section of your POMs are just ignored and not downloaded...

The consequence is that your build will fail from time to time when one artifact declared in one of your <parent> is not present in your local repository... If you have decided to have one Maven repository per Job in Jenkins - which would seem a good idea to isolate your jobs and to be able to use the Parallel build feature of Jenkins - then you'll hit the problem very frequently...

So in the end you have to choose between Parallel builds and Incremental builds but you cannot have both at the same time!

Note that even with Parallel builds turned off, you build will fail from time to time, just less frequently...

One solution for the XWiki project would be to break our big job that builds the Platform (over 100 modules located in the same Git repo that we release together under a single version) into 100 jobs. But doing this manually is a no go so we would need to script this or wait for some Jenkins dev to implement this idea...

Big pain!

Jan 27 2015

A strategy for maintaining backward compatibility

I'd like to explain what we do on the XWiki open source project to preserve backward compatibility of our Java APIs. We've done several iterations and we now have a process that is working quite well, even though there are still some issues but they're unavoidable with Java for the time being.

So here it goes:

  • We start by having a special package named internal and everything we put in there is considered implementation details and should never be used by users of our APIs. We're allowed to modify anything at any time in the internal package. For example: org.xwiki.instance.internal.*
  • We also use a Component-based approach, which forces us to separate interfaces (which are public) from implementation (which go in the internal package).
  • We have defined an advanced deprecation mechanism which allows us to move deprecated code that we no longer use to legacy maven module (using Aspects with AspectJ) that can be optionally installed at runtime, and which we never use at build time. This prevents us from using any deprecated legacy code and it allows us to push away the cruft under the carpet emoticon_wink (without breaking backward compatibility!)
  • We have the notion of "Young API" (a.k.a Unstable APIs) and when a new API is introduced, the developer can (and should) use the @Unstable annotation to signify to users that this API is unstable for the moment and can change at any time. We only allow an API to remain unstable for a full development cycle. We've recently introduced a custom Checkstyle check that enforces this!
  • Last, we use the Maven CLIRR plugin to make sure we never break an API unintentionally (it fails our build). This allows us to only carefully and intentionally break our APIs. It also allows to us to mention what we break in our Release Notes (example).

The important point IMO is that we have automated tools to ensure that our strategy is applied, namely:

  • CLIRR
  • Our Unstable Annotation Checker (a custom checkstyle rule)

This is working pretty well for us even though we do break backward compatibility from time to time, when we judge that the code touched is unlikely to be used and working around the breakage would be too complex and would take too much time (for example adding a method in an interface requires writing a new interface and modifying all code accepting that interface). Luckily this is going to be made somewhat simpler in Java in the future with Default Methods (introduced in Java 8). It won't fit all cases though.

Another example of backward compatibility aspect that we don't handle is when someone changes what an API returns or what a API does. A simple example is if a method returns an object of type X but the implementation changes to return another implementation of type X that behaves differently... This is a tough one to detect and prevent.

WDYT? Are you doing something else in your projects? (I'd be curious of ways to improve even further what we do!).

Jun 29 2013

Issue: Jenkins and large Maven projects

This is a call to Jenkins developers and experts. I'm looking for a solution to the following problem.

Statement of the problem:

  • On the XWiki project we have a Git repository with a lot of top level Maven modules (about 85 as of today)
  • Each of these top level modules have several sub modules including modules that run functional tests (long to execute)
  • Right now we build the full platform in a single job on Jenkins and this takes too long: about 1 hour

Note that we do not want to create many Git repositories like one repo per top level module since that makes it a lot harder to release all modules at once (we release them together with the same version) and it would mean creating lots of jobs manually (85!).

The ideal solution I can think of would be:

  • Give to Jenkins the ability to create "virtual" jobs (one per top level module)
  • Each such job would automatically have its dependencies on other jobs defined based on the Maven dependencies so that jobs wait automatically for jobs that need to be run before them
  • This would allow to dispatch the full build to all the agents available

Doing this manually would be a real pain since it would mean creating 85 jobs and recreating the Maven dependencies with job triggers. Of course it should be possible to use the Scriptler plugin to automate this but parsing the Maven POMs to create the matching job hierarchy is not something trivial to do...

Does anyone have a solution for this? Do you agree it's a legitimate use case?

Thanks!

Jan 11 2013

Tip: Find out max Clover TPC for Maven modules

On the XWiki project we have a Jenkins Job that runs every night and checks that the quality of Maven modules have not decreased.

This is done by using the Maven Clover plugin and failing the build if the TPC is under a specified threshold for each module.

It's defined in a quality profile since it takes some time to execute (this is why we execute it once per night ATM).

Here's the setup:

   <!-- Profile for QA verifications that takes time -->
   <profile>
     <id>quality</id>
     <build>
       <plugins>
         <!-- Fail the build if the test coverage is below a given value. -->
         <plugin>
           <groupId>com.atlassian.maven.plugins</groupId>
           <artifactId>maven-clover2-plugin</artifactId>
           <configuration>
             <targetPercentage>${xwiki.clover.targetPercentage}</targetPercentage>
           </configuration>
           <executions>
             <execution>
               <id>clover-check</id>
               <phase>verify</phase>
               <goals>
                 <goal>instrument-test</goal>
                 <goal>check</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
     </build>
   </profile>

Then in each Maven, we have (for example):

...
 <properties>
   <xwiki.clover.targetPercentage>74.6%</xwiki.clover.targetPercentage>
 </properties>
...

Now that's fine and it allows to find out when someone adds code in an existing module and doesn't add enough unit tests to keep the TPC above the current threshold.

There remains an issue. Imagine that I add some code with unit tests. I also need to not forget to update the TPC value in the pom.xml file.

So here's a quick command line tip to find out the current TPC max threshold for all modules located under the directory when you run it:

mvn clean install -Pquality -Dxwiki.clover.targetPercentage=100%
-Dmaven.clover.failOnViolation=false 2>&1 |
awk '/Checking for coverage of/ { module = $9; }
/^Total/ { split(module, array, "/"); print array[length(array)-3],$4 }'

For example when run in xwiki-commons/xwiki-commons-core, it gives:

xwiki-commons-test-simple 0%
xwiki-commons-text 93.5%
xwiki-commons-component-api 22.7%
xwiki-commons-classloader-api 0%
xwiki-commons-classloader-protocol-jar 0%
xwiki-commons-observation-api 15.9%
xwiki-commons-component-observation 76.2%
xwiki-commons-component-default 74.6%
xwiki-commons-context 76.7%
xwiki-commons-script 0%
xwiki-commons-configuration-api 0%
xwiki-commons-test-component 0%
xwiki-commons-environment-api -100%
xwiki-commons-environment-common 0%
xwiki-commons-environment-standard 67.3%
xwiki-commons-environment-servlet 84.6%
xwiki-commons-properties 76.6%
xwiki-commons-logging-api 29.5%
xwiki-commons-observation-local 90.8%
xwiki-commons-job 36.1%
xwiki-commons-logging-logback 91.8%

Now the next step is to write a script that will automatically change the pom.xml files with the max TPC threshold values.

UPDATE 2013-01-31: To do the same with Jacoco you would use:

mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=100
-Djacoco.haltOnFailure=false 2>&1 |
awk '/jacoco-check/ { module = $6 } /Insufficient/ { print module, $7 }'

UPDATE 2013-07-09: Starting with Jacoco 0.6.4 we need to use:

mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=2.00
-Djacoco.haltOnFailure=false 2>&1 |
awk '/jacoco-check/ { module = $6 } /instructions covered ratio is/ { print module, $(NF-5) }'

Dec 22 2012

Jenkins: Don't send emails for false positives

Having a CI tool set up is great. However if your CI starts sending a lot of build failure emails and a portion of these are false positives (i.e. for example CI infrastructure issues or flickering tests) then what will happen is that the developers are going to trust less and less your CI and builds will stay failing more and more till your release day when you're going to spend several days trying to stabilize your build and your tests before you can release. Thus delaying the release and starting on a downward slope...

It's thus very important that your CI only sends real build failures to the developers.

I've been looking for a way to do with Jenkins and I've found a solution that works (although I would have liked something simpler - If Jenkins experts read this and there's a better please let me know! emoticon_smile).

So my solution needs 2 Jenkins plugins:

  • The Mail Ext plugin
  • The Groovy Postbuild plugin
  • The Scriptler plugin, for automation.

Without further ado, here's the script that you can run in Scriptler and which will modify the email notification of all your jobs:

import jenkins.model.*
import hudson.plugins.emailext.*
import hudson.plugins.emailext.plugins.trigger.*
import hudson.tasks.*
import hudson.maven.reporters.*
import org.jvnet.hudson.plugins.groovypostbuild.*
 
def instance = jenkins.model.Jenkins.instance

def jobNames = [
 "xwiki-commons.*",
 "xwiki-rendering.*",
 "xwiki-platform.*",
 "xwiki-enterprise.*",
 "xwiki-manager.*"
]
 
def deleteEmailConfiguration(def item, def publishers)
{
   // Remove the Maven Mailer if any since we want to use the Mail Ext plugin exclusively
   def reporters = item.reporters
   def mavenMailer = reporters.get(MavenMailer.class)
   if (mavenMailer) {
      println "  - Removing default mailer reporter with recipients [${mavenMailer.recipients}]"
      reporters.remove(mavenMailer)
    }

   // Remove any default Mailer Publisher
   def mailer = publishers.get(Mailer.class)
   if (mailer) {
      println "  - Removing ${Mailer.class.name} publisher"
      publishers.remove(Mailer.class)
    }

   // Remove any default ExtendedEmailPublisher Publisher
   def extMailer = publishers.get(ExtendedEmailPublisher.class)
   if (extMailer) {
      println "  - Removing ${ExtendedEmailPublisher.class.name} publisher"
      publishers.remove(ExtendedEmailPublisher.class)
    }
   
   // Remove any Groovy Postbuild Plugin
   def groovyPostbuild = publishers.get(GroovyPostbuildRecorder.class)
   if (groovyPostbuild) {
      println "  - Removing ${groovyPostbuild.class.name} publisher"
      publishers.remove(GroovyPostbuildRecorder.class)
    }    
}

 
instance.items.each() { item ->
  println "Checking ${item.name}..."

 def match = false;
  jobNames.each() {
   if (item.name.matches(it)) {
      match = true;
    }
  }

 if (match) {
   if (configureEmailForMatching.equals("true")) {
      println "  - Modifying ${item.name}..."
      
     def publishers = item.publishersList
      deleteEmailConfiguration(item, publishers)

     // Add the Groovy Postbuild Plugin
     def script = '''
import hudson.model.*

def messages = [
  [".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
  [".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
  [".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
  [".hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
  [".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
  [".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
  [".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"]
]

def shouldSendEmail = true
messages.each() { message ->
  if (manager.logContains(message.get(0))) {
    manager.addWarningBadge(message.get(1))
    manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
    manager.buildUnstable()
    shouldSendEmail = false
  }
}

if (!shouldSendEmail) {
  def pa = new ParametersAction([
    new BooleanParameterValue("noEmail", true)
  ])
  manager.build.addAction(pa)
}
    '''


     def gp = new GroovyPostbuildRecorder(script, [], 0)
      println("  - Adding new ${gp.class.name} publisher")
      publishers.replace(gp);      
     
     // Add an Email Ext Publisher and configure it
     def ep = new ExtendedEmailPublisher()
      ep.defaultSubject = "\$DEFAULT_SUBJECT"
      ep.defaultContent = "\$DEFAULT_CONTENT"      
      ep.recipientList = "notifications@xwiki.org"
   
     // Add script to not send email for false positives
      ep.presendScript = '''
import hudson.model.*

build.actions.each() { action ->
  if (action instanceof ParametersAction) {
    if (action.getParameter("noEmail")) {
      cancel = true
    }
  }
}                                         
    '''

     
      ep.configuredTriggers.add(new FailureTrigger(
        email : new EmailType(sendToRecipientList : true,
          body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
          subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))

     // We don't want en email fro fixed builds now since it means build that have failed because of infra reasons will generate an email when they work again
     /*
      ep.configuredTriggers.add(new FixedTrigger(
        email : new EmailType(sendToRecipientList : true,
          body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
          subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
     */
      println("  - Adding new ${ep.class.name} publisher")
      publishers.replace(ep);      
     
      item.save()
    } else {
      println "  - Skipping ${item.name}"
    }
  } else {
   if (removeEmailForNonMatching.equals("true")) {
      println "  - Removing email configuration for ${item.name}..."
     def publishers = item.publishersList
      deleteEmailConfiguration(item, publishers)
      item.save()      
    } else {
      println "  - Skipping ${item.name}"
    }
  }
}

After you run this script, if you check a job's configuration you'll find a Groovy postbuild step with the following script, which will set up a variable to tell the Email Ext plugin whether to skip sending an email or not:

import hudson.model.*

def messages = [
  [".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
  [".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
  [".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
  [".hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
  [".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
  [".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
  [".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"]
]

def shouldSendEmail = true
messages.each() { message ->
 if (manager.logContains(message.get(0))) {
    manager.addWarningBadge(message.get(1))
    manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
    manager.buildUnstable()
    shouldSendEmail = false
  }
}

if (!shouldSendEmail) {
 def pa = new ParametersAction([
    new BooleanParameterValue("noEmail", true)
  ])
  manager.build.addAction(pa)
}

And the Email Ext plugin will also have been configured and it will contain a prerun script that verifies if it should send an email or not.

import hudson.model.*

build.actions.each() { action ->
 if (action instanceof ParametersAction) {
   if (action.getParameter("noEmail")) {
      cancel = true
    }
  }
}                                         

Of course you should adapt the messages the script is looking for in your build log and adapt it to your cases.

For our need we catch the following:

  • Check if the JVM has crashed
  • Check if VNC is down (we run Selenium tests)
  • Check for Browser crash (we run Selenium tests)
  • Check for Git connection issue
  • Check for machine slowness (if XWiki cannot start under 2 minutes then it means the machine has some problems)
  • Check if Nexus is up (we deploy our artifacts to a Nexus reporitory)

Enjoy!
...

Dec 03 2012

Screen Recording for Selenium2

I just did a POC for recording the screen when a Selenium2 test executes. It was easy thanks to the Monte Media Library. I also adapted my code from this blog post.

Since I run in Maven the first step was to download the JAR (version 0.7.5 at the time of writing) and to install it in my local repository:

mvn install:install-file -Dfile=~/Downloads/MonteScreenRecorder.jar -DgroupId=org.monte -DartifactId=monte-screen-recorder -Dversion=0.7.5 -Dpackaging=jar

Then add it as a Maven dependency in my project:

<dependency>
 <groupId>org.monte</groupId>
 <artifactId>monte-screen-recorder</artifactId>
 <version>0.7.5</version>
 <scope>test</scope>
</dependency>

And then to make sure my test starts and stop the recording:

...
import java.awt.*;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.monte.media.Format;
import org.monte.media.math.Rational;
import org.monte.screenrecorder.ScreenRecorder;
import static org.monte.media.VideoFormatKeys.*;

...
   private ScreenRecorder screenRecorder;

   public void startRecording() throws Exception
   {
        GraphicsConfiguration gc = GraphicsEnvironment
           .getLocalGraphicsEnvironment()
           .getDefaultScreenDevice()
           .getDefaultConfiguration();

       this.screenRecorder = new ScreenRecorder(gc,
           new Format(MediaTypeKey, MediaType.FILE, MimeTypeKey, MIME_AVI),
           new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                CompressorNameKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                DepthKey, 24, FrameRateKey, Rational.valueOf(15),
                QualityKey, 1.0f,
                KeyFrameIntervalKey, 15 * 60),
           new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey, "black",
                FrameRateKey, Rational.valueOf(30)),
           null);

       this.screenRecorder.start();
   }

   public void stopRecording() throws Exception
   {
       this.screenRecorder.stop();
   }
...

Just make sure that startRecording is called before your setup runs so that you can record it too. The default directory where the recording is saved depends on your OS. On Mac it's in Movies. You can control that programmatically of course.

For the record, with the settings above, recording a test that ran for 23 seconds took only 2.1MB. Not bad emoticon_smile

Tags:
Created by Vincent Massol on 2008/12/18 22:21
This wiki is licensed under a Creative Commons 2.0 license