Wiki Home » On software quality, collaborative development, wikis and Open Source...

On software quality, collaborative development, wikis and Open Source...

Last modified by Vincent Massol on 2010/12/22 14:19
Comments (0) · Annotations (0)

Apr 20 2013

CodeCamp Iasi 2013

I'm in Iasi, Romania for 1 week visiting the XWiki SAS Romanian office and I was invited to talk at CodeCamp 2013. It was my first time I spoke in Romania and it was fun!

The conference is nice (about 300 participants I'd say); the only issue for me being that all sessions are in ... Romanian... emoticon_smile

I gave the talk I did at Devoxx France 2013 last month. It's a technical talk and I wasn't sure how it would get appreciated since it requires some prerequisite knowledge of Maven, Jenkins and Software factories in general... Seems people like it overall.

Here are the slides:

Mar 31 2013

Devoxx France 2013

I was very happy to present at Devoxx France again! Big kudo to the organizers, it was a great event. As usual it was great to meet all my Java friends.

This year I was happy to be on 4 events:

Here are the slides of my quality on Java projects:

In the slides I mention some Scriptler script to automated modifying all jobs to prevent false positive emails.

A big thank you to Frédéric Bouquet who volunteered during the Hackergarten to work on participating to XWiki. We ended up coding a new CRaSH integration. It's still in progress but I hope we can release a 1.0 version real soon.

See you next year!

Jan 11 2013

Tip: Find out max Clover TPC for Maven modules

On the XWiki project we have a Jenkins Job that runs every night and checks that the quality of Maven modules have not decreased.

This is done by using the Maven Clover plugin and failing the build if the TPC is under a specified threshold for each module.

It's defined in a quality profile since it takes some time to execute (this is why we execute it once per night ATM).

Here's the setup:

   <!-- Profile for QA verifications that takes time -->
   <profile>
     <id>quality</id>
     <build>
       <plugins>
         <!-- Fail the build if the test coverage is below a given value. -->
         <plugin>
           <groupId>com.atlassian.maven.plugins</groupId>
           <artifactId>maven-clover2-plugin</artifactId>
           <configuration>
             <targetPercentage>${xwiki.clover.targetPercentage}</targetPercentage>
           </configuration>
           <executions>
             <execution>
               <id>clover-check</id>
               <phase>verify</phase>
               <goals>
                 <goal>instrument-test</goal>
                 <goal>check</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
     </build>
   </profile>

Then in each Maven, we have (for example):

...
 <properties>
   <xwiki.clover.targetPercentage>74.6%</xwiki.clover.targetPercentage>
 </properties>
...

Now that's fine and it allows to find out when someone adds code in an existing module and doesn't add enough unit tests to keep the TPC above the current threshold.

There remains an issue. Imagine that I add some code with unit tests. I also need to not forget to update the TPC value in the pom.xml file.

So here's a quick command line tip to find out the current TPC max threshold for all modules located under the directory when you run it:

mvn clean install -Pquality -Dxwiki.clover.targetPercentage=100%
-Dmaven.clover.failOnViolation=false 2>&1 |
awk '/Checking for coverage of/ { module = $9; }
/^Total/ { split(module, array, "/"); print array[length(array)-3],$4 }'

For example when run in xwiki-commons/xwiki-commons-core, it gives:

xwiki-commons-test-simple 0%
xwiki-commons-text 93.5%
xwiki-commons-component-api 22.7%
xwiki-commons-classloader-api 0%
xwiki-commons-classloader-protocol-jar 0%
xwiki-commons-observation-api 15.9%
xwiki-commons-component-observation 76.2%
xwiki-commons-component-default 74.6%
xwiki-commons-context 76.7%
xwiki-commons-script 0%
xwiki-commons-configuration-api 0%
xwiki-commons-test-component 0%
xwiki-commons-environment-api -100%
xwiki-commons-environment-common 0%
xwiki-commons-environment-standard 67.3%
xwiki-commons-environment-servlet 84.6%
xwiki-commons-properties 76.6%
xwiki-commons-logging-api 29.5%
xwiki-commons-observation-local 90.8%
xwiki-commons-job 36.1%
xwiki-commons-logging-logback 91.8%

Now the next step is to write a script that will automatically change the pom.xml files with the max TPC threshold values.

UPDATE 2013-01-31: To do the same with Jacoco you would use:

mvn clean install -Pquality -Dxwiki.jacoco.instructionRatio=100
-Djacoco.haltOnFailure=false 2>&1 |
awk '/jacoco-check/ { module = $6 } /Insufficient/ { print module, $7 }'

Dec 22 2012

Jenkins: Don't send emails for false positives

Having a CI tool set up is great. However if your CI starts sending a lot of build failure emails and a portion of these are false positives (i.e. for example CI infrastructure issues or flickering tests) then what will happen is that the developers are going to trust less and less your CI and builds will stay failing more and more till your release day when you're going to spend several days trying to stabilize your build and your tests before you can release. Thus delaying the release and starting on a downward slope...

It's thus very important that your CI only sends real build failures to the developers.

I've been looking for a way to do with Jenkins and I've found a solution that works (although I would have liked something simpler - If Jenkins experts read this and there's a better please let me know! emoticon_smile).

So my solution needs 2 Jenkins plugins:

  • The Mail Ext plugin
  • The Groovy Postbuild plugin
  • The Scriptler plugin, for automation.

Without further ado, here's the script that you can run in Scriptler and which will modify the email notification of all your jobs:

import jenkins.model.*
import hudson.plugins.emailext.*
import hudson.plugins.emailext.plugins.trigger.*
import hudson.tasks.*
import hudson.maven.reporters.*
import org.jvnet.hudson.plugins.groovypostbuild.*
 
def instance = jenkins.model.Jenkins.instance

def jobNames = [
 "xwiki-commons.*",
 "xwiki-rendering.*",
 "xwiki-platform.*",
 "xwiki-enterprise.*",
 "xwiki-manager.*"
]
 
def deleteEmailConfiguration(def item, def publishers)
{
    // Remove the Maven Mailer if any since we want to use the Mail Ext plugin exclusively
    def reporters = item.reporters
    def mavenMailer = reporters.get(MavenMailer.class)
    if (mavenMailer) {
      println "  - Removing default mailer reporter with recipients [${mavenMailer.recipients}]"
      reporters.remove(mavenMailer)
    }

    // Remove any default Mailer Publisher
    def mailer = publishers.get(Mailer.class)
    if (mailer) {
      println "  - Removing ${Mailer.class.name} publisher"
      publishers.remove(Mailer.class)
    }

    // Remove any default ExtendedEmailPublisher Publisher
    def extMailer = publishers.get(ExtendedEmailPublisher.class)
    if (extMailer) {
      println "  - Removing ${ExtendedEmailPublisher.class.name} publisher"
      publishers.remove(ExtendedEmailPublisher.class)
    }
   
    // Remove any Groovy Postbuild Plugin
    def groovyPostbuild = publishers.get(GroovyPostbuildRecorder.class)
    if (groovyPostbuild) {
      println "  - Removing ${groovyPostbuild.class.name} publisher"
      publishers.remove(GroovyPostbuildRecorder.class)
    }    
}

 
instance.items.each() { item ->
  println "Checking ${item.name}..."

  def match = false;
  jobNames.each() {
    if (item.name.matches(it)) {
      match = true;
    }
  }

  if (match) {
    if (configureEmailForMatching.equals("true")) {
      println "  - Modifying ${item.name}..."
      
      def publishers = item.publishersList
      deleteEmailConfiguration(item, publishers)

      // Add the Groovy Postbuild Plugin
      def script = '''
import hudson.model.*

def messages = [
 [".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
 [".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
 [".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
 [".hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
 [".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
 [".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
 [".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"]
]

def shouldSendEmail = true
messages.each() { message ->
  if (manager.logContains(message.get(0))) {
    manager.addWarningBadge(message.get(1))
    manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
    manager.buildUnstable()
    shouldSendEmail = false
  }
}

if (!shouldSendEmail) {
  def pa = new ParametersAction([
   new BooleanParameterValue("noEmail", true)
 ])
  manager.build.addAction(pa)
}
    '''

      def gp = new GroovyPostbuildRecorder(script, [], 0)
      println("  - Adding new ${gp.class.name} publisher")
      publishers.replace(gp);      
     
      // Add an Email Ext Publisher and configure it
      def ep = new ExtendedEmailPublisher()
      ep.defaultSubject = "\$DEFAULT_SUBJECT"
      ep.defaultContent = "\$DEFAULT_CONTENT"      
      ep.recipientList = "notifications@xwiki.org"
   
      // Add script to not send email for false positives
      ep.presendScript = '''
import hudson.model.*

build.actions.each() { action ->
  if (action instanceof ParametersAction) {
    if (action.getParameter("noEmail")) {
      cancel = true
    }
  }
}                                         
    '''
     
      ep.configuredTriggers.add(new FailureTrigger(
        email : new EmailType(sendToRecipientList : true,
          body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
          subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))

      // We don't want en email fro fixed builds now since it means build that have failed because of infra reasons will generate an email when they work again
      /*
      ep.configuredTriggers.add(new FixedTrigger(
        email : new EmailType(sendToRecipientList : true,
          body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
          subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
      */
      println("  - Adding new ${ep.class.name} publisher")
      publishers.replace(ep);      
     
      item.save()
    } else {
      println "  - Skipping ${item.name}"
    }
  } else {
    if (removeEmailForNonMatching.equals("true")) {
      println "  - Removing email configuration for ${item.name}..."
      def publishers = item.publishersList
      deleteEmailConfiguration(item, publishers)
      item.save()      
    } else {
      println "  - Skipping ${item.name}"
    }
  }
}

After you run this script, if you check a job's configuration you'll find a Groovy postbuild step with the following script, which will set up a variable to tell the Email Ext plugin whether to skip sending an email or not:

import hudson.model.*

def messages = [
 [".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
 [".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
 [".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
 [".hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
 [".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
 [".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
 [".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"]
]

def shouldSendEmail = true
messages.each() { message ->
  if (manager.logContains(message.get(0))) {
    manager.addWarningBadge(message.get(1))
    manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
    manager.buildUnstable()
    shouldSendEmail = false
  }
}

if (!shouldSendEmail) {
  def pa = new ParametersAction([
   new BooleanParameterValue("noEmail", true)
 ])
  manager.build.addAction(pa)
}

And the Email Ext plugin will also have been configured and it will contain a prerun script that verifies if it should send an email or not.

import hudson.model.*

build.actions.each() { action ->
 if (action instanceof ParametersAction) {
   if (action.getParameter("noEmail")) {
      cancel = true
    }
  }
}                                         

Of course you should adapt the messages the script is looking for in your build log and adapt it to your cases.

For our need we catch the following:

  • Check if the JVM has crashed
  • Check if VNC is down (we run Selenium tests)
  • Check for Browser crash (we run Selenium tests)
  • Check for Git connection issue
  • Check for machine slowness (if XWiki cannot start under 2 minutes then it means the machine has some problems)
  • Check if Nexus is up (we deploy our artifacts to a Nexus reporitory)

Enjoy!
...

Dec 20 2012

Devoxx 2012 Belgium

It was a long since I last attended Devoxx Belgium. It was a pleasure to be there again and meet all my friends. I was also happy to be able to present XWiki even though it was only for a quickie (15 minutes).

I presented the ability to quickly create applications within a wiki with the "Application Within Minute" feature of XWiki.

Here's the video:


...

Dec 03 2012

Screen Recording for Selenium2

I just did a POC for recording the screen when a Selenium2 test executes. It was easy thanks to the Monte Media Library. I also adapted my code from this blog post.

Since I run in Maven the first step was to download the JAR (version 0.7.5 at the time of writing) and to install it in my local repository:

mvn install:install-file -Dfile=~/Downloads/MonteScreenRecorder.jar -DgroupId=org.monte -DartifactId=monte-screen-recorder -Dversion=0.7.5 -Dpackaging=jar

Then add it as a Maven dependency in my project:

<dependency>
 <groupId>org.monte</groupId>
 <artifactId>monte-screen-recorder</artifactId>
 <version>0.7.5</version>
 <scope>test</scope>
</dependency>

And then to make sure my test starts and stop the recording:

...
import java.awt.*;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.monte.media.Format;
import org.monte.media.math.Rational;
import org.monte.screenrecorder.ScreenRecorder;
import static org.monte.media.VideoFormatKeys.*;

...
   private ScreenRecorder screenRecorder;

   public void startRecording() throws Exception
   {
        GraphicsConfiguration gc = GraphicsEnvironment
           .getLocalGraphicsEnvironment()
           .getDefaultScreenDevice()
           .getDefaultConfiguration();

       this.screenRecorder = new ScreenRecorder(gc,
           new Format(MediaTypeKey, MediaType.FILE, MimeTypeKey, MIME_AVI),
           new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                CompressorNameKey, ENCODING_AVI_TECHSMITH_SCREEN_CAPTURE,
                DepthKey, 24, FrameRateKey, Rational.valueOf(15),
                QualityKey, 1.0f,
                KeyFrameIntervalKey, 15 * 60),
           new Format(MediaTypeKey, MediaType.VIDEO, EncodingKey, "black",
                FrameRateKey, Rational.valueOf(30)),
           null);

       this.screenRecorder.start();
   }

   public void stopRecording() throws Exception
   {
       this.screenRecorder.stop();
   }
...

Just make sure that startRecording is called before your setup runs so that you can record it too. The default directory where the recording is saved depends on your OS. On Mac it's in Movies. You can control that programmatically of course.

For the record, with the settings above, recording a test that ran for 23 seconds took only 2.1MB. Not bad emoticon_smile

Oct 12 2012

XWiki @ the Alpes and Mars JUG 2012

I was invited to speak about XWiki at the Alpes JUG and the Mars JUG on 10th and 11th of October 2012. I gave 3 presentations at each JUG:

  • Presentation of XWiki from a user point of view (sorry, no slides since this was done purely as a demo) (30 mn)
  • Explain how XWiki can be used to develop web applications quickly and why it can be considered a development platform (1.5 hours)
  • Present how the XWiki open source development is performed (1 hour)

I had a great time at both JUGs. The Mars JUG was a bit more active than the Alpes one with over 30 persons present vs 15 (come on guys, now you need to beat the Mars JUG next time! emoticon_wink). 

OTOH my stay at Grenoble was a bit safer than the Marseille trip... Indeed, I got to witness a murder in Marseille a few minutes after it happened...

On the food topic, I had some great sushis with Julien Viet in Marseille (seems the murder we witnessed didn't make us loose our appetite... emoticon_wink).

Ok so here are the slides (I also did lots of demo during the talk and unfortunately you won't be able to see them - I also spoke a lot more than on the slides, sorry about that - But then you should haven been there! ).

Thanks to Emmanuel Hugonnet and Julien Viet for inviting me.

Jun 21 2012

Breizhcamp 2012: XWiki used to build a Conference Site

Breizhcamp 2012 was a very nice conference (200 attendees roughly) organized by Nicolas DeLoof and friends.

I had the opportunity to present XWiki and more specifically I did a demonstration of XWiki used as a web development platform to construct a possible Breizhcamp 2013 web site, featuring the ability to register talks and speakers, and automatically generate a session calendar based on the entered talks. Ability to see all speaker avatars and to search for sessions. Also demonstrated how to apply a Boostrap-based skin to make the site look like a web site.

I also had some fun showing how to import Gravatars automatically using a Groovy Script.

In addition I also got to record a live LesCastCodeurs episode.

Well done to Nicolas and all who helped organize this event. It was very nice and I'll happily come back next year (especially if the website is done using XWiki emoticon_wink).

Here are some screenshots of the result.

Conference home page demo with default XWiki Skin
breizhcamp5.png

Creation of the application to register Talks
breizhcamp7.png

Registration of Talks
breizhcamp6.png

Edition of a Talk
breizhcamp8.png

Conference home page demo with the Lyrebird Skin (bootstrap-based)
breizhcamp1.png

Calendar generated automatically by the registered Talks
(Technically a wiki page is created to generate JSON with some Groovy scripting on the XWiki Model, this JSON is then used as input for the Javascript Calendar)
breizhcamp2.png

Ability to search for Talks with a Livetable
(Note: The Lyrebird Skin used was missing some CSS for a nice Livetable styling)
breizhcamp3.png

Dynamic list of all speaker avatars
breizhcamp4.png

Oct 30 2011

Paris JUG: The job of Developer

I was happy to be invited to talk about the job of Developer at the Paris JUG along with Jean-Laurent De Morlhon, Regis Medina, Didier Girard  and Gregory Weinbach.

The idea was to give a glimpse about how we became developers through our personal experience so that younger developers or developers-to-be could understand what it means to be a developer.

We then tried to answer questions from the room.

The pictures of the even have now been put online:

It was a great event and as you can see from the pictures I enjoyed it a lot! emoticon_smile

Thanks Nicolas for organizing this.

Sep 14 2011

Lausanne JUG: XWiki

Last week I've presented 2 sessions at the Lausanne JUG (organized by Philippe Kernevez from OCTO Technology:

There were 20-30 people attending and I've had a lot of fun presenting. The evening was followed by a dinner with several attendees. I've had a very nice time! 

The presentation slides are available with voiceover and videos of demos on Parleys.com.

Tags:
Created by Admin on 2005/01/28 15:50

This wiki is licensed under a Creative Commons 2.0 license
XWiki Enterprise 5.0.1-NODE1 - Documentation