The STAMP research project has been a boon for the XWiki open source project. It has allowed the project to gain a lot in terms of quality. There have been lots of areas of improvements but the main domains that have substantially benefited the quality are:

• Increase of the test coverage. At the start of STAMP the XWiki project already had a substantial automated suite of tests covering 65.29% of the whole case base. Not only the project was able to increase it to over 71.4% (a major achievement for a large code base of 500K NCLOC such as XWiki) but also improve the quality of these tests themselves thanks to increasing the mutation score for them by using PIT/Descartes.
  • XWiki’s build and CI now fail when either coverage or mutation score are reduced
• Addition of configuration testing. At the start of STAMP the XWiki project was only testing automatically a single configuration (latest HSQLDB + latest Jetty + latest Firefox). From time to time some testers were doing manual tests on different configurations but this was a very intensive process and very random and ad hoc. We were having a substantial number of issues raised in the XWiki issue tracker about configuration-related bugs. Thanks to STAMP the XWiki project has been able to cover all the configurations it supports and to execute its functional UI tests on all of them (every day, every week and every month based on different criteria). This is leading to a huge improvement in quality and in developer's productivity since developers don't need to manually setup multiple environments on their machine to test their new code (that was very time-consuming and difficult when onboarding new developers).

XWiki has spread its own engineering practices to the other STAMP project members and has benefitted from the interactions with the students, researchers and project members to accrue and firm up its own practices.

Last but not least, the STAMP research project has allowed the XWiki project to get time to work on testing in general, on its CI/CD pipeline, on adding new distribution packagings (such as the new Docker-based distribution which has increased XWiki's Active Installs) which were prerequisites for STAMP-developed tools, but which have tremendous benefits by themselves even outside of pure testing. Globally, this has raised the bar for the project, placing it even higher in the category of projects with a strong engineering practice with controlled quality. The net result is a better and more stable product with an increased development productivity. Globally STAMP has allowed a European software editor (XWiki SAS) and open source software (XWiki) to match and possibly even surpass non-European (American, etc) software editors in terms of engineering practices.

So while it's the end of STAMP, its results continue to live on inside the XWiki project.

I'm personally very happy to have been part of STAMP and I have learnt a lot about the topics at hand but also about European research projects (it was my first project).

On the XWiki project we wanted to generate custom XWiki WARs directly from our unit tests (to deploy XWiki in Docker containers automatically and directly from the tests).

I was really excited when I discovered the SkrinkWrap Resolver library. It looked exactly to be what I needed. I didn't want to use Aether (now deprecated) or the new Maven Resolver (wasn't sure what the state was and very little doc to use it).

So I coded the whole WAR generation (pom.xml). Here are some extracts showing you how easy it is to use ShrinkWrap.

Example to find all dependencies of an Artifact:

Here's another example to read the version from a resolved pom.xml file (didn't find how to do that easily with Maven Resolver BTW):

And here's how to resolve a single Artifact:

Pretty nice, isn't it?

It looked nice till I tried to use the generated WAR... Then, all my hopes disappeared... There's one big issue: it seems that ShrinkWrap will resolve dependencies by using a strategy different than what Maven does:

• Maven: Maven takes the artifact closest to the top (From the Maven web site: "Dependency mediation - this determines what version of a dependency will be used when multiple versions of an artifact are encountered. Currently, Maven 2.0 only supports using the "nearest definition" which means that it will use the version of the closest dependency to your project in the tree of dependencies.").
• ShrinkWrap: First artifact found in the tree (navigating each dependency node to the deepest).

So this led to a non-functional XWiki WAR with completely different JAR versions than what is generated by our Maven build.

To this day, I still don't know if that's a known bug and since nobody was replying to my thread on the ShrinkWrap forum I created an issue to track it. So far no answer. I hope someone from the ShrinkWrap project will reply.

Conclusion: Time to use the Maven Resolver library... Spoiler: I've succeeded in doing the same thing with it (and I get the same result as with mvn on the command line) and I'll report that in a blog post soon.

I've been trying to define the best solution for doing functional testing in multiple environments (different DBs, Servlet containers, browsers) for XWiki. Right now on XWiki we test automatically on a single environment (Firefox, Jetty, HSQLDB) and we do the other environment tests manually.

So I've been going through different experimentations, finding out issues and limitations with them and progressing towards the perfect solution for us. Here are the various experiments we conducted. Note that this work is being done as part of the STAMP Research Project.

Here are the use cases that we want to support ideally:

• UC1: Fast to start XWiki in a given environment/configuration
• UC2: Solution must be usable both for running the functional tests and for distributing XWiki
• UC3: Be able to execute tests both on CI and locally on developer's machines
• UC4: Be able to debug functional tests easily locally
• UC5: Support the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
• UC6: Choose a solution that's as fast as possible for functional test executions

Experiments:

• Experimentation 1: CAMP from STAMP
• Experimentation 2: Docker on CI
• Experimentation 3: Maven build with Fabric8
• Experimentation 4: In Java Tests using Selenium Jupiter
• Experimentation 5: in Java Tests using TestContainers
• Conclusion

Experimentation 1: CAMP from STAMP

CAMP is a tool developed by some partners on the STAMP research project and it acts as a remote testing service. You give it some Dockerfile and it'll create the image, start a container, execute some commands (that you also provide to it and that are used to validate that the instance is working fine) and then stop the container. In addition it performs configuration mutation on the provided Dockerfile. This means it'll make some variations to this file, regenerate the image and re-run the container.

Here's how CAMP works (more details including how to use it on XWiki can be found on the CAMP home page):

Image from the CAMP web site

Limitations for the XWiki use case needs:

• Relies on a service. This service can be installed on a server on premises too but that means more infrastructure to maintain for the CI subsystem. Would be better if integrated directly in Jenkins for example.
• Cannot easily run on the developer machine which is important so that devs can test what they develop on various environments and so that they can debug reported issues on various environments. This fails at least UC3 and UC4.
• Even though mutation of configuration is an interesting concept, it's not a use case for XWiki which has several well-defined configurations that are supported. It's true that it could be interesting to have fixed topologies and only vary versions of servers (DB version, Servlet Container version and Java version - We don't need to vary Browser versions since we support only the latest version) but we think the added value vs the infrastructure cost might not be that interesting for us. However, it could still be interesting for example by randomizing the mutated configuration and only running tests on one such configuration per day to reduce the need of having too many agents and leaving them free for the other jobs.

Experimentation 2: Docker on CI

I blogged in the past about this strategy.

The main idea for this experiment was to use a Jenkins Pipeline with the Jenkins Plugin for Docker, allowing to write pipelines like this:

Limitations:

• Similar to experimentation 1 with CAMP, this relies on the CI to execute the tests and doesn't allow developers to test and reproduce issues on their local machines. This fails at least UC3 and UC4.

Experimentation 3: Maven build with Fabric8

The next idea was to implement the logic in the Maven build so that it could be executed on developer machines. I found the very nice Fabric8 Maven plugin and came up with the following architecture that I tried to implement:

The main ideas:

• Generate the various Docker images we need (the Servlet Container one and the one containing Maven and the Browsers) using Fabric8 in a Maven module. For example to generate the Docker image containing Tomcat and XWiki:

  pom.xml file:

  [...]
  <plugin>
   <groupId>io.fabric8</groupId>
   <artifactId>docker-maven-plugin</artifactId>
   <configuration>
     <imagePullPolicy>IfNotPresent</imagePullPolicy>
     <images>
       <image>
         <alias>xwiki</alias>
         <name>xwiki:latest</name>
         <build>
           <tags>
             <tag>${project.version}-mysql-tomcat</tag>
             <tag>${project.version}-mysql</tag>
             <tag>${project.version}</tag>
           </tags>
           <assembly>
             <name>xwiki</name>
             <targetDir>/maven</targetDir>
             <mode>dir</mode>
             <descriptor>assembly.xml</descriptor>
           </assembly>
           <dockerFileDir>.</dockerFileDir>
           <filter>@</filter>
         </build>
       </image>
     </images>
   </configuration>
  </plugin>
  [...]

  The assembly.xml file will generate the XWiki WAR:

  <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
   <id>xwiki</id>
   <dependencySets>
     <dependencySet>
       <includes>
         <include>org.xwiki.platform:xwiki-platform-distribution-war</include>
       </includes>
       <outputDirectory>.</outputDirectory>
       <outputFileNameMapping>xwiki.war</outputFileNameMapping>
       <useProjectArtifact>false</useProjectArtifact>
    </dependencySet>
  </dependencySets>
  <fileSets>
    <fileSet>
      <directory>${project.basedir}/src/main/docker</directory>
      <outputDirectory>.</outputDirectory>
      <includes>
        <include>**/*.sh</include>
      </includes>
      <fileMode>755</fileMode>
    </fileSet>
    <fileSet>
      <directory>${project.basedir}/src/main/docker</directory>
      <outputDirectory>.</outputDirectory>
      <excludes>
        <exclude>**/*.sh</exclude>
      </excludes>
    </fileSet>
  </fileSets>
  </assembly> 

  And all the Dockerfile and ancillary files required to generate the image are in src/main/docker/*.

• Then in the test modules, start the Docker containers from the generated Docker images. Check the full POM:
  [...]
  <plugin>
   <groupId>io.fabric8</groupId>
   <artifactId>docker-maven-plugin</artifactId>
   <executions>
     <execution>
       <id>start</id>
         <phase>pre-integration-test</phase>
         <goals>
           <goal>start</goal>
         </goals>
         <configuration>
           <imagePullPolicy>IfNotPresent</imagePullPolicy>
           <showLogs>true</showLogs>
           <images>
             <image>
               <alias>mysql-xwiki</alias>
               <name>mysql:5.7</name>
               <run>
                [...]
            <image>
               <alias>xwiki</alias>
               <name>xwiki:latest</name>
               <run>
                [...]
            <image>
               <name>xwiki-maven</name>
               <run>
                [...]
                 <volumes>
                   <bind>
                     <volume>${project.basedir}:/usr/src/mymaven</volume>
                     <volume>${user.home}/.m2:/root/.m2</volume>
                   </bind>
                 </volumes>
                 <workingDir>/usr/src/mymaven</workingDir>
                 <cmd>
                   <arg>mvn</arg>
                   <arg>verify</arg>
                   <arg>-Pdocker-maven</arg>
                 </cmd>
      [...]
     <execution>
       <id>stop</id>
       <phase>post-integration-test</phase>
       <goals>
         <goal>stop</goal>
       </goals>
     </execution>
   </executions>
  </plugin>
• Notice that last container we start, i.e. xwiki-maven is configured to map the current Maven source as a directory inside the Docker container and it starts Maven inside the container to run the functional tests using the docker-maven Maven profile.

Limitations:

• The environment setup is done from the build (Maven), which means that the developer needs to start it before executing the test from his IDE. This can cause frictions in the developer workflow.
• We found issues when running Docker inside Docker and Maven inside Maven, specifically when having Maven start the docker container containing the browsers, itself starting a Maven build which starts the browser and then the tests. This resulted in the Maven build slowing down and cringing to a halt. This was probably due to the fact that Docker will use up a lot of memory by default and we would need to control all processes (Maven, Surefire, Docker, etc) and control very precisely the memory they use. Java10 would help but we're not using it yet and we're currently stuck on Java8.

Experimentation 4: In Java Tests using Selenium Jupiter

The idea is to use Selenium Jupiter to automatically start/stop the various Browsers to be used by Selenium directly from the JUnit5 tests.

Note that XWiki has its own test framework on top of Selenium, with a class named TestUtil providing various APIs to help set up tests. Thus we need to make this class available to the test too, by injecting it as a test method parameter for example. Thus I developed a XWikiDockerExtension JUnit5 extension that initializes the XWiki testing framework and that does this injection.

Here's how a very simple test looks like:

Limitations:

• Works great for spawning Browser containers but doesn't support other types of containers such as DBs or Servlet Containers. Would need to implement the creation and start of them in a custom manner which is a lot of work.

Experimentation 5: in Java Tests using TestContainers

This idea builds on the Selenium Jupiter idea but using a different library, called TestContainers. It's the same idea but it's more generic since TestContainers allows creating all sorts of Docker containers (Selenium containers, DB containers, custom containers).

Here's how it works:

And here's an example of a Selenium test using it:

Some explanations:

• The @UITest annotation triggers the execution of the XWiki Docker Extension for JUnit5 (XWikiDockerExtension)
• In turn this extension will perform a variety of tasks:
  • Verify if an XWiki instance is already running locally. If not, it will generate a minimal XWiki WAR containing just what's needed to test the module the test is defined in. Then in turn, it will start a database and start a servlet container and deploy the XWiki WAR in it by using a Docker volume mapping
  • Initialize Selenium and start a Browser (the exact browser to start can be controlled in a variety of ways, with system properties and through parameters of the @UITest annotation).
  • It will also start a VNC docker container to record all the test execution, which is nice when one needs to debug a failing test and see what happened.

Current Status as of 2018-07-13:

• Browser containers are working and we can test in both Firefox and Chrome. Currently XWiki is started the old way, i.e. by using the XWiki Maven Package Plugin which generates a full XWiki distribution based on Jetty and HSQLDB.
• We have implemented the ability to fully generate an XWiki WAR directly from the tests (using the ShrinkWrapp library), which was the prerequisite for being able to deploy XWiki in a Servlet Container running in a Docker container and to start/stop it.
• Work in progress:
  • Support an existing running XWiki and in this case don't generate the WAR and don't start/stop the DB and Servlet Container Docker containers.
  • Implement the start/stop of the DB Container (MySQL and PostgreSQL to start with) from within the test using TestContainer's existing MySQLContainer and PostgresSQL containers.
  • Implement the start/stop of the Servlet Container (Tomcat to start with) from within the test using TestContainer's GenericContainer feature.

Note that most of the implementation is generic and can be easily reused and ported to software other than XWiki.

Limitations:

• Only supports 2 browsers FTM: Firefox and Chrome. More will come. However it's going to be very hard to support browsers requiring Windows (IE11, Edge) or Mac OSX (Safari). Preliminary work is in progress in TestContainers but it's unlikely to result in any usable solution anytime soon.
• Note that forced us to allow using Selenium 3.x while all our tests are currently on Selenium 2.x. Thus we implemented a solution to have the 2 versions run side by side and we modified our Page Objects to use reflection and call the right Selenium API depending on the version. Luckily there aren't too many places where the Selenium API has changed from 2.x to 3.x. Our goal is now to write new functional UI tests in Selenium 3 with the new TestContainer-based tedting framework and progressively migrate tests using Selenium 2.x to this new framework.
• The full execution of the tests take a bit longer than what we used to have with a single environment made of HSQLDB+Jetty. Measures will be taken when the full implementation is finished to evaluate the total time it takes.

Future ideas:

• Discuss with the CAMP developers to see how their mutation engine could be executed as a Java library so that it could be integrated in the XWiki testing framework. Namely to issues were open on the CAMP issue tracker to discuss this:
  • Be able to run configuration testing on developer machines and as part of tests
  • Be able to use the configuration mutation algorithm as a library

Conclusion

At this point in time I'm happy with our last experiment and implementation based on TestContainers. It allows to run environment tests directly from your IDE with no other prerequisite than having Docker installed on your machine. This means it also works from Maven or from any CI. We need to finish the implementation and this will give the XWiki project the ability to run tests on various combinations of configurations. 

Once this is done we should be able to tackle the next step which involves more exotic configurations such as running XWiki in a cluster, configuring a LibreOffice server to test importing office documents in the XWiki, and even configuring an external SOLR instance. However once the whole framework is in place, I don't expect this to cause any special problems.

Last, but not least, once we get this ability to execute various configurations, it'll be interesting to use a configuration mutation engine such as the one provided by CAMP in order to test various configurations in our CI. Since testing lots of them would be very costly in term of number of Agents required and CPU power, one idea is to have a job that executes, say, once per day with a random configuration selected and that reports how the tests perform in it.

Another Devoxx France and another huge success

This time I had 2 talks.

New Generation of Tests

Title: Advanced testing in action on a Java project
Subtitle: Demonstrating testing practices using Java, Maven, Docker and Jenkins.
Abstract: This talk will demonstrate advanced testing practices used by the XWiki open source project, and using Java, Maven, Docker and Jenkins:

• Testing for backward compatibility with Revapi and an associated strategy
• Testing for coverage with Jacoco and defining a viable strategy for slowing improving the situation
• Testing the quality of your tests with Descartes Mutation testing
• Automatically enriching your test suite with DSpot
• Testing various configurations with Docker containers and Jenkins

This is a talk I also did at FOSDEM 2018 (In English). The part about Mutation testing and configuration testing concerns results from the STAMP research project I'm participating to.

I got lucky to get a lot of votes for the talk and thus I was in the big amphitheater (capacity of 800-850 people) and it was almost full!

LCC Podcast Recording

As usual we did a live recording of the LCC Podcast.

Misc

What I enjoy the most of technical conferences is the ability my friends and get some news from them about the fun stuff they're doing.

I also sent to the speaker dinner organized by Devoxx and we got a nice sight-seeing bus tour of Paris.

I also got some ideas to try out on the XWiki project:

• Generating flame graphs for java code> with async-profiler. Works for CPU, memory and works in prod (uses less than 2% CPU).
• Very nice presentation about implementing Chaos Engineering. Could be a nice idea to implement in XWiki SAS.

I had the honor of being invited to a seminar on "Automatic Quality Assurance and Release" at Dagstuhl by Benoit Baudry (we collaborate together on the STAMP research project). Our seminar was organized as un unconference and one session I proposed and led was the "Onboarding" one described below. The following persons participated to the discussion: V. Massol, D. Gagliardi, B. Danglot, H. Wright, B. Baudry.

Onboarding Discussions

When you're developing a project (be it some internal project or some open source project) one key element is how easy it is to onboard new users to your project. For open source projects this is essential to attract more contributors and have a lively community. For internal projects, it's useful to be able to have new employees or newcomers in general be able to get up to speed rapidly on your project.

This brainstorming session was about ideas of tools and practices to use to ease onboarding.

Here's the list of ideas we had (in no specific order):

• 1 - Tag issues in your issue tracker as onboarding issues to make it easy for newcomer to get started on something easy and be in success quickly. This also validates that they're able to use your software.
• 2 - Have a complete package of your software that can be installed and used as easily as possible. It should just work out of the box without having to perform any configuration or additional steps. A good strategy for applications is to provide a Docker image (or a Virtual Machine) with everything setup.
• 3 - Similarly, provide a packaged development environment. For example you can provide a VM with some preinstalled and configured IDE (with plugins installed and configured using the project's rules). One downside of such an approach is the time it takes to download the VM (which could several GB in size). 
• 4 - A similar and possibly better approach would be to use an online IDE (e.g. Eclipse Che) to provide a complete prebuilt dev environment that wouldn't even require any downloading. This provides the fastest dev experience you can get. The downside is that if you need to onboard a potentially large number of developers, you'll need some important infra space/CPU on your server(s) hosting the online IDE, for hosting all the dev workspaces. This makes this option difficult to implement for open source projects for example. But it's viable and interesting in a company environment.
• 5 - Obviously having good documentation is a given. However too many projects still don't provide this or only provide good user documentation but not good developer documentation with project practices not being well documented or only a small portion being documented. Specific ideas:
  • Document the code structure
  • Document the practices for development
  • Develop a tool that supports newcomers by letting them know when they follow / don't follow the rules
  • Good documentation shall explicit assumptions (e.g. when you read this piece of documentation, I assume that you know X and Y)
  • Have a good system to contribute to the documentation of the project (e.g. a wiki)
  • Different documentation for users and for developers
• 6 - Have homogeneous practices and tools inside a project. This is especially true in a company environment where you may have various projects, each using its own tools and practices, making it harder to move between projects.
• 7 - Use standard tools that are well known (e.g. Maven or Docker). That increases the likelihood that a newcomer would already know the tool and be able to developer for your project.
• 8 - It's good to have documentation about best practices but it's even better if the important "must" rules be enforced automatically by a checking tool (can be part of the build for example, or part of your IDE setup). For example instead of saying "this @Unstable annotation should be removed after one development cycle", you could write a Maven Enforcer rule (or a Checkstyle rule, or a Spoon rule) to break the build if it happens, with a message explaining the reason and what is to be done. Usually humans may prefer to have a tool telling them that than a way telling them that they haven't been following the best practices documented at such location...
• 9 - Have a bot to help you discover documentation pages about a topic. For example by having a chat bot located in the project's chat, that when asked about will give you the link to it.
• 10 - Projects must have a medium to ask questions and get fast answers (such as a chat tool). Forum or mailing lists are good but less interesting when onboarding when the newcomer has a lot of questions in the initial phase and requires a conversation.
• 11 - Have an answer strategy so that when someone asks a question, the doc is updated (new FAQ entry for example) so that the next person who comes can find the answer or be given the link to the doc.
• 12 - Mentoring (human aspect of onboarding): have a dedicated colleague to whom you're not afraid to ask questions and who is a referent to you.
• 13 - Supporting a variety of platforms for your software will make it simpler for newcomers to contribute to your project.
• 14 - Split your projects into smaller parts. While it's hard and a daunting experience to contribute to the core code of a project, if this project has a core as small as possible and the rest is made of plugins/extensions then it becomes simpler to start contributing to those extensions first.
• 15 - Have some interactive tutorial to learn about your software or about its development. A good example of nice tutorial can be found at www.katacoda.com (for example for Docker, https://www.katacoda.com/courses/docker).
• 16 - Human aspect: have an environment that makes you feel welcome. Work and discuss how to best answer Pull Requests, how to communicate when someone joins the project, etc. Think of the newcomer as you would a child: somebody who will occasionally stumble and need encouragment. Try to have as much empathy as possible.
• 17 - Make sure that people asking questions always get an answer quickly, perhaps by establishing a role on the team to ensure answers are provided.
• 18 - Last but not least, an interesting thought experiment to verify that you have some good onboarding processes: imagine that 1000 developers join your project / company on the same day. How do you handle this?

Onboarding on XWiki

I was also curious to see how those ideas apply to the XWiki open source project and what part we implement.

Ideas	Implemented on XWiki?
1 - Tag simple issues	Onboarding issues
2 - Complete install package	Debian apt-get, Docker images.
3 - Dev packaged environment	We have a Developer VM
4 - Online IDE onboarding	Hard to do provide for an OSS project in term of infra resources but would love to provide this
5 - Good documentation	User guide, Admin guide, Dev guide + there's a wiki dedicated to development practices and tools.
6 - Have homogeneous practices and tools inside a project	See http://dev.xwiki.org
7 - Use standard tools that are well known	Maven, Jenkins, Java, JUnit, Mockito, Selenium
8 - Automatically enforced important rules	See Automatic checks in build
9 - Have a bot to help you discover documentation pages about a topic	IRC bot (used here - been broken since 2017-05-09).
10 - Projects must have a medium to ask questions and get fast answers	XWiki Chat
11 - Have an answer strategy so that when someone asks a question	XWiki answer strategy and FAQ
12 - Mentoring (human aspect of onboarding)	Done to some extent by employees of XWiki SAS who are committers on the open source project but not a generic open source project practice.
13 - Supporting a variety of platforms for your software	Windows, Linux, Mac, multiple DBs, multiple browsers, multiple Servlet containers.
14 - Split your projects into smaller parts	Core getting smaller and more and more Extensions.
15 - Have some interactive tutorial to learn about your software	Would be nice to have
16 - Human aspect: have an environment that makes you feel welcome.	This is subjective. Sometimes we may be a bit abrupt when answering (especially me! Sorry guys if I've been abrupt, it's more a consequence of doing too many things. I need to improve. I think we're globally a welcoming community, WDYT?
17 - Make sure that people asking questions always get an answer quickly	I think we're very good at answering fast. See the Forum for example. We also answer fast on Matrix/IRC (we try).
18 - 1000 devs joining at once experiment	Actually we participated to Google CodeIn 2017 and this is exactly what we experienced: 756 students interacting with us.

So globally I'd say XWiki is pretty good at onboarding. I'd love to hear about things that we could improve on for onboarding. Any ideas?

If you own a project, we would be interested to hear about your ideas and how you perform onboarding. You could also use the list above as a way to measure your level of onboarding for your project and find out how you could improve it further.

Once more I was happy to go to FOSDEM. This year XWiki SAS, the company I work for, had 12 employees going there and we had about 8 talks accepted + we had a stand for the XWiki open source project that we shared with our friends @ Tiki and Foswiki.

Here were the highlights for me:

• I talked about what's next on Java testing and covered test coverage, backward compatibility enforcement, mutation testing and environment testing. My experience on the last two types of tests are directly issued from my participation the STAMP research project where we develop and use tools to amplify existing tests.

  Slides: 

  Advanced Java Testing from Vincent Massol

  Video:

• I did another talk about "Addressing the Long Tail of (web) applications", explaining how an advanced structured wiki such as XWiki can be used to quickly create ad-hoc application in wiki pages.

  Slides:

  Addressing the long tail of (web) applications from Vincent Massol

  Video:

• Since we had a stand shared between 3 wiki communities (Tiki, Foswiki and XWiki), I was also interested in comparing our features, and how our communities work.
  • I met the nice folks of Tiki at their TikiHouse and had long discussions about how similar and differently we do things. Maybe the topic for a future blog post?
  • Then I had Michael Daum do a demo to me of the nice features of Foswiki. I was quite impressed and found a lot of similarities in features. 
  • Funnily our 3 wiki solutions are written in 3 different technologies: Tiki in PHP, Foswiki in Perl and XWiki in Java. Nice diversity!
• I met a lot of people of course (Fosdem is really THE place to be to meet people from the FOSS communities) but I'd like to thank especially Damien Duportal who took the time to sit with me and go over several questions I had about Jenkins pipelines and Docker. I'll most likely blog about some of those solutions in the near future.

All in all, an excellent FOSDEM again, with lots of talks and waffles

My company (XWiki SAS) had a booth at the Paris Open Source Summit 2017 (POSS) and we also organized a track about "One job, one solution!", promoting open source solutions.

I was asked to talk about using XWiki as an alternative to Confluence or Sharepoint.

Here are the slides of the talk:

There were about 30 persons in the room. I focused on showing what I believe are the major differences, especially with Confluence that I know better than SharePoint.

If you're interested by more details you can find a comparison between XWiki and Confluence on xwiki.org.

On the XWiki project, we use Clover to compute our global test coverage. We do this over several Git repositories and include functional tests (and more generally the coverage brought by some modules into other modules).

Now I wanted to see the difference between 2 reports that were generated:

• Report from 2016-12-20

  

• Report from 2017-11-09

  

I was surprised to see a drop in the global TPC, from 73.2% down to 71.3%. So I took the time to understand the issue.

It appears that Clover classifies your code classes as Application Code and Test Code (I have no idea what strategy it uses to differentiate them) and even though we've used the same version of Clover (4.1.2) for both reports, the test classes were not categorized similarly. It also seems that the TPC value given in the HTML report is from Application Code.

Luckily we asked the Clover Maven plugin to generate not only HTML reports but also XML reports. Thus I was able to write the following Groovy script that I executed in a wiki page in XWiki. I aggregated Application Code and Test code together in order to be able to compare the reports and the global TPC value.

And the result was quite different from what the HTML report was giving us!

We went from 74.07% in 2016-12-20 to 76.28% in 2017-11-09 (so quite different from the 73.2% to 71.3% figure given by the HTML report). Much nicer!

Note that one reason I wanted to compare the TPC values was to see if our strategy of failing the build if a module's TPC is below the current threshold was working or not (I had tried to assess it before but it wasn't very conclusive).

Now I know that we won 1.9% of TPC in a bit less than a year and that looks good

EDIT: I'm aware of the Historical feature of Clover but:

• We haven't set it up so it's too late to compare old reports
• I don't think it would help with the issue we faced with test code being counted as Application Code, and that being done differently depending on the generated reports.

Flaky tests are a plague because they lower the credibility in your CI strategy, by sending false positive notification emails.

In a previous blog post, I detailed a solution we use on the XWiki project to handle false positives caused by the environment on which the CI build is running. However this solution wasn't handling flaky tests. This blog post is about fixing this!

So the strategy I'm proposing for Flaky tests is the following:

• When a Flaky test is discovered, create a JIRA issue to remember to work on it and fix it (we currently have the following open issues related to Flaky tests)
• The JIRA issue is marked as containing a flaky test by filling a custom field called "Flickering Test", using the following format: <package name of test class>.<test class name>#<test method name>. There can be several entries separated by commas.

  Example:

  

• In our Pipeline script, after the tests have executed, review the failing ones and check if they are in the list of known flaky tests in JIRA. If so, indicate it in the Jenkins test report. If all failing tests are flickers, don't send a notification email.

  Indication in the job history:

  

  Indication on the job result page:

  

  Information on the test page itself:

  

Note that there's an alternate solution that can also work:

• When a Flaky test is discovered, create a JIRA issue to remember to work on it and fix it
• Add an @Ignore annotation in the test with a detail pointing to the JIRA issue (something like @Ignore("WebDriver doesn't support uploading multiple files in one input, see http://code.google.com/p/selenium/issues/detail?id=2239")
  ). This will prevent the build from executing this flaky test.

This last solution is certainly low-tech compared to the first one. I prefer the first one though for the following reasons:

• It allows flaky tests to continue executing on the CI and thus serve as a constant reminder that something needs to be fixed. Adding the @Ignore annotation feels like putting the dust under the carpet and there's little chance you're going to come back to it in the future...
• Since our script acts as postbuild script on the CI agent, there's the possibility to add some logic to auto-discover flaky tests that have not yet been marked as flaky.

Also note that there's a Jenkins plugin for Flaky test but I don't like the strategy involved which is to re-run failing tests a number of times to see if they pass. In theory it can work. In practice this means CI jobs that will take a lot longer to execute, making it impractical for functional UI tests (which is where we have flaky tests in XWiki). In addition, flakiness sometimes only happens when the full test suite is executed (i.e. it depends on what executes before) and sometimes require a large number of runs before passing.

So without further ado, here's the Jenkins Pipeline script to implement the strategy we defined above (you can check the full pipeline script):

Hope you like it! Let me know in comments how you're handling Flaky tests in your project so that we can compare/discuss.

I had the chance to participate to Softshake (2017 edition)  for the first time. It's a small but very nice conference held in Geneva, Switzerland.

From what I gathered, this year there were less attendees than in the previous years (About 150 vs 300 before). However, the organization was very nice:

• Located inside the Hepia school with plenty of rooms available
• 6 tracks in parallel, which is incredible for a small conference
• Breakfast, lunch and snacks organized with good food
• Speaker dinner with Fondue and all

I got to present 2 talks:

• XWiki: The Web's Swiss Army Knife. This is my usual "XWiki: A web development platform" talk that I've given a few times already but with a more Swiss-related title
  XWiki: The web's Swiss Army Knife from Vincent Massol
• Creating your own project's Quality Dashboard. This one was brand new and was a big love demo of how to use XWiki to create a custom Quality Dashboard by aggregating metrics from other sites (Jenkins, SonarQube, JIRA and GitHub), saving them locally to draw history graphs and sending emails when combined metric thresholds are crossed. A lot more people attended this one and I like this new angle of defining a real-life use case and using XWiki just as a tool to achieve it. I'll continue exploring this new way of presenting XWiki since people liked it a lot more and it feels more natural.
  Creating your own project's Quality Dashboard from Vincent Massol

I was also very happy to see my friend and ex-OCTO Technology colleague Philippe Kernevez, and to meet new OCTO consultants. Reminded me of the good times at OCTO