Environment Testing Experimentations

Version 8.1 by Vincent Massol on 2018/07/13 16:02

Jun 25 2018

This blog post is not published yet.

I've been trying to define the best solution for doing functional testing in multiple environments (different DBs, Servlet containers, browsers) for XWiki. Right now on XWiki we test automatically on a single environment (Firefox, Jetty, HSQLDB) and we do the other environment tests manually.

So I've been going through different experimentations, finding out issues and limitations with them and progressing towards the perfect solution for us. Here are the various experiments we conducted. Note that this work is being done as part of the STAMP Research Project.

Here are the use cases that we want to support ideally:

  • UC1: Fast to start XWiki in a given environment/configuration
  • UC2: Solution must be usable both for running the functional tests and for distributing XWiki
  • UC3: Be able to execute tests both on CI and locally on developer's machines
  • UC4: Be able to debug functional tests easily locally
  • UC5: Support the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
  • UC6: Choose a solution that's as fast as possible for functional test executions

Experimentation 1: CAMP from STAMP

CAMP is a tool developed by some partners on the STAMP research project and it acts as a remote testing service. You give it some Dockerfile and it'll create the image, start a container, execute some commands (that you also provide to it and that are used to validate that the instance is working fine) and then stop the container. In addition it performs configuration mutation on the provided Dockerfile. This means it'll make some variations to this file, regenerate the image and re-run the container.

Here's how CAMP works (more details including how to use it on XWiki can be found on the CAMP home page):

https://github.com/STAMP-project/camp/raw/master/src/doc/camp_idea.png
Image from the CAMP web site

Limitations for the XWiki use case needs:

  • Relies on a service. This service can be installed on a server on premises too but that means more infrastructure to maintain for the CI subsystem. Would be better if integrated directly in Jenkins for example.
  • Cannot easily run on the developer machine which is important so that devs can test what they develop on various environments and so that they can debug reported issues on various environments. This fails at least UC3 and UC4.
  • Even though mutation of configuration is an interesting concept, it's not a use case for XWiki which has several well-defined configurations that are supported. It's true that it could be interesting to have fixed topologies and only vary versions of servers (DB version, Servlet Container version and Java version - We don't need to vary Browser versions since we support only the latest version) but we think the added value vs the infrastructure cost might not be that interesting for us. However, it could still be interesting for example by randomizing the mutated configuration and only running tests on one such configuration per day to reduce the need of having too many agents and leaving them free for the other jobs.

Experimentation 2: Docker on CI

I blogged in the past about this strategy.

architecture.png

The main idea for this experiment was to use a Jenkins Pipeline with the Jenkins Plugin for Docker, allowing to write pipelines like this:

pipeline {
  agent {
    docker {
      image 'xwiki-maven-firefox'
      args '-v $HOME/.m2:/root/.m2'
    }
  }
  stages {
    stage('Test') {
      steps {
        docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->                    
          docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
            [...]
            wrap([$class: 'Xvnc']) {
              withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
                [...]
                sh "mvn ..."
              }
            }
          }
        }
      }
    }
  }
}

Limitations:

  • Similar to experimentation 1 with CAMP, this relies on the CI to execute the tests and doesn't allow developers to test and reproduce issues on their local machines. This fails at least UC3 and UC4.

Experimentation 3: Maven build with Fabric8

The next idea was to implement the logic in the Maven build so that it could be executed on developer machines. I found the very nice Fabric8 Maven plugin and came up with the following architecture that I tried to implement:

fabric8.png

The main ideas:

  • Generate the various Docker images we need (the Servlet Container one and the one containing Maven and the Browsers) using Fabric8 in a Maven module. For example to generate the Docker image containing Tomcat and XWiki:

    pom.xml file:

    [...]
    <plugin>
     <groupId>io.fabric8</groupId>
     <artifactId>docker-maven-plugin</artifactId>
     <configuration>
       <imagePullPolicy>IfNotPresent</imagePullPolicy>
       <images>
         <image>
           <alias>xwiki</alias>
           <name>xwiki:latest</name>
           <build>
             <tags>
               <tag>${project.version}-mysql-tomcat</tag>
               <tag>${project.version}-mysql</tag>
               <tag>${project.version}</tag>
             </tags>
             <assembly>
               <name>xwiki</name>
               <targetDir>/maven</targetDir>
               <mode>dir</mode>
               <descriptor>assembly.xml</descriptor>
             </assembly>
             <dockerFileDir>.</dockerFileDir>
             <filter>@</filter>
           </build>
         </image>
       </images>
     </configuration>
    </plugin>
    [...]

    The assembly.xml file will generate the XWiki WAR:

    <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
     <id>xwiki</id>
     <dependencySets>
       <dependencySet>
         <includes>
           <include>org.xwiki.platform:xwiki-platform-distribution-war</include>
         </includes>
         <outputDirectory>.</outputDirectory>
         <outputFileNameMapping>xwiki.war</outputFileNameMapping>
         <useProjectArtifact>false</useProjectArtifact>
      </dependencySet>
    </dependencySets>
    <fileSets>
      <fileSet>
        <directory>${project.basedir}/src/main/docker</directory>
        <outputDirectory>.</outputDirectory>
        <includes>
          <include>**/*.sh</include>
        </includes>
        <fileMode>755</fileMode>
      </fileSet>
      <fileSet>
        <directory>${project.basedir}/src/main/docker</directory>
        <outputDirectory>.</outputDirectory>
        <excludes>
          <exclude>**/*.sh</exclude>
        </excludes>
      </fileSet>
    </fileSets>
    </assembly> 

    And all the Dockerfile and ancillary files required to generate the image are in src/main/docker/*.

  • Then in the test modules, start the Docker containers from the generated Docker images. Check the full POM:
    [...]
    <plugin>
     <groupId>io.fabric8</groupId>
     <artifactId>docker-maven-plugin</artifactId>
     <executions>
       <execution>
         <id>start</id>
           <phase>pre-integration-test</phase>
           <goals>
             <goal>start</goal>
           </goals>
           <configuration>
             <imagePullPolicy>IfNotPresent</imagePullPolicy>
             <showLogs>true</showLogs>
             <images>
               <image>
                 <alias>mysql-xwiki</alias>
                 <name>mysql:5.7</name>
                 <run>
                  [...]
              <image>
                 <alias>xwiki</alias>
                 <name>xwiki:latest</name>
                 <run>
                  [...]
              <image>
                 <name>xwiki-maven</name>
                 <run>
                  [...]
                   <volumes>
                     <bind>
                       <volume>${project.basedir}:/usr/src/mymaven</volume>
                       <volume>${user.home}/.m2:/root/.m2</volume>
                     </bind>
                   </volumes>
                   <workingDir>/usr/src/mymaven</workingDir>
                   <cmd>
                     <arg>mvn</arg>
                     <arg>verify</arg>
                     <arg>-Pdocker-maven</arg>
                   </cmd>
        [...]
       <execution>
         <id>stop</id>
         <phase>post-integration-test</phase>
         <goals>
           <goal>stop</goal>
         </goals>
       </execution>
     </executions>
    </plugin>
  • Notice that last container we start, i.e. xwiki-maven is configured to map the current Maven source as a directory inside the Docker container and it starts Maven inside the container to run the functional tests using the docker-maven Maven profile.

Limitations:

  • The environment setup is done from the build (Maven), which means that the developer needs to start it before executing the test from his IDE. This can cause frictions in the developer workflow.
  • We found issues when running Docker inside Docker and Maven inside Maven, specifically when having Maven start the docker container containing the browsers, itself starting a Maven build which starts the browser and then the tests. This resulted in the Maven build slowing down and cringing to a halt. This was probably due to the fact that Docker will use up a lot of memory by default and we would need to control all processes (Maven, Surefire, Docker, etc) and control very precisely the memory they use. Java10 would help but we're not using it yet and we're currently stuck on Java8.

Experimentation 4: In Java Tests using Jupiter Selenium

 

TODO

Experimentation 5: in Java Tests using TestContainers

TODO

Created by Vincent Massol  · Permalink · Comments (0)

Created by Vincent Massol on 2018/06/25 10:07