Environment Testing Experimentations

Version 9.4 by Vincent Massol on 2018/07/13 16:24

Jun 25 2018

This blog post is not published yet.

I've been trying to define the best solution for doing functional testing in multiple environments (different DBs, Servlet containers, browsers) for XWiki. Right now on XWiki we test automatically on a single environment (Firefox, Jetty, HSQLDB) and we do the other environment tests manually.

So I've been going through different experimentations, finding out issues and limitations with them and progressing towards the perfect solution for us. Here are the various experiments we conducted. Note that this work is being done as part of the STAMP Research Project.

Here are the use cases that we want to support ideally:

  • UC1: Fast to start XWiki in a given environment/configuration
  • UC2: Solution must be usable both for running the functional tests and for distributing XWiki
  • UC3: Be able to execute tests both on CI and locally on developer's machines
  • UC4: Be able to debug functional tests easily locally
  • UC5: Support the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
  • UC6: Choose a solution that's as fast as possible for functional test executions

Experiments:

Experimentation 1: CAMP from STAMP

CAMP is a tool developed by some partners on the STAMP research project and it acts as a remote testing service. You give it some Dockerfile and it'll create the image, start a container, execute some commands (that you also provide to it and that are used to validate that the instance is working fine) and then stop the container. In addition it performs configuration mutation on the provided Dockerfile. This means it'll make some variations to this file, regenerate the image and re-run the container.

Here's how CAMP works (more details including how to use it on XWiki can be found on the CAMP home page):

https://github.com/STAMP-project/camp/raw/master/src/doc/camp_idea.png
Image from the CAMP web site

Limitations for the XWiki use case needs:

  • Relies on a service. This service can be installed on a server on premises too but that means more infrastructure to maintain for the CI subsystem. Would be better if integrated directly in Jenkins for example.
  • Cannot easily run on the developer machine which is important so that devs can test what they develop on various environments and so that they can debug reported issues on various environments. This fails at least UC3 and UC4.
  • Even though mutation of configuration is an interesting concept, it's not a use case for XWiki which has several well-defined configurations that are supported. It's true that it could be interesting to have fixed topologies and only vary versions of servers (DB version, Servlet Container version and Java version - We don't need to vary Browser versions since we support only the latest version) but we think the added value vs the infrastructure cost might not be that interesting for us. However, it could still be interesting for example by randomizing the mutated configuration and only running tests on one such configuration per day to reduce the need of having too many agents and leaving them free for the other jobs.

Experimentation 2: Docker on CI

I blogged in the past about this strategy.

architecture.png

The main idea for this experiment was to use a Jenkins Pipeline with the Jenkins Plugin for Docker, allowing to write pipelines like this:

pipeline {
  agent {
    docker {
      image 'xwiki-maven-firefox'
      args '-v $HOME/.m2:/root/.m2'
    }
  }
  stages {
    stage('Test') {
      steps {
        docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->                    
          docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
            [...]
            wrap([$class: 'Xvnc']) {
              withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
                [...]
                sh "mvn ..."
              }
            }
          }
        }
      }
    }
  }
}

Limitations:

  • Similar to experimentation 1 with CAMP, this relies on the CI to execute the tests and doesn't allow developers to test and reproduce issues on their local machines. This fails at least UC3 and UC4.

Experimentation 3: Maven build with Fabric8

The next idea was to implement the logic in the Maven build so that it could be executed on developer machines. I found the very nice Fabric8 Maven plugin and came up with the following architecture that I tried to implement:

fabric8.png

The main ideas:

  • Generate the various Docker images we need (the Servlet Container one and the one containing Maven and the Browsers) using Fabric8 in a Maven module. For example to generate the Docker image containing Tomcat and XWiki:

    pom.xml file:

    [...]
    <plugin>
     <groupId>io.fabric8</groupId>
     <artifactId>docker-maven-plugin</artifactId>
     <configuration>
       <imagePullPolicy>IfNotPresent</imagePullPolicy>
       <images>
         <image>
           <alias>xwiki</alias>
           <name>xwiki:latest</name>
           <build>
             <tags>
               <tag>${project.version}-mysql-tomcat</tag>
               <tag>${project.version}-mysql</tag>
               <tag>${project.version}</tag>
             </tags>
             <assembly>
               <name>xwiki</name>
               <targetDir>/maven</targetDir>
               <mode>dir</mode>
               <descriptor>assembly.xml</descriptor>
             </assembly>
             <dockerFileDir>.</dockerFileDir>
             <filter>@</filter>
           </build>
         </image>
       </images>
     </configuration>
    </plugin>
    [...]

    The assembly.xml file will generate the XWiki WAR:

    <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
     <id>xwiki</id>
     <dependencySets>
       <dependencySet>
         <includes>
           <include>org.xwiki.platform:xwiki-platform-distribution-war</include>
         </includes>
         <outputDirectory>.</outputDirectory>
         <outputFileNameMapping>xwiki.war</outputFileNameMapping>
         <useProjectArtifact>false</useProjectArtifact>
      </dependencySet>
    </dependencySets>
    <fileSets>
      <fileSet>
        <directory>${project.basedir}/src/main/docker</directory>
        <outputDirectory>.</outputDirectory>
        <includes>
          <include>**/*.sh</include>
        </includes>
        <fileMode>755</fileMode>
      </fileSet>
      <fileSet>
        <directory>${project.basedir}/src/main/docker</directory>
        <outputDirectory>.</outputDirectory>
        <excludes>
          <exclude>**/*.sh</exclude>
        </excludes>
      </fileSet>
    </fileSets>
    </assembly> 

    And all the Dockerfile and ancillary files required to generate the image are in src/main/docker/*.

  • Then in the test modules, start the Docker containers from the generated Docker images. Check the full POM:
    [...]
    <plugin>
     <groupId>io.fabric8</groupId>
     <artifactId>docker-maven-plugin</artifactId>
     <executions>
       <execution>
         <id>start</id>
           <phase>pre-integration-test</phase>
           <goals>
             <goal>start</goal>
           </goals>
           <configuration>
             <imagePullPolicy>IfNotPresent</imagePullPolicy>
             <showLogs>true</showLogs>
             <images>
               <image>
                 <alias>mysql-xwiki</alias>
                 <name>mysql:5.7</name>
                 <run>
                  [...]
              <image>
                 <alias>xwiki</alias>
                 <name>xwiki:latest</name>
                 <run>
                  [...]
              <image>
                 <name>xwiki-maven</name>
                 <run>
                  [...]
                   <volumes>
                     <bind>
                       <volume>${project.basedir}:/usr/src/mymaven</volume>
                       <volume>${user.home}/.m2:/root/.m2</volume>
                     </bind>
                   </volumes>
                   <workingDir>/usr/src/mymaven</workingDir>
                   <cmd>
                     <arg>mvn</arg>
                     <arg>verify</arg>
                     <arg>-Pdocker-maven</arg>
                   </cmd>
        [...]
       <execution>
         <id>stop</id>
         <phase>post-integration-test</phase>
         <goals>
           <goal>stop</goal>
         </goals>
       </execution>
     </executions>
    </plugin>
  • Notice that last container we start, i.e. xwiki-maven is configured to map the current Maven source as a directory inside the Docker container and it starts Maven inside the container to run the functional tests using the docker-maven Maven profile.

Limitations:

  • The environment setup is done from the build (Maven), which means that the developer needs to start it before executing the test from his IDE. This can cause frictions in the developer workflow.
  • We found issues when running Docker inside Docker and Maven inside Maven, specifically when having Maven start the docker container containing the browsers, itself starting a Maven build which starts the browser and then the tests. This resulted in the Maven build slowing down and cringing to a halt. This was probably due to the fact that Docker will use up a lot of memory by default and we would need to control all processes (Maven, Surefire, Docker, etc) and control very precisely the memory they use. Java10 would help but we're not using it yet and we're currently stuck on Java8.

Experimentation 4: In Java Tests using Selenium Jupiter

 

The idea is to use Selenium Jupiter to automatically start/stop the various Browsers to be used by Selenium directly from the JUnit5 tests.

Note that XWiki has test framework on top of Selenium, with a class named TestUtil providing various APIs to help set up tests. Thus we need to make this class available to the test too by injecting it as test method parameter for example. Thus I developed a XWikiDockerExtension JUnit5 extension that initializes the XWiki testing framework and that does this injection.

Here's how a very simple test look like:

@ExtendWith(XWikiDockerExtension.class)
public class SeleniumTest
{
   @BeforeAll
   static void setup()
   {
       // TODO: move to the pom
        SeleniumJupiter.config().setVnc(true);
        SeleniumJupiter.config().setRecording(true);
        SeleniumJupiter.config().useSurefireOutputFolder();
        SeleniumJupiter.config().takeScreenshotAsPng();
        SeleniumJupiter.config().setDefaultBrowser("firefox-in-docker");
   }

   @Test
   public void test(WebDriver driver, TestUtils setup)
   {
        driver.get("http://xwiki.org");
        assertThat(driver.getTitle(), containsString("XWiki - The Advanced Open Source Enterprise and Application Wiki"));
        driver.findElement(By.linkText("XWiki's concept")).click();
   }
}

Limitations:

  • Works great for spawning Browser containers but doesn't support other types of containers such as DBs or Servlet Containers. Would need to implement the creation and start of them in a custom manner which is a lot of work.

Experimentation 5: in Java Tests using TestContainers

This idea builds on the Selenium Jupiter idea but using a different library, called TestContainers. It's the same idea but it's more generic since TestContainers allows creating all sorts of Docker containers (Selenium containers, DB containers, custom containers).

Created by Vincent Massol  · Permalink · Comments (0)

Created by Vincent Massol on 2018/06/25 10:07