I've been trying to define the best solution for doing functional testing in multiple environments (different DBs, Servlet containers, browsers) for XWiki. Right now on XWiki we test automatically on a single environment (Firefox, Jetty, HSQLDB) and we do the other environment tests manually.
So I've been going through different experimentations, finding out issues and limitations with them and progressing towards the perfect solution for us. Here are the various experiments we conducted. Note that this work is being done as part of the STAMP Research Project.
Here are the use cases that we want to support ideally:
- UC1: Fast to start XWiki in a given environment/configuration
- UC2: Solution must be usable both for running the functional tests and for distributing XWiki
- UC3: Be able to execute tests both on CI and locally on developer's machines
- UC4: Be able to debug functional tests easily locally
- UC5: Support the following configuration options (i.e we can test with variations of those and different versions): OS, Servlet container, Database, Clustering, Office Server, external SOLR, Browser
- UC6: Choose a solution that's as fast as possible for functional test executions
Experiments:
- Experimentation 1: CAMP from STAMP
- Experimentation 2: Docker on CI
- Experimentation 3: Maven build with Fabric8
- Experimentation 4: In Java Tests using Selenium Jupiter
- Experimentation 5: in Java Tests using TestContainers
- Conclusion
Experimentation 1: CAMP from STAMP
CAMP is a tool developed by some partners on the STAMP research project and it acts as a remote testing service. You give it some Dockerfile and it'll create the image, start a container, execute some commands (that you also provide to it and that are used to validate that the instance is working fine) and then stop the container. In addition it performs configuration mutation on the provided Dockerfile. This means it'll make some variations to this file, regenerate the image and re-run the container.
Here's how CAMP works (more details including how to use it on XWiki can be found on the CAMP home page):
Image from the CAMP web site
Limitations for the XWiki use case needs:
- Relies on a service. This service can be installed on a server on premises too but that means more infrastructure to maintain for the CI subsystem. Would be better if integrated directly in Jenkins for example.
- Cannot easily run on the developer machine which is important so that devs can test what they develop on various environments and so that they can debug reported issues on various environments. This fails at least UC3 and UC4.
- Even though mutation of configuration is an interesting concept, it's not a use case for XWiki which has several well-defined configurations that are supported. It's true that it could be interesting to have fixed topologies and only vary versions of servers (DB version, Servlet Container version and Java version - We don't need to vary Browser versions since we support only the latest version) but we think the added value vs the infrastructure cost might not be that interesting for us. However, it could still be interesting for example by randomizing the mutated configuration and only running tests on one such configuration per day to reduce the need of having too many agents and leaving them free for the other jobs.
Experimentation 2: Docker on CI
I blogged in the past about this strategy.
The main idea for this experiment was to use a Jenkins Pipeline with the Jenkins Plugin for Docker, allowing to write pipelines like this:
agent {
docker {
image 'xwiki-maven-firefox'
args '-v $HOME/.m2:/root/.m2'
}
}
stages {
stage('Test') {
steps {
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->
docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
[...]
wrap([$class: 'Xvnc']) {
withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
[...]
sh "mvn ..."
}
}
}
}
}
}
}
}
Limitations:
- Similar to experimentation 1 with CAMP, this relies on the CI to execute the tests and doesn't allow developers to test and reproduce issues on their local machines. This fails at least UC3 and UC4.
Experimentation 3: Maven build with Fabric8
The next idea was to implement the logic in the Maven build so that it could be executed on developer machines. I found the very nice Fabric8 Maven plugin and came up with the following architecture that I tried to implement:
The main ideas:
- Generate the various Docker images we need (the Servlet Container one and the one containing Maven and the Browsers) using Fabric8 in a Maven module. For example to generate the Docker image containing Tomcat and XWiki:
pom.xml file:
[...]
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<configuration>
<imagePullPolicy>IfNotPresent</imagePullPolicy>
<images>
<image>
<alias>xwiki</alias>
<name>xwiki:latest</name>
<build>
<tags>
<tag>${project.version}-mysql-tomcat</tag>
<tag>${project.version}-mysql</tag>
<tag>${project.version}</tag>
</tags>
<assembly>
<name>xwiki</name>
<targetDir>/maven</targetDir>
<mode>dir</mode>
<descriptor>assembly.xml</descriptor>
</assembly>
<dockerFileDir>.</dockerFileDir>
<filter>@</filter>
</build>
</image>
</images>
</configuration>
</plugin>
[...]The assembly.xml file will generate the XWiki WAR:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
<id>xwiki</id>
<dependencySets>
<dependencySet>
<includes>
<include>org.xwiki.platform:xwiki-platform-distribution-war</include>
</includes>
<outputDirectory>.</outputDirectory>
<outputFileNameMapping>xwiki.war</outputFileNameMapping>
<useProjectArtifact>false</useProjectArtifact>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
<directory>${project.basedir}/src/main/docker</directory>
<outputDirectory>.</outputDirectory>
<includes>
<include>**/*.sh</include>
</includes>
<fileMode>755</fileMode>
</fileSet>
<fileSet>
<directory>${project.basedir}/src/main/docker</directory>
<outputDirectory>.</outputDirectory>
<excludes>
<exclude>**/*.sh</exclude>
</excludes>
</fileSet>
</fileSets>
</assembly>And all the Dockerfile and ancillary files required to generate the image are in src/main/docker/*.
- Then in the test modules, start the Docker containers from the generated Docker images. Check the full POM:[...]
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
<configuration>
<imagePullPolicy>IfNotPresent</imagePullPolicy>
<showLogs>true</showLogs>
<images>
<image>
<alias>mysql-xwiki</alias>
<name>mysql:5.7</name>
<run>
[...]
<image>
<alias>xwiki</alias>
<name>xwiki:latest</name>
<run>
[...]
<image>
<name>xwiki-maven</name>
<run>
[...]
<volumes>
<bind>
<volume>${project.basedir}:/usr/src/mymaven</volume>
<volume>${user.home}/.m2:/root/.m2</volume>
</bind>
</volumes>
<workingDir>/usr/src/mymaven</workingDir>
<cmd>
<arg>mvn</arg>
<arg>verify</arg>
<arg>-Pdocker-maven</arg>
</cmd>
[...]
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin> - Notice that last container we start, i.e. xwiki-maven is configured to map the current Maven source as a directory inside the Docker container and it starts Maven inside the container to run the functional tests using the docker-maven Maven profile.
Limitations:
- The environment setup is done from the build (Maven), which means that the developer needs to start it before executing the test from his IDE. This can cause frictions in the developer workflow.
- We found issues when running Docker inside Docker and Maven inside Maven, specifically when having Maven start the docker container containing the browsers, itself starting a Maven build which starts the browser and then the tests. This resulted in the Maven build slowing down and cringing to a halt. This was probably due to the fact that Docker will use up a lot of memory by default and we would need to control all processes (Maven, Surefire, Docker, etc) and control very precisely the memory they use. Java10 would help but we're not using it yet and we're currently stuck on Java8.
Experimentation 4: In Java Tests using Selenium Jupiter
The idea is to use Selenium Jupiter to automatically start/stop the various Browsers to be used by Selenium directly from the JUnit5 tests.
Note that XWiki has test framework on top of Selenium, with a class named TestUtil providing various APIs to help set up tests. Thus we need to make this class available to the test too by injecting it as test method parameter for example. Thus I developed a XWikiDockerExtension JUnit5 extension that initializes the XWiki testing framework and that does this injection.
Here's how a very simple test look like:
public class SeleniumTest
{
@BeforeAll
static void setup()
{
// TODO: move to the pom
SeleniumJupiter.config().setVnc(true);
SeleniumJupiter.config().setRecording(true);
SeleniumJupiter.config().useSurefireOutputFolder();
SeleniumJupiter.config().takeScreenshotAsPng();
SeleniumJupiter.config().setDefaultBrowser("firefox-in-docker");
}
@Test
public void test(WebDriver driver, TestUtils setup)
{
driver.get("http://xwiki.org");
assertThat(driver.getTitle(), containsString("XWiki - The Advanced Open Source Enterprise and Application Wiki"));
driver.findElement(By.linkText("XWiki's concept")).click();
}
}
Limitations:
- Works great for spawning Browser containers but doesn't support other types of containers such as DBs or Servlet Containers. Would need to implement the creation and start of them in a custom manner which is a lot of work.
Experimentation 5: in Java Tests using TestContainers
This idea builds on the Selenium Jupiter idea but using a different library, called TestContainers. It's the same idea but it's more generic since TestContainers allows creating all sorts of Docker containers (Selenium containers, DB containers, custom containers).
Here's how it works:
And here's an example of a Selenium test using it:
public class MenuTest
{
@Test
public void verifyMenu(TestUtils setup)
{
verifyMenuInApplicationsPanel(setup);
verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
}
private void verifyMenuInApplicationsPanel(TestUtils setup)
{
// Log in as superadmin
setup.loginAsSuperAdmin();
// Verify that the menu app is displayed in the Applications Panel
ApplicationsPanel applicationPanel = ApplicationsPanel.gotoPage();
ViewPage vp = applicationPanel.clickApplication("Menu");
// Verify we're on the right page!
assertEquals(MenuHomePage.getSpace(), vp.getMetaDataValue("space"));
assertEquals(MenuHomePage.getPage(), vp.getMetaDataValue("page"));
// Now log out to verify that the Menu entry is not displayed for guest users
setup.forceGuestUser();
// Navigate again to the Application Menu page to perform the verification
applicationPanel = ApplicationsPanel.gotoPage();
assertFalse(applicationPanel.containsApplication("Menu"));
// Log in as superadmin again for the rest of the tests
setup.loginAsSuperAdmin();
}
...
Some explanations:
- The @UITest annotation triggers the execution of the XWiki Docker Extension for JUnit5 (XWikiDockerExtension)
- In turn this extension will perform a variety of tasks:
- Verify if an XWiki instance is already running locally. If not, it will generate a minimal XWiki WAR containing just what's needed to test the module the test is defined in. Then in turn, it will start a database and start a servlet container and deploy the XWiki WAR in it by using a Docker volume mapping
- Initialize Selenium and start a Browser (the exact browser to start can be controlled in a variety of ways, with system properties and through parameters of the @UITest annotation).
- It will also start a VNC docker container to record all the test execution, which is nice when one needs to debug a failing test and see what happened.
Current status:
Current Status as of 2018-07-13:
- Browser containers are working and we can test in both Firefox and Chrome. Currently XWiki is started the old way, i.e. by using the XWiki Maven Package Plugin which generates a full XWiki distribution based on Jetty and HSQLDB.
- We have implemented the ability to fully generate an XWiki WAR directly from the tests (using the ShrinkWrapp library), which was the prerequisite for being able to deploy XWiki in a Servlet Container running in a Docker container and to start/stop it.
- Work in progress:
- Support an existing running XWiki and in this case don't generate the WAR and don't start/stop the DB and Servlet Container Docker containers.
- Implement the start/stop of the DB Container (MySQL and PostgreSQL to start with) from within the test using TestContainer's existing MySQLContainer and PostgresSQL containers.
- Implement the start/stop of the Servlet Container (Tomcat to start with) from within the test using TestContainer's GenericContainer feature.
Note that most of the implementation is generic and can be easily reused and ported to software other than XWiki.
- Only supports 2 browsers FTM: Firefox and Chrome. More will come. However it's going to be very hard to support browsers requiring Windows (IE11, Edge) or Mac OSX (Safari). Preliminary work is in progress in TestContainers but it's unlikely to result in any usable solution anytime soon.
- Note that forced us to allow using Selenium 3.x while all our tests are currently on Selenium 2.x. Thus we implemented a solution to have the 2 versions run side by side and we modified our Page Objects to use reflection and call the right Selenium API depending on the version. Luckily there aren't too many places where the Selenium API has changed from 2.x to 3.x. Our goal is now to write new functional UI tests in Selenium 3 with the new TestContainer-based tedting framework and progressively migrate tests using Selenium 2.x to this new framework.
- The full execution of the tests take a bit longer than what we used to have with a single environment made of HSQLDB+Jetty. Measures will be taken when the full implementation is finished to evaluate the total time it takes.
Future ideas:
- Discuss with the CAMP developers to see how their mutation engine could be executed as a Java library so that it could be integrated in the XWiki testing framework. Namely to issues were open on the CAMP issue tracker to discuss this:
Conclusion
At this point in time I'm happy with our last experiment and implementation based on TestContainers. It allows to run environment tests directly from your IDE with no other prerequisite than having Docker installed on your machine. This means it also works from Maven or from any CI. We need to finish the implementation and this will give the XWiki project the ability to run tests on various combinations of configurations.
Once this is done we should be able to tackle the next step which involves more exotic configurations such as running XWiki in a cluster, configuring a LibreOffice server to test importing office documents in the XWiki, and even configuring an external SOLR instance. However once the whole framework is in place, I don't expect this to cause any special problems.
Last, but not least, once we get this ability to execute various configurations, it'll be interesting to use a configuration mutation engine such as the one provided by CAMP in order to test various configurations in our CI. Since testing lots of them would be very costly in term of number of Agents required and CPU power, one idea is to have a job that executes, say, once per day with a random configuration selected and that reports how the tests perform in it.