I've been trying to define the best solution for doing functional testing in multiple environments (different DBs, Servlet containers, browsers) for XWiki. Right now on XWiki we test automatically on a single environment (Firefox, Jetty, HSQLDB) and we do the other environment tests manually.
So I've been going through different experimentations, finding out issues and limitations with them and progressing towards the perfect solution for us. Here are the various experiments we conducted. Note that this work is being done as part of the STAMP Research Project.
Experimentation 1: CAMP from STAMP
CAMP is a tool developed by some partners on the STAMP research project and it acts as a remote testing service. You give it some Dockerfile and it'll create the image, start a container, execute some commands (that you also provide to it and that are used to validate that the instance is working fine) and then stop the container. In addition it performs configuration mutation on the provided Dockerfile. This means it'll make some variations to this file, regenerate the image and re-run the container.
Here's how CAMP works (more details including how to use it on XWiki can be found on the CAMP home page):
Image from the CAMP web site
Limitations for the XWiki use case needs:
- Relies on a service. This service can be installed on a server on premises too but that means more infrastructure to maintain for the CI subsystem. Would be better if integrated directly in Jenkins for example.
- Cannot easily run on the developer machine which is important so that devs can test what they develop on various environments and so that they can debug reported issues on various environments
- Even though mutation of configuration is an interesting concept, it's not a use case for XWiki which has several well-defined configurations that are supported. It's true that it could be interesting to have fixed topologies and only vary versions of servers (DB version, Servlet Container version and Java version - We don't need to vary Browser versions since we support only the latest version) but we think the added value vs the infrastructure cost might not be that interesting for us. However, it could still be interesting for example by randomizing the mutated configuration and only running tests on one such configuration per day to reduce the need of having too many agents and leaving them free for the other jobs.
Experimentation 2: Docker on CI
I blogged in the past about this strategy.
The main idea for this experiment was to use a Jenkins Pipeline with the Jenkins Plugin for Docker, allowing to write pipelines like this:
agent {
docker {
image 'xwiki-maven-firefox'
args '-v $HOME/.m2:/root/.m2'
}
}
stages {
stage('Test') {
steps {
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->
docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
[...]
wrap([$class: 'Xvnc']) {
withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
[...]
sh "mvn ..."
}
}
}
}
}
}
}
}
Limitations:
- Similar to experimentation 1 with CAMP, this relies on the CI to execute the tests and doesn't allow developers to test and reproduce issues on their local machines.
Experimentation 3: Maven build with Fabric8
The next idea was to implement the logic in the Maven build so that it could be executed on developer machines. I found the very nice Fabric8 Maven plugin and came up with the following architecture that I tried to implement:
The main ideas:
- Generate XXXX
Experimentation 4: In Java Tests using Jupiter Selenium
TODO
Experimentation 5: in Java Tests using TestContainers
TODO
