On the XWiki project, we currently have automated functional tests that use Selenium and Jenkins. However they exercise only a single configuration: HSQLDB, Jetty and FireFox (and all on a fixed version).
XWiki SAS is part of the STAMP research project and one domain of this research is improving configuration testing.
As a first step I've worked on providing official XWiki images but I've only provided 2 configurations (XWiki on Tomcat + Mysql and on Tomcat + PostgreSQL) and they're not currently exercised by our functional tests.
Thus I'm proposing below an architecture that should allow XWiki to be tested on various configurations:
- Various supported databases and versions
- Various Servlet containers and versions
- Various Browsers and versions
Here's what I think it would mean in term of a Jenkins Pipeline (note that at this stage this is pseudo code and should not be understood literally):
agent {
docker {
image 'xwiki-maven-firefox'
args '-v $HOME/.m2:/root/.m2'
}
}
stages {
stage('Test') {
steps {
docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->
docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
[...]
wrap([$class: 'Xvnc']) {
withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
[...]
sh "mvn ..."
}
}
}
}
}
}
}
}
Some explanations:
- We would setup a custom Docker Registry so that we can prepare images that would be used by the Jenkins pipeline to create containers
- Those images could themselves be refreshed regularly based on another pipeline that would use the docker.build() construct
- We would use a Jenkins Agent dynamically provisioned from an image that would contain: sshd and a Jenkins user (so that Jenkins Master can communicate with it), Maven, VNC Server and a browser (FireFox for ex). We would have several such images, one per browser we want to test with.
- Note that since we want to support only the latest browsers versions for FF/Chrome/Safari we could use apt to update (and commit) the browser version in the container prior to starting it, from the pipeline script.
- Then the pipeline would spawn two containers: one for the DB and one for the Servlet container. Importantly for the Servlet container, I think we should mount a volume that points to a local directory on the agent, which would contain the XWiki exploded WAR (done as a pre-step by the Maven build). This would save time and not have to recreate a new image every time there's a commit on the XWiki codebase!
- The build that contains the tests will be started by the Agent (and we would mount the the Maven local repository as a volume in order to sped up build times across runs).
- Right now the XWiki build already knows how to run the functional tests by fetching/exploding the XWiki WAR in the target directory and then starting XWiki directly from the tests, so all we would need to do is to make sure we map this directory in the container containing the Servlet container (e.g. in Tomcat it would be mapped to [TOMCATHOME]/webapps/xwiki).
This is just architecture at this stage. Now we need to put that in practice and find the gotchas (there always are 
).
WDYT? Could this work? Are you doing this yourself?
Stay tuned, I should be able to report about it went in the coming weeks/months.
