On the XWiki project, we currently have automated functional tests that use Selenium and Jenkins. However they exercise only a single configuration: HSQLDB, Jetty and FireFox (and all on a fixed version).

XWiki SAS is part of the STAMP research project and one domain of this research is improving configuration testing.

As a first step I've worked on providing official XWiki images but I've only provided 2 configurations (XWiki on Tomcat + Mysql and on Tomcat + PostgreSQL) and they're not currently exercised by our functional tests.

Thus I'm proposing below an architecture that should allow XWiki to be tested on various configurations:

• Various supported databases and versions
• Various Servlet containers and versions
• Various Browsers and versions

Here's what I think it would mean in term of a Jenkins Pipeline (note that at this stage this is pseudo code and should not be understood literally):

Some explanations:

• We would setup a custom Docker Registry so that we can prepare images that would be used by the Jenkins pipeline to create containers
• Those images could themselves be refreshed regularly based on another pipeline that would use the docker.build() construct
• We would use a Jenkins Agent dynamically provisioned from an image that would contain: sshd and a Jenkins user (so that Jenkins Master can communicate with it), Maven, VNC Server and a browser (FireFox for ex). We would have several such images, one per browser we want to test with.
  • Note that since we want to support only the latest browsers versions for FF/Chrome/Safari we could use apt to update (and commit) the browser version in the container prior to starting it, from the pipeline script.
• Then the pipeline would spawn two containers: one for the DB and one for the Servlet container. Importantly for the Servlet container, I think we should mount a volume that points to a local directory on the agent, which would contain the XWiki exploded WAR (done as a pre-step by the Maven build). This would save time and not have to recreate a new image every time there's a commit on the XWiki codebase!
• The build that contains the tests will be started by the Agent (and we would mount the the Maven local repository as a volume in order to sped up build times across runs). 
• Right now the XWiki build already knows how to run the functional tests by fetching/exploding the XWiki WAR in the target directory and then starting XWiki directly from the tests, so all we would need to do is to make sure we map this directory in the container containing the Servlet container (e.g. in Tomcat it would be mapped to [TOMCATHOME]/webapps/xwiki).

This is just architecture at this stage. Now we need to put that in practice and find the gotchas (there always are ).

WDYT? Could this work? Are you doing this yourself?

Stay tuned, I should be able to report about it went in the coming weeks/months.

Imagine that you have a software project and you wish to have a web site to document everything related to the project (user documentation, dev documentation, news, etc).

You may wonder whether you should go with a statically-generated site (using GitHub Pages for example or some homemade solution) or use a wiki such as XWiki.

I've tried to list the pros of each solution below, trying to be as impartial as possible (not easy since I'm one of the developers of the XWiki project ). Don't hesitate to comment if you have other points or if some of my points are not fully accurate, and I'll update this blog post. Thanks!

Pros of a statically-generated site

• Hosting is easier, as it only consists of static pages and assets. More generally, it's simpler to get started (but compensated by the need to set up some DSL and/or build if you don't want to enter content in HTML)
• Maintenance is simplified, no database to backup for example or software to upgrade
• Documentation can be versioned along with the code it documents
• (GitHub) You get a review system built-in with Pull Requests
  • Update 2023-08-04: XWiki now has a Change Request Extension that offers similar features.
• (GitHub) You can tag the whole documentation and have branches per released versions
• Easier to scale. It's easy to make web servers scale to massively large number of users.

Pros of a wiki with XWiki

• Easy for anyone to enter content, including for non-technical users. No HTML to know nor any specific DSL to understand. No need for an account on GitHub nor the need to understand how to make a PR.
• Much faster to enter content through the WYSIWYG editor or through wiki markup.
• Changes are immediately visible. You edit a page and click save or preview and you can see the result. No need to go through a build that will push the changes. With preview you can go back to editing the page if you're not satisfied and that's very fast. With WYSIWYG editor you don't even need to preview (since WYSIWYG is... WYSIWYG).
• Richer search, see for example the XWiki.org Search UI vs the Groovy Search UI.
• Ability for users to comment the website pages. 
• Ability for users to watch pages and be notified when there are changes to those pages
• Ability to see what's new in the documentation and the changes made
• Your pages are not saved along the code in a single SCM. However XWiki pages can be exported to an XML format and the exported pages can be saved in the same SCM as the code. There are even GitHub Extension and SVN Extensionto help you do that.
• Pages can be exported in different formats: OpenOffice, Word, PDF, etc. Note that it's also possible to export to HTML in order to offer a static web site for example.
• Ability to display large quantity of filterable data in tables with great scalability. For example:
  • display references of some sites using XWiki 
  • Page Index
  • Release Notes
  • FAQ
  • Extensions
• Ability to have dynamic examples that can be tested directly in the wiki. For example the XWiki Rendering can be tested live.
• Perform dynamic actions, such as generating GitHub statistics for your project.
• Perform dynamic actions by writing some scripts in wiki page. For example imagine you'd like to list all Extensions having a name containing User and located in the the extensions subwiki, you'd simply write the following in a wiki page (you can try it on XWiki Playground):
  {{velocity}}
  #set ($query = $services.query.xwql("where doc.object(ExtensionCode.ExtensionClass).name like '%User%'").setWiki('extensions'))
  #foreach ($itemDoc in $query.execute())
    * [[extensions:$itemDoc]]
  #end
  {{/velocity}}
• More generally write some applications to enter data easily for your website. It's easy with Applications within Minutes.

Conclusion

IMO the choice will hugely depend on your needs from the above list but also on how easy/hard it is for you to get some hosting for XWiki:

• If it's an internal company project, it shouldn't be hard to install and host an XWiki instance (unless your company doesn't have any IT department, see below in this case)
• If you have some budget you could use XWiki SAS's cloud solution (starts at 10 euros/month for 10 users)
• If you're an open source project and have no budget, then you have 2 choices:
  • The XWiki projet has a free farm and if your project doesn't require professional hosting then it could be a good option. 
  • If your project is quite visible/large or if you need professional hosting, XWiki SAS offers free hosting for Open Source projects.

It would be great if more open source forges such as the Apache Software Foundation, the Eclipse Foundation and others were offering XWiki hosting for their projects as an option.

So what would you choose for your project?

Got to do a quickie at OW2Conf'17 for 15 minutes about What's New in XWiki. Since I didn't know if the audience already was using XWiki, I did a demo mixing some older but interesting stuff with some new features.

Features demoed:

• Nested Pages
• Page Templates
• Menu Application (bundled now)
• Color Theme Editor (with live preview)
• OnlyOffice Extension installed with the Extension Manager
• Application Within Minutes to create your own custom forms/structures

On the XWiki project we've started moving to Jenkins 2.0 and to using the Pipeline feature through Jenkinsfiles.

When we run our functional tests (we use Selenium2/Webdriver), we record a screenshot when a test fails. Previously we had a Groovy Scriptler script (written by Eduard Moraru, an XWiki committer) to automatically change the description of a Jenkins test page to include the screenshot as on:

So we needed to port this script to a Jenkinsfile. Here's the solution I came up with:

Note that for this to work you need to:

• Install the Groovy Postbuild plugin. This exposes the manager variable needed by the script.
• Add the required security exceptions to http://<jenkins server ip>/scriptApproval/ if need be
• Install the Pegdown Formatter plugin and set the description syntax to be Pegdown in the Global Security configuration (http://<jenkins server ip>/configureSecurity). Without this you won't be able to display HTML (and the default safe HTML option will strip out the datauri content).

Enjoy!

The XWiki project is using a strategy to try to ensure that quality goes in the upward direction.

In short we fail the build if the Jacoco-computed coverage is below a per-module threshold. Devs can only increase the threshold but are not supposed to lower it.

However, from time to time, it happens that dev reduce the threshold (for example, when fixing a bug and this removes some lines of code and the coverage is lowered and the dev doesn't have the time to improve existing tests, etc).

Since we've been following this strategy for a long time now (at least since 2013), I thought it would be interesting to check, for a small subset of XWiki, how we fared.

Note that - denotates some modules that do not exist at a given date or for which the coverage is empty (for example a module with only Java interfaces).

Conclusions:

• Coverage has not increased substantially in general. However this is computed on xwiki-commons and those modules are pretty stable and don't change much. It would be interesting to compute something similar for xwiki-platform.
• Out of 14 modules that have seen their TPC modified between 2013 and May 2017, 10 have seen their coverage increase (that's 71%). So 4 have seen their coverage be reduced by up to -3.6% max.

So while we could better, it's still not too bad and the strategy seems to be globally working.

The Jenkins Pipeline plugin includes a very nice feature which is the "GitHub Organization" job type. This job type will scan a given GitHub organization repositories for Jenkinsfile files and when found will automatically create a pipeline job for them.

This has some nice advantages:

• You save your Jenkins job configuration in your SCM (git in our case, in the Jenkinsfile), next to your code. You can receive email diffs to show who made modifications to it, the reason and understand the changes.
• It supports branches: when you create a branch it's automatically discovered by Jenkins and the build is executed on it. And if the branch gets removed, it's removed automatically from Jenkins too. This point is awesome for us since we used to have groovy script to execute to copy jobs when there were new branches and when branches were removed.

So we started exploring this for the XWiki project, starting with Contrib Extensions.

Here's a screenshot of our Github Organization job for XWiki Contrib:

And here's an example of a pipeline job executing:

Now if you implement this you'll quickly find that you want to share pipeline scripts between Jenkinsfile, in order to not have duplicates.

FYI here's what the Jenkinsfile for the syntax-markdown pipeline job shown above looks like:

Simple, isn't it?  The trick is that we've configured Jenkins to automatically load a Global Pipeline Library (implicit load). You can do that by saving libraries at the root of SCM repositories and configure Jenkins to load them from the SCM sources (see this Jenkins doc for more details).

So we've created this GitHub repository and we've coded a vars/xwikiModule.groovy file. At the moment of writing this is its content (I expect it to be improved a lot in the near future):

Ideas of some next steps:

• Put back our support for false positives in the pipeline script
• Make xvnc optional with a xvnc = true | false parameter option

Right now there's one limitation I've found: It seems I need to manually click on "Re-scan Organization" in the Jenkins UI so that new Jenkinsfile added to repositories are taken into account. I hope that will get fixed soon. One workaround would be to add another Jenkins job to do that regularly but it's not perfect. Also note that you absolutely must authenticate against GitHub as otherwise you'll quickly reach the GitHub API request limit (when authenticated you are allowed 5000 requests per hour).

Anyway it's great and I love it.

On the XWiki project it's been years that we've been checking automatically for backward compatibilities in our APIs as part of our build (It's even documented here).

Recently, we've moved to Java 8 for XWiki 8.1 and we've discovered that the tool we were using, CLIRR, doesn't work anymore with Java 8. So I searched, without much hope, for an alternative, and I was surprised (and delighted) to find that there are 2 valid solutions nowadays that support Java 8:

• japicmp
• Revapi

I've tried both and they both had issues. However, I've discovered that the maintainers or these 2 solutions were really cool guys and they very quickly fixed any issue I've raised. Big thanks to Lukas Krejci and Martin Mois, you're awesome guys!

In the end the XWiki project has had to make a difficult choice and we chose Revapi (Some reasons mentioned here but honestly they're both valid choices).

So here's how we use it now:

• In our top level POM:
  ...
       <plugins>
  ...
         <!-- Used for checking backward compatibility (binary and source) -->
         <plugin>
           <groupId>org.revapi</groupId>
           <artifactId>revapi-maven-plugin</artifactId>
           <!-- Lock down plugin version for build reproducibility -->
           <version>0.4.5</version>
           <dependencies>
             <dependency>
               <groupId>org.revapi</groupId>
               <artifactId>revapi-java</artifactId>
               <version>0.9.0</version>
             </dependency>
           </dependencies>
           <executions>
             <execution>
               <id>revapi-check</id>
               <goals>
                 <goal>check</goal>
               </goals>
             </execution>
           </executions>
           <configuration>
             <oldVersion>${xwiki.compatibility.previous.version}</oldVersion>
             <skip>${xwiki.revapi.skip}</skip>
             <analysisConfiguration>
              {
                "revapi": {
                  "java": {
                    "filter": {
                      "packages": {
                        "regex": true,
                        "include": ["org\\.xwiki\\..*"],
                        "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                      }
                    }
                  }
                }
              }
             </analysisConfiguration>
           </configuration>
         </plugin>
  ...
• Then in specific module POMs we override the configuration to add excludes (for backward and source incompatibilities that we know about and have voluntarily decided to do). For example:
  ...
       <plugin>
         <groupId>org.revapi</groupId>
         <artifactId>revapi-maven-plugin</artifactId>
         <configuration>
           <analysisConfiguration><![CDATA[
              {
                "revapi": {
                  "java": {
                    "filter": {
                      "packages": {
                        "regex": true,
                        "include": ["org\\.xwiki\\..*"],
                        "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                      }
                    }
                  },
                  "ignore": [
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.List<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                      "new": "method java.util.List<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    },
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                      "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    },
                    {
                      "code": "java.method.returnTypeTypeParametersChanged",
                      "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                      "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                      "justification": "? return type makes signature more complex for nothing"
                    }         
                  ]
                }
              }
            ]]></analysisConfiguration>
         </configuration>
       </plugin>
  ...
• Now the interesting part is in generating reports. We've chosen an original way of doing this: We dynamically generate the Revapi report on our release notes wiki pages using Groovy. The big advantage is that there's no work to be done for the Release Manager:
  • Here's an example from the XWiki 8.1M1 Release Notes

    

  • The Groovy script below does the following:
    • Use the GitHub REST API to get the content of the pom.xml containing the Revapi excludes.
    • Parse the XML using Groovy
    • Display a report out of it
  • And here's the Groovy script packaged as a Wiki Macro for the curious ones:
    {{groovy}}
    import groovy.json.*
    
    def getIgnores(def repo, def path)
    {
     def url = "https://api.github.com/repos/xwiki/${repo}/contents/${path}".toURL().text
     def result = new JsonSlurper().parseText(url)
     def xml = new String(result.content.decodeBase64())
      result = new XmlSlurper().parseText(xml)
     def revapi = result.build.plugins.plugin.'**'.find { node ->
        node.artifactId.text() == 'revapi-maven-plugin'
     }
     if (revapi) {
        result = new JsonSlurper().parseText(revapi.configuration.analysisConfiguration.text())
       return result.revapi.ignore
     } else {
       return ''
     }
    }
    
    def displayIgnores(def ignores)
    {
      result = new JsonSlurper().parseText(ignores)
      result.each() {
        it.each() {
          println "* {{{${it.justification}}}}"
          println "** Violation type: {{code}}${it.code}{{/code}}"
          println "** Old: {{code}}${it.old}{{/code}}"
         if (it.new) {
            println "** New: {{code}}${it.new}{{/code}}"
         }
       }
     }
    }
    
    def getViolations(def version)
    {
     def xobject = doc.getObject('ReleaseNotes.BackwardCompatibility')
     if (!xobject) {
        xobject = doc.newObject('ReleaseNotes.BackwardCompatibility')
       def commonsTag
       def renderingTag
       def platformTag
       if (version == 'master') {
          commonsTag = renderingTag = platformTag = 'master'
       } else {
          commonsTag = "xwiki-commons-${version}"
          renderingTag = "xwiki-rendering-${version}"
          platformTag = "xwiki-platform-${version}"
       }
       def jsonCommons = getIgnores('xwiki-commons', "xwiki-commons-core/pom.xml?ref=${commonsTag}")
       def jsonRendering = getIgnores('xwiki-rendering', "pom.xml?ref=${renderingTag}")
       def jsonPlatform = getIgnores('xwiki-platform', "xwiki-platform-core/pom.xml?ref=${platformTag}")
        xobject.set('violations', JsonOutput.prettyPrint(JsonOutput.toJson([jsonCommons, jsonRendering, jsonPlatform])))
        doc.save('Added backward-compatiblity violations data', true)
     }
     return xobject.getProperty('violations').value
    }
    
    displayIgnores(getViolations(xcontext.macro.params.version))
    {{/groovy}}

In conclusion we're very happy with the move, especially since we now have Java 8 support but also because Revapi provides more checks than what CLIRR was doing. ...

On the XWiki project we have enabled Jenkins's Incremental Build feature:

This seemed like a nice feature to speed up our CI when building our Maven jobs. Alas, it doesn't work!

The problem is that from time to time you'll get build failure such as:

You'd think it's not possible, especially as we use Maven's -U flag and thus, even if the artifact is not present in the local repository it should be downloaded from the Maven Remote Repository that we use (and it's available there!).

The reason is because of a Maven bug: MNG-5542. What happens is that the Incremental Build feature will use the -pl Maven parameter to list all the Maven projects to build and when this feature is used, artifacts declared in the <parent> section of your POMs are just ignored and not downloaded...

The consequence is that your build will fail from time to time when one artifact declared in one of your <parent> is not present in your local repository... If you have decided to have one Maven repository per Job in Jenkins - which would seem a good idea to isolate your jobs and to be able to use the Parallel build feature of Jenkins - then you'll hit the problem very frequently...

So in the end you have to choose between Parallel builds and Incremental builds but you cannot have both at the same time!

Note that even with Parallel builds turned off, you build will fail from time to time, just less frequently...

One solution for the XWiki project would be to break our big job that builds the Platform (over 100 modules located in the same Git repo that we release together under a single version) into 100 jobs. But doing this manually is a no go so we would need to script this or wait for some Jenkins dev to implement this idea...

Big pain!

JIRA is a pain when you need to create lots of projects with custom Schemes. On the XWiki open source project we allow the community to contribute extensions and we offer a place to host the sources in the xwiki-contrib GitHub organization. We also offer for each project a JIRA project where users can report issues and where developers can track what they're working on. 

Thus we have the need to create lots of JIRA projects. Today, I got bored of repeating over and over the same setup for creating such projects. It involves 9 steps:

1. Setting the category
2. Setting the custom XWiki Workflow Scheme
3. Setting the custom XWiki Screen Scheme
4. Setting the custom XWiki Field Configuration Scheme
5. Setting the custom XWiki Notifications Scheme
6. Setting the custom XWiki Permission Scheme
7. Setting the Project Lead
8. Setting the project URL
9. Setting some groups in one of the project's Roles

Following this tutorial (and a lots of googling ) I was able to automate steps 1-6 in the above list (Source code).

Now when we create a new JIRA project we get this nice template:

The main code doing the work is in XWikiAddProjectHook.java:

Enjoy!

I'd like to explain what we do on the XWiki open source project to preserve backward compatibility of our Java APIs. We've done several iterations and we now have a process that is working quite well, even though there are still some issues but they're unavoidable with Java for the time being.

So here it goes:

• We start by having a special package named internal and everything we put in there is considered implementation details and should never be used by users of our APIs. We're allowed to modify anything at any time in the internal package. For example: org.xwiki.instance.internal.*
• We also use a Component-based approach, which forces us to separate interfaces (which are public) from implementation (which go in the internal package).
• We have defined an advanced deprecation mechanism which allows us to move deprecated code that we no longer use to legacy maven module (using Aspects with AspectJ) that can be optionally installed at runtime, and which we never use at build time. This prevents us from using any deprecated legacy code and it allows us to push away the cruft under the carpet  (without breaking backward compatibility!)
• We have the notion of "Young API" (a.k.a Unstable APIs) and when a new API is introduced, the developer can (and should) use the @Unstable annotation to signify to users that this API is unstable for the moment and can change at any time. We only allow an API to remain unstable for a full development cycle. We've recently introduced a custom Checkstyle check that enforces this!
• Last, we use the Maven CLIRR plugin to make sure we never break an API unintentionally (it fails our build). This allows us to only carefully and intentionally break our APIs. It also allows to us to mention what we break in our Release Notes (example).

The important point IMO is that we have automated tools to ensure that our strategy is applied, namely:

• CLIRR
• Our Unstable Annotation Checker (a custom checkstyle rule)

This is working pretty well for us even though we do break backward compatibility from time to time, when we judge that the code touched is unlikely to be used and working around the breakage would be too complex and would take too much time (for example adding a method in an interface requires writing a new interface and modifying all code accepting that interface). Luckily this is going to be made somewhat simpler in Java in the future with Default Methods (introduced in Java 8). It won't fit all cases though.

Another example of backward compatibility aspect that we don't handle is when someone changes what an API returns or what a API does. A simple example is if a method returns an object of type X but the implementation changes to return another implementation of type X that behaves differently... This is a tough one to detect and prevent.

WDYT? Are you doing something else in your projects? (I'd be curious of ways to improve even further what we do!).