Category: \X\W\i\k\i (29 posts) /xwiki/resources/icons/silk/feed.png

Sep 17 2017

Using Docker + Jenkins to test configurations

On the XWiki project, we currently have automated functional tests that use Selenium and Jenkins. However they exercise only a single configuration: HSQLDB, Jetty and FireFox (and all on a fixed version).

XWiki SAS is part of the STAMP research project and one domain of this research is improving configuration testing.

As a first step I've worked on providing official XWiki images but I've only provided 2 configurations (XWiki on Tomcat + Mysql and on Tomcat + PostgreSQL) and they're not currently exercised by our functional tests.

Thus I'm proposing below an architecture that should allow XWiki to be tested on various configurations:

architecture.png

Here's what I think it would mean in term of a Jenkins Pipeline (note that at this stage this is pseudo code and should not be understood literally):

pipeline {
  agent {
    docker {
      image 'xwiki-maven-firefox'
      args '-v $HOME/.m2:/root/.m2'
    }
  }
  stages {
    stage('Test') {
      steps {
        docker.image('mysql:5').withRun('-e "MYSQL_ROOT_PASSWORD=my-secret-pw"') { c ->                    
          docker.image('tomcat:8').withRun('-v $XWIKIDIR:/usr/local/tomcat/webapps/xwiki').inside("--link ${c.id}:db") {
            [...]
            wrap([$class: 'Xvnc']) {
              withMaven(maven: mavenTool, mavenOpts: mavenOpts) {
                [...]
                sh "mvn ..."
              }
            }
          }
        }
      }
    }
  }
}

Some explanations:

  • We would setup a custom Docker Registry so that we can prepare images that would be used by the Jenkins pipeline to create containers
  • Those images could themselves be refreshed regularly based on another pipeline that would use the docker.build() construct
  • We would use a Jenkins Agent dynamically provisioned from an image that would contain: sshd and a Jenkins user (so that Jenkins Master can communicate with it), Maven, VNC Server and a browser (FireFox for ex). We would have several such images, one per browser we want to test with.
    • Note that since we want to support only the latest browsers versions for FF/Chrome/Safari we could use apt to update (and commit) the browser version in the container prior to starting it, from the pipeline script.
  • Then the pipeline would spawn two containers: one for the DB and one for the Servlet container. Importantly for the Servlet container, I think we should mount a volume that points to a local directory on the agent, which would contain the XWiki exploded WAR (done as a pre-step by the Maven build). This would save time and not have to recreate a new image every time there's a commit on the XWiki codebase!
  • The build that contains the tests will be started by the Agent (and we would mount the the Maven local repository as a volume in order to sped up build times across runs). 
  • Right now the XWiki build already knows how to run the functional tests by fetching/exploding the XWiki WAR in the target directory and then starting XWiki directly from the tests, so all we would need to do is to make sure we map this directory in the container containing the Servlet container (e.g. in Tomcat it would be mapped to [TOMCATHOME]/webapps/xwiki).

This is just architecture at this stage. Now we need to put that in practice and find the gotchas (there always are emoticon_wink).

WDYT? Could this work? Are you doing this yourself?

Stay tuned, I should be able to report about it went in the coming weeks/months.

Jul 15 2017

XWiki vs statically-generated site

Imagine that you have a software project and you wish to have a web site to document everything related to the project (user documentation, dev documentation, news, etc).

You may wonder whether you should go with a statically-generated site (using GitHub Pages for example or some homemade solution) or use a wiki such as XWiki.

I've tried to list the pros of each solution below, trying to be as impartial as possible (not easy since I'm one of the developers of the XWiki project emoticon_wink). Don't hesitate to comment if you have other points or if some of my points are not fully accurate, and I'll update this blog post. Thanks!

Pros of a statically-generated site

  • Hosting is easier, as it only consists of static pages and assets. More generally, it's simpler to get started (but compensated by the need to set up some DSL and/or build if you don't want to enter content in HTML)
  • Maintenance is simplified, no database to backup for example or software to upgrade
  • Documentation can be versioned along with the code it documents
  • (GitHub) You get a review system built-in with Pull Requests
  • (GitHub) You can tag the whole documentation and have branches per released versions
  • Easier to scale. It's easy to make web servers scale to massively large number of users.

Pros of a wiki with XWiki

  • Easy for anyone to enter content, including for non-technical users. No HTML to know nor any specific DSL to understand. No need for an account on GitHub nor the need to understand how to make a PR.
  • Much faster to enter content through the WYSIWYG editor or through wiki markup.
  • Changes are immediately visible. You edit a page and click save or preview and you can see the result. No need to go through a build that will push the changes. With preview you can go back to editing the page if you're not satisfied and that's very fast. With WYSIWYG editor you don't even need to preview (since WYSIWYG is... WYSIWYG).
  • Richer search, see for example the XWiki.org Search UI vs the Groovy Search UI.
  • Ability for users to comment the website pages. 
  • Ability for users to watch pages and be notified when there are changes to those pages
  • Ability to see what's new in the documentation and the changes made
  • Your pages are not saved along the code in a single SCM. However XWiki pages can be exported to an XML format and the exported pages can be saved in the same SCM as the code. There are even GitHub Extension and SVN Extensionto help you do that.
  • Pages can be exported in different formats: OpenOffice, Word, PDF, etc. Note that it's also possible to export to HTML in order to offer a static web site for example.
  • Ability to display large quantity of filterable data in tables with great scalability. For example:
  • Ability to have dynamic examples that can be tested directly in the wiki. For example the XWiki Rendering can be tested live.
  • Perform dynamic actions, such as generating GitHub statistics for your project.
  • Perform dynamic actions by writing some scripts in wiki page. For example imagine you'd like to list all Extensions having a name containing User and located in the the extensions subwiki, you'd simply write the following in a wiki page (you can try it on XWiki Playground):
    {{velocity}}
    #set ($query = $services.query.xwql("where doc.object(ExtensionCode.ExtensionClass).name like '%User%'").setWiki('extensions'))
    #foreach ($itemDoc in $query.execute())
      * [[extensions:$itemDoc]]
    #end
    {{/velocity}}
  • More generally write some applications to enter data easily for your website. It's easy with Applications within Minutes.

Conclusion 

IMO the choice will hugely depend on your needs from the above list but also on how easy/hard it is for you to get some hosting for XWiki:

  • If it's an internal company project, it shouldn't be hard to install and host an XWiki instance (unless your company doesn't have any IT department, see below in this case)
  • If you have some budget you could use XWiki SAS's cloud solution (starts at 10 euros/month for 10 users)
  • If you're an open source project and have no budget, then the choice is a bit harder. The XWiki projet has a free farm and if your project doesn't require professional hosting then it could be a good option. If your project is quite visible/large you could also contact XWiki SAS which has often offered some free professional hosting for open source projects or non-profit associations in the past.

It would be great if more open source forges such as the Apache Software Foundation, the Eclipse Foundation and others were offering XWiki hosting for their projects as an option.

So what would you choose for your project? emoticon_smile

Jul 05 2017

What's new in XWiki @ OW2Conf'17

Got to do a quickie at OW2Conf'17 for 15 minutes about What's New in XWiki. Since I didn't know if the audience already was using XWiki, I did a demo mixing some older but interesting stuff with some new features.

Features demoed:

  • Nested Pages
  • Page Templates
  • Menu Application (bundled now)
  • Color Theme Editor (with live preview)
  • OnlyOffice Extension installed with the Extension Manager
  • Application Within Minutes to create your own custom forms/structures

Jun 28 2017

Voxxed Luxembourg 2017

Last week I've participated to Voxxed Luxembourg. It was a nice event, very well organized, with about 450 participants. Well done PAG!.

I presented a session on Leading a community-driven open source project.

Would love to hear feedback.

Feb 02 2017

Jenkins GitHub Organization Jobs

The Jenkins Pipeline plugin includes a very nice feature which is the "GitHub Organization" job type. This job type will scan a given GitHub organization repositories for Jenkinsfile files and when found will automatically create a pipeline job for them.

This has some nice advantages:

  • You save your Jenkins job configuration in your SCM (git in our case, in the Jenkinsfile), next to your code. You can receive email diffs to show who made modifications to it, the reason and understand the changes.
  • It supports branches: when you create a branch it's automatically discovered by Jenkins and the build is executed on it. And if the branch gets removed, it's removed automatically from Jenkins too. This point is awesome for us since we used to have groovy script to execute to copy jobs when there were new branches and when branches were removed.

So we started exploring this for the XWiki project, starting with Contrib Extensions.

Here's a screenshot of our Github Organization job for XWiki Contrib:

github-organization-contrib.png 

And here's an example of a pipeline job executing:

pipeline.png 

Now if you implement this you'll quickly find that you want to share pipeline scripts between Jenkinsfile, in order to not have duplicates.

FYI here's what the Jenkinsfile for the syntax-markdown pipeline job shown above looks like:

xwikiModule {
    name = 'syntax-markdown'
}

Simple, isn't it? emoticon_smile The trick is that we've configured Jenkins to automatically load a Global Pipeline Library (implicit load). You can do that by saving libraries at the root of SCM repositories and configure Jenkins to load them from the SCM sources (see this Jenkins doc for more details).

So we've created this GitHub repository and we've coded a vars/xwikiModule.groovy file. At the moment of writing this is its content (I expect it to be improved a lot in the near future):

// Example usage:
//   xwikiModule {
//     name = 'application-faq'
//     goals = 'clean install' (default is 'clean deploy')
//     profiles = 'legacy,integration-tests,jetty,hsqldb,firefox' (default is 'quality,legacy,integration-tests')
//  }

def call(body) {
   // evaluate the body block, and collect configuration into the object
   def config = [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

   // Now build, based on the configuration provided, using the followong configuration:
   // - config.name: the name of the module in git, e.g. "syntax-markdown"

    node {
       def mvnHome
       stage('Preparation') {
           // Get the Maven tool.
           // NOTE: Needs to be configured in the global configuration.
           mvnHome = tool 'Maven'
       }
        stage('Build') {
            dir (config.name) {
                checkout scm
               // Execute the XVNC plugin (useful for integration-tests)
               wrap([$class: 'Xvnc']) {
                    withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx1024m']) {
                     try {
                         def goals = config.goals ?: 'clean deploy'
                         def profiles = config.profiles ?: 'quality,legacy,integration-tests'
                          sh "mvn ${goals} jacoco:report -P${profiles} -U -e -Dmaven.test.failure.ignore"
                          currentBuild.result = 'SUCCESS'
                     } catch (Exception err) {
                          currentBuild.result = 'FAILURE'
                          notifyByMail(currentBuild.result)
                         throw e
                     }
                  }
               }
           }
       }
        stage('Post Build') {
           // Archive the generated artifacts
           archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
           // Save the JUnit test report
           junit testResults: '**/target/surefire-reports/TEST-*.xml'
       }
   }
}

def notifyByMail(String buildStatus) {
    buildStatus =  buildStatus ?: 'SUCCESSFUL'
   def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
   def summary = "${subject} (${env.BUILD_URL})"
   def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
    <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""


   def to = emailextrecipients([
           [$class: 'CulpritsRecipientProvider'],
           [$class: 'DevelopersRecipientProvider'],
           [$class: 'RequesterRecipientProvider']
   ])
   if (to != null && !to.isEmpty()) {
        mail to: to, subject: subject, body: details
   }
}

Ideas of some next steps:

Right now there's one limitation I've found: It seems I need to manually click on "Re-scan Organization" in the Jenkins UI so that new Jenkinsfile added to repositories are taken into account. I hope that will get fixed soon. One workaround would be to add another Jenkins job to do that regularly but it's not perfect. Also note that you absolutely must authenticate against GitHub as otherwise you'll quickly reach the GitHub API request limit (when authenticated you are allowed 5000 requests per hour).

Anyway it's great and I love it.

Dec 10 2016

Full Automated Test Coverage with Jenkins and Clover

Generating test coverage reports for a single Maven project is simple. You can use the Clover maven plugin easily for that. For example:

mvn clean clover:setup install clover:clover

Generating a report for several modules in the same Maven reactor (same build) is also easy since that's supported out of the box. For example:

mvn clean clover:setup install clover:aggregate clover:clover

However, generating a full coverage report for a multi-reactor project is much harder. Let's take the example of the XWiki project which has 4 separate Github repositories and thus 4 builds:

So the question is: How do we generate a single test coverage report for those 4 maven reactor builds. For example we want that tests that execute in the xwiki-enterprise repository generate coverage for source code located, say, in xwiki-commons.

Here's what we want to get:

dashboard.png 

The way to do this manually is to tell the Maven Clover plugin to use a single location for generating its data. Manually this can be achieved like this (more details can be found on the XWiki Test page):

# In xwiki-commons:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db
...
# In xwiki-enterprise:
mvn clean clover:setup install -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

# From xwiki-enterprise, generate the full Clover report:
mvn clover:clover -N -Dmaven.clover.cloverDatabase=/path/to/clover/data/clover.db

This is already pretty cool. However it's taking a lot of time and it would be nicer if it could be executed on the CI (on http://ci.xwiki.org in our case).

One important note is that Clover modifies the artifacts and thus you need to be careful to not push them into production or make sure they're not used in other builds (since they'd fail since they'd need to have the Clover runtime JAR at execution time).

So, I chose to use Jenkins 2 and the new Pipeline plugin and used the following script (see the XWiki Clover Job):

node() {
  def mvnHome
  def localRepository
  def cloverDir
  stage('Preparation') {
    def workspace = pwd()
    localRepository = "${workspace}/maven-repository"
    // Make sure that the special Maven local repository for Clover exists
    sh "mkdir -p ${localRepository}"
    // Remove all XWiki artifacts from it
    sh "rm -Rf ${localRepository}/org/xwiki"
    sh "rm -Rf ${localRepository}/com/xpn"
    // Make sure that the directory where clover will store its data exists in
    // the workspace and that it's clean
    cloverDir = "${workspace}/clover-data"
    sh "rm -Rf ${cloverDir}"
    sh "mkdir -p ${cloverDir}"
    // Get the Maven tool.
    // NOTE: Needs to be configured in the global configuration.           
    mvnHome = tool 'Maven'
  }
  // each() has problems in pipeline, thus using a standard for()
  // See https://issues.jenkins-ci.org/browse/JENKINS-26481
  for (String repoName : ["xwiki-commons", "xwiki-rendering", "xwiki-platform", "xwiki-enterprise"]) {
    stage("Cloverify ${repoName}") {
      dir (repoName) {
        git "https://github.com/xwiki/${repoName}.git"
        runCloverAndGenerateReport(mvnHome, localRepository, cloverDir)
      }  
    }      
  }
  stage("Publish Clover Reports") {
    ...
  }
}
def runCloverAndGenerateReport(def mvnHome, def localRepository, def cloverDir) {
  wrap([$class: 'Xvnc']) {
    withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx2048m']) {
      sh "mvn -Dmaven.repo.local='${localRepository}' clean clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true"
      sh "mvn -Dmaven.repo.local='${localRepository}' clover:clover -N -Dmaven.clover.cloverDatabase=${cloverDir}/clover.db"
    }
  }
}

Note that we use the "Xvnc" Jenkins plugin because we run Selenium2 functional tests which require a display.

When this Jenkins job is executed is results in:

pipeline.png 

Over 5 hours of build time... Now you understand why we want to have this running on the CI agent and not on my local machine emoticon_wink

And the generated reports can be seen on xwiki.org.

Good news, we have an overall coverage of 73.2% for the full XWiki Java codebase, that's not too bad (I thought it would be lower emoticon_wink).

Next blog post will be about trying to achieve this with the Jacoco Maven plugin and the associated challenges and issues... Hint: it's harder than with the Clover Maven plugin.

May 03 2016

Bye Bye CLIRR, Welcome Revapi!

On the XWiki project it's been years that we've been checking automatically for backward compatibilities in our APIs as part of our build (It's even documented here).

Recently, we've moved to Java 8 for XWiki 8.1 and we've discovered that the tool we were using, CLIRR, doesn't work anymore with Java 8. So I searched, without much hope, for an alternative, and I was surprised (and delighted) to find that there are 2 valid solutions nowadays that support Java 8:

I've tried both and they both had issues. However, I've discovered that the maintainers or these 2 solutions were really cool guys and they very quickly fixed any issue I've raised. Big thanks to Lukas Krejci and Martin Mois, you're awesome guys! emoticon_smile

In the end the XWiki project has had to make a difficult choice and we chose Revapi (Some reasons mentioned here but honestly they're both valid choices).

So here's how we use it now:

  • In our top level POM:
    ...
         <plugins>
    ...
           <!-- Used for checking backward compatibility (binary and source) -->
           <plugin>
             <groupId>org.revapi</groupId>
             <artifactId>revapi-maven-plugin</artifactId>
             <!-- Lock down plugin version for build reproducibility -->
             <version>0.4.5</version>
             <dependencies>
               <dependency>
                 <groupId>org.revapi</groupId>
                 <artifactId>revapi-java</artifactId>
                 <version>0.9.0</version>
               </dependency>
             </dependencies>
             <executions>
               <execution>
                 <id>revapi-check</id>
                 <goals>
                   <goal>check</goal>
                 </goals>
               </execution>
             </executions>
             <configuration>
               <oldVersion>${xwiki.compatibility.previous.version}</oldVersion>
               <skip>${xwiki.revapi.skip}</skip>
               <analysisConfiguration>
                {
                  "revapi": {
                    "java": {
                      "filter": {
                        "packages": {
                          "regex": true,
                          "include": ["org\\.xwiki\\..*"],
                          "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                        }
                      }
                    }
                  }
                }
               </analysisConfiguration>
             </configuration>
           </plugin>
    ...
  • Then in specific module POMs we override the configuration to add excludes (for backward and source incompatibilities that we know about and have voluntarily decided to do). For example:
    ...
         <plugin>
           <groupId>org.revapi</groupId>
           <artifactId>revapi-maven-plugin</artifactId>
           <configuration>
             <analysisConfiguration><![CDATA[
                {
                  "revapi": {
                    "java": {
                      "filter": {
                        "packages": {
                          "regex": true,
                          "include": ["org\\.xwiki\\..*"],
                          "exclude": ["org\\.xwiki\\..*\\.internal(\\..*)?", "org\\.xwiki\\..*\\.test(\\..*)?"]
                        }
                      }
                    },
                    "ignore": [
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.List<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                        "new": "method java.util.List<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.AbstractExtension::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      },
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                        "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.Extension::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      },
                      {
                        "code": "java.method.returnTypeTypeParametersChanged",
                        "old": "method java.util.Collection<? extends org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                        "new": "method java.util.Collection<org.xwiki.extension.ExtensionDependency> org.xwiki.extension.wrap.WrappingExtension<E extends org.xwiki.extension.Extension>::getDependencies()",
                        "justification": "? return type makes signature more complex for nothing"
                      }         
                    ]
                  }
                }
              ]]>
    </analysisConfiguration>
           </configuration>
         </plugin>
    ...
  • Now the interesting part is in generating reports. We've chosen an original way of doing this: We dynamically generate the Revapi report on our release notes wiki pages using Groovy. The big advantage is that there's no work to be done for the Release Manager:
    • Here's an example from the XWiki 8.1M1 Release Notes

      releasenotes.png

    • The Groovy script below does the following:
      • Use the GitHub REST API to get the content of the pom.xml containing the Revapi excludes.
      • Parse the XML using Groovy
      • Display a report out of it
    • And here's the Groovy script packaged as a Wiki Macro for the curious ones:
      {{groovy}}
      import groovy.json.*

      def getIgnores(def repo, def path)
      {
       def url = "https://api.github.com/repos/xwiki/${repo}/contents/${path}".toURL().text
       def result = new JsonSlurper().parseText(url)
       def xml = new String(result.content.decodeBase64())
        result = new XmlSlurper().parseText(xml)
       def revapi = result.build.plugins.plugin.'**'.find { node ->
          node.artifactId.text() == 'revapi-maven-plugin'
       }
       if (revapi) {
          result = new JsonSlurper().parseText(revapi.configuration.analysisConfiguration.text())
         return result.revapi.ignore
       } else {
         return ''
       }
      }

      def displayIgnores(def ignores)
      {
        result = new JsonSlurper().parseText(ignores)
        result.each() {
          it.each() {
            println "* {{{${it.justification}}}}"
            println "** Violation type: {{code}}${it.code}{{/code}}"
            println "** Old: {{code}}${it.old}{{/code}}"
           if (it.new) {
              println "** New: {{code}}${it.new}{{/code}}"
           }
         }
       }
      }

      def getViolations(def version)
      {
       def xobject = doc.getObject('ReleaseNotes.BackwardCompatibility')
       if (!xobject) {
          xobject = doc.newObject('ReleaseNotes.BackwardCompatibility')
         def commonsTag
         def renderingTag
         def platformTag
         if (version == 'master') {
            commonsTag = renderingTag = platformTag = 'master'
         } else {
            commonsTag = "xwiki-commons-${version}"
            renderingTag = "xwiki-rendering-${version}"
            platformTag = "xwiki-platform-${version}"
         }
         def jsonCommons = getIgnores('xwiki-commons', "xwiki-commons-core/pom.xml?ref=${commonsTag}")
         def jsonRendering = getIgnores('xwiki-rendering', "pom.xml?ref=${renderingTag}")
         def jsonPlatform = getIgnores('xwiki-platform', "xwiki-platform-core/pom.xml?ref=${platformTag}")
          xobject.set('violations', JsonOutput.prettyPrint(JsonOutput.toJson([jsonCommons, jsonRendering, jsonPlatform])))
          doc.save('Added backward-compatiblity violations data', true)
       }
       return xobject.getProperty('violations').value
      }

      displayIgnores(getViolations(xcontext.macro.params.version))
      {{/groovy}}

In conclusion we're very happy with the move, especially since we now have Java 8 support but also because Revapi provides more checks than what CLIRR was doing.
...

Feb 26 2015

Custom JIRA Project Template

JIRA is a pain when you need to create lots of projects with custom Schemes. On the XWiki open source project we allow the community to contribute extensions and we offer a place to host the sources in the xwiki-contrib GitHub organization. We also offer for each project a JIRA project where users can report issues and where developers can track what they're working on. 

Thus we have the need to create lots of JIRA projects. Today, I got bored of repeating over and over the same setup for creating such projects. It involves 9 steps:

  1. Setting the category
  2. Setting the custom XWiki Workflow Scheme
  3. Setting the custom XWiki Screen Scheme
  4. Setting the custom XWiki Field Configuration Scheme
  5. Setting the custom XWiki Notifications Scheme
  6. Setting the custom XWiki Permission Scheme
  7. Setting the Project Lead
  8. Setting the project URL
  9. Setting some groups in one of the project's Roles

Following this tutorial (and a lots of googling emoticon_wink) I was able to automate steps 1-6 in the above list (Source code).

Now when we create a new JIRA project we get this nice template:

jira.png

The main code doing the work is in XWikiAddProjectHook.java:

/*
 * See the NOTICE file distributed with this work for additional
 * information regarding copyright ownership.
 *
 * This is free software; you can redistribute it and/or modify it
 * under the terms of the GNU Lesser General Public License as
 * published by the Free Software Foundation; either version 2.1 of
 * the License, or (at your option) any later version.
 *
 * This software is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this software; if not, write to the Free
 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
 */

package org.xwiki.devtools.jira.template;

import java.util.Arrays;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;

import com.atlassian.jira.bc.projectroles.ProjectRoleService;
import com.atlassian.jira.blueprint.api.AddProjectHook;
import com.atlassian.jira.blueprint.api.ConfigureData;
import com.atlassian.jira.blueprint.api.ConfigureResponse;
import com.atlassian.jira.blueprint.api.ValidateData;
import com.atlassian.jira.blueprint.api.ValidateResponse;
import com.atlassian.jira.component.ComponentAccessor;
import com.atlassian.jira.issue.fields.layout.field.FieldLayoutManager;
import com.atlassian.jira.issue.fields.layout.field.FieldLayoutScheme;
import com.atlassian.jira.issue.fields.screen.issuetype.IssueTypeScreenScheme;
import com.atlassian.jira.issue.fields.screen.issuetype.IssueTypeScreenSchemeManager;
import com.atlassian.jira.notification.NotificationSchemeManager;
import com.atlassian.jira.permission.PermissionSchemeManager;
import com.atlassian.jira.project.Project;
import com.atlassian.jira.project.ProjectCategory;
import com.atlassian.jira.project.ProjectManager;
import com.atlassian.jira.scheme.Scheme;
import com.atlassian.jira.security.roles.ProjectRole;
import com.atlassian.jira.util.SimpleErrorCollection;
import com.atlassian.jira.workflow.WorkflowSchemeManager;

public class XWikiAddProjectHook implements AddProjectHook
{
   private static final Logger LOGGER = LogManager.getLogger(XWikiAddProjectHook.class);

   @Override
   public ValidateResponse validate(final ValidateData validateData)
   {
       return ValidateResponse.create();
   }

   @Override
   public ConfigureResponse configure(final ConfigureData configureData)
   {
        Project project = configureData.project();

       // Set Workflow Scheme
       WorkflowSchemeManager workflowSchemeManager = ComponentAccessor.getWorkflowSchemeManager();
        Scheme xwikiScheme = workflowSchemeManager.getSchemeObject("XWiki Workflow Scheme");
       if (xwikiScheme != null) {
            workflowSchemeManager.removeSchemesFromProject(project);
            workflowSchemeManager.addSchemeToProject(project, xwikiScheme);
       } else {
            LOGGER.warn(String.format("[XWiki] Failed to find the \"XWiki Workflow Scheme\" scheme. "
               + "It is not set for the new project [%s]", project.getName()));
       }

       // Set Notification Scheme
       NotificationSchemeManager notificationSchemeManager = ComponentAccessor.getNotificationSchemeManager();
        Scheme notificationScheme = notificationSchemeManager.getSchemeObject("XWiki Notification Scheme");
       if (notificationScheme != null) {
            notificationSchemeManager.removeSchemesFromProject(project);
            notificationSchemeManager.addSchemeToProject(project, notificationScheme);
       } else {
            LOGGER.warn(String.format("[XWiki] Failed to find the \"XWiki Notification Scheme\" scheme. "
               + "It is not set for the new project [%s]", project.getName()));
       }

       // Set Permission Scheme
       PermissionSchemeManager permissionSchemeManager = ComponentAccessor.getPermissionSchemeManager();
        Scheme permissionScheme = permissionSchemeManager.getSchemeObject("XWiki Open");
       if (permissionScheme != null) {
           // Remove all defined permissions screen since there can be only ony apparently
           permissionSchemeManager.removeSchemesFromProject(project);
            permissionSchemeManager.addSchemeToProject(project, permissionScheme);
       } else {
            LOGGER.warn(String.format("[XWiki] Failed to find the \"XWiki Open\" scheme. "
               + "It is not set for the new project [%s]", project.getName()));
       }

       // Set Project Category
       ProjectManager projectManager = ComponentAccessor.getProjectManager();
        ProjectCategory contribCategory = projectManager.getProjectCategoryObjectByName("XWiki Contributed Projects");
       if (contribCategory != null) {
            projectManager.setProjectCategory(project, contribCategory);
       } else {
            LOGGER.warn(String.format("[XWiki] Failed to find the \"XWiki Contributed Projects\" category. "
               + "It is not set for the new project [%s]", project.getName()));
       }

       // Set Field Configuration Scheme
       FieldLayoutManager fieldLayoutManager = ComponentAccessor.getFieldLayoutManager();
       // Note: I couldn't find a way to get the Field Configuration Scheme by String so I'm iterating over all of them
       FieldLayoutScheme fieldLayoutScheme = null;
       for (FieldLayoutScheme scheme : fieldLayoutManager.getFieldLayoutSchemes()) {
           if (scheme.getName().equals("XWiki Open Field Configuration Scheme")) {
                fieldLayoutScheme = scheme;
               break;
           }
       }
       if (fieldLayoutScheme != null) {
            fieldLayoutManager.addSchemeAssociation(project, fieldLayoutScheme.getId());
       }

       // Set Screen Scheme
       IssueTypeScreenSchemeManager issueTypeScreenSchemeManager = ComponentAccessor.getIssueTypeScreenSchemeManager();
       // Note: I couldn't find a way to get the Screen Scheme by String so I'm iterating over all of them
       IssueTypeScreenScheme screenScheme = null;
       for (IssueTypeScreenScheme scheme : issueTypeScreenSchemeManager.getIssueTypeScreenSchemes()) {
           if (scheme.getName().equals("Basic Issue Creation Scheme")) {
                screenScheme = scheme;
               break;
           }
       }
       if (screenScheme != null) {
            issueTypeScreenSchemeManager.addSchemeAssociation(project, screenScheme);
       } else {
            LOGGER.warn(String.format("[XWiki] Failed to find the \"Basic Issue Creation Scheme\" scheme. "
               + "It is not set for the new project [%s]", project.getName()));
       }

       // Set project committers role to xwiki-committers and contrib-committers
       ProjectRoleService roleService = ComponentAccessor.getComponentOfType(ProjectRoleService.class);
        SimpleErrorCollection errorCollection = new SimpleErrorCollection();
        ProjectRole commmitterRole = roleService.getProjectRoleByName("Committers", errorCollection);
       // Make sure to remove any of the groups we wish to add to the Committers role as otherwise the addition fails
       roleService.removeActorsFromProjectRole(Arrays.asList("xwiki-committers", "contrib-committers"), commmitterRole,
            project, "atlassian-group-role-actor", errorCollection);
        roleService.addActorsToProjectRole(Arrays.asList("xwiki-committers", "contrib-committers"), commmitterRole,
            project, "atlassian-group-role-actor", errorCollection);

       return ConfigureResponse.create().setRedirect("/plugins/servlet/project-config/" + project.getKey()
           + "/summary");
   }
}

Enjoy!

Jan 27 2015

A strategy for maintaining backward compatibility

I'd like to explain what we do on the XWiki open source project to preserve backward compatibility of our Java APIs. We've done several iterations and we now have a process that is working quite well, even though there are still some issues but they're unavoidable with Java for the time being.

So here it goes:

  • We start by having a special package named internal and everything we put in there is considered implementation details and should never be used by users of our APIs. We're allowed to modify anything at any time in the internal package. For example: org.xwiki.instance.internal.*
  • We also use a Component-based approach, which forces us to separate interfaces (which are public) from implementation (which go in the internal package).
  • We have defined an advanced deprecation mechanism which allows us to move deprecated code that we no longer use to legacy maven module (using Aspects with AspectJ) that can be optionally installed at runtime, and which we never use at build time. This prevents us from using any deprecated legacy code and it allows us to push away the cruft under the carpet emoticon_wink (without breaking backward compatibility!)
  • We have the notion of "Young API" (a.k.a Unstable APIs) and when a new API is introduced, the developer can (and should) use the @Unstable annotation to signify to users that this API is unstable for the moment and can change at any time. We only allow an API to remain unstable for a full development cycle. We've recently introduced a custom Checkstyle check that enforces this!
  • Last, we use the Maven CLIRR plugin to make sure we never break an API unintentionally (it fails our build). This allows us to only carefully and intentionally break our APIs. It also allows to us to mention what we break in our Release Notes (example).

The important point IMO is that we have automated tools to ensure that our strategy is applied, namely:

  • CLIRR
  • Our Unstable Annotation Checker (a custom checkstyle rule)

This is working pretty well for us even though we do break backward compatibility from time to time, when we judge that the code touched is unlikely to be used and working around the breakage would be too complex and would take too much time (for example adding a method in an interface requires writing a new interface and modifying all code accepting that interface). Luckily this is going to be made somewhat simpler in Java in the future with Default Methods (introduced in Java 8). It won't fit all cases though.

Another example of backward compatibility aspect that we don't handle is when someone changes what an API returns or what a API does. A simple example is if a method returns an object of type X but the implementation changes to return another implementation of type X that behaves differently... This is a tough one to detect and prevent.

WDYT? Are you doing something else in your projects? (I'd be curious of ways to improve even further what we do!).

Sep 11 2014

Atlassian Summit 2014

I've noticed that Atlassian was holding a Summit yesterday. Apparently it was a success with 2100 attendees. That's great to see and I wanted to congratulate them on that!.

As a developer on the XWiki project what sparked my interest were the announcements related to Confluence. Although I haven't viewed the video yet (it wasn't posted when I checked), I've gathered the following from tweets and blog posts I've noticed:

  • New File Collaboration:

     "You’ll be able to share files through Confluence and collaborate on them in much the same way you already do with pages, using @mentions and in-line comments.

  • Real time collaborative editing:

     And it’s not uncommon for two or more people to need to make updates to the same page at the same time. That’s why we’re bringing real-time collaborative editing to Confluence. (yes!) Timeframe for delivery is a bit nebulous at this point, but the dev team is already plugging away on it

  • Inline comments on highlighted text:

     Oh, and in-line commenting? Yeah, that’s coming to Confluence pages in 5.7, too.

These are all very nice features. Actually they're so nice that the XWiki project has been working on them for some time too, some being there for years and some being more recent. So here's how XWiki would compare on those features:

  • File Collaboration. We have a very nice File Manager allowing users to easily import lots of files inside XWiki, manage them inside the wiki. You can comment on files, tag them, add wiki content related to the files, and more.

    http://extensions.xwiki.org/xwiki/bin/download/Extension/File%20Manager%20Application/HomePage.png

  • Real time collaboration. We have been working on years on real time collaboration in partnership with Academia (INRIA to be precise). We wanted to have realtime for both the Wiki editor and also the WYSIWYG editor. We've tested various solutions and in the end we've found some algorithms that seem to work well. Thus we now have 2 extensions for Real time:
  • Inline comments. XWiki has had support for Annotations for years now (4 years to be precise), allowing users to select text and put an annotation on that text. You can even reply to annotations and they are resilient to reasonable changes from the highlighted content!

    http://extensions.xwiki.org/xwiki/bin/download/Extension/Annotations+Application/HoverAnnotation.png

IMO this shows several things:

  • The open source XWiki project is keeping its innovation rate and is still ahead in term of features
  • A small team can do miracles (the XWiki dev team is much smaller than the Confluence one)

Go XWiki, go! emoticon_smile

Tags:
Created by Vincent Massol on 2008/12/20 13:47
This wiki is licensed under a Creative Commons 2.0 license