The Jenkins Pipeline plugin includes a very nice feature which is the "GitHub Organization" job type. This job type will scan a given GitHub organization repositories for Jenkinsfile files and when found will automatically create a pipeline job for them.
This has some nice advantages:
- You save your Jenkins job configuration in your SCM (git in our case, in the Jenkinsfile), next to your code. You can receive email diffs to show who made modifications to it, the reason and understand the changes.
- It supports branches: when you create a branch it's automatically discovered by Jenkins and the build is executed on it. And if the branch gets removed, it's removed automatically from Jenkins too. This point is awesome for us since we used to have groovy script to execute to copy jobs when there were new branches and when branches were removed.
So we started exploring this for the XWiki project, starting with Contrib Extensions.
Here's a screenshot of our Github Organization job for XWiki Contrib:
And here's an example of a pipeline job executing:
Now if you implement this you'll quickly find that you want to share pipeline scripts between Jenkinsfile, in order to not have duplicates.
FYI here's what the Jenkinsfile for the syntax-markdown pipeline job shown above looks like:
name = 'syntax-markdown'
}
Simple, isn't it? 
The trick is that we've configured Jenkins to automatically load a Global Pipeline Library (implicit load). You can do that by saving libraries at the root of SCM repositories and configure Jenkins to load them from the SCM sources (see this Jenkins doc for more details).
So we've created this GitHub repository and we've coded a vars/xwikiModule.groovy file. At the moment of writing this is its content (I expect it to be improved a lot in the near future):
// xwikiModule {
// name = 'application-faq'
// goals = 'clean install' (default is 'clean deploy')
// profiles = 'legacy,integration-tests,jetty,hsqldb,firefox' (default is 'quality,legacy,integration-tests')
// }
def call(body) {
// evaluate the body block, and collect configuration into the object
def config = [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
body()
// Now build, based on the configuration provided, using the followong configuration:
// - config.name: the name of the module in git, e.g. "syntax-markdown"
node {
def mvnHome
stage('Preparation') {
// Get the Maven tool.
// NOTE: Needs to be configured in the global configuration.
mvnHome = tool 'Maven'
}
stage('Build') {
dir (config.name) {
checkout scm
// Execute the XVNC plugin (useful for integration-tests)
wrap([$class: 'Xvnc']) {
withEnv(["PATH+MAVEN=${mvnHome}/bin", 'MAVEN_OPTS=-Xmx1024m']) {
try {
def goals = config.goals ?: 'clean deploy'
def profiles = config.profiles ?: 'quality,legacy,integration-tests'
sh "mvn ${goals} jacoco:report -P${profiles} -U -e -Dmaven.test.failure.ignore"
currentBuild.result = 'SUCCESS'
} catch (Exception err) {
currentBuild.result = 'FAILURE'
notifyByMail(currentBuild.result)
throw e
}
}
}
}
}
stage('Post Build') {
// Archive the generated artifacts
archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true
// Save the JUnit test report
junit testResults: '**/target/surefire-reports/TEST-*.xml'
}
}
}
def notifyByMail(String buildStatus) {
buildStatus = buildStatus ?: 'SUCCESSFUL'
def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
def summary = "${subject} (${env.BUILD_URL})"
def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
<p>Check console output at "<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>"</p>"""
def to = emailextrecipients([
[$class: 'CulpritsRecipientProvider'],
[$class: 'DevelopersRecipientProvider'],
[$class: 'RequesterRecipientProvider']
])
if (to != null && !to.isEmpty()) {
mail to: to, subject: subject, body: details
}
}
Ideas of some next steps:
- Put back our support for false positives in the pipeline script
- Make xvnc optional with a xvnc = true | false parameter option
Right now there's one limitation I've found: It seems I need to manually click on "Re-scan Organization" in the Jenkins UI so that new Jenkinsfile added to repositories are taken into account. I hope that will get fixed soon. One workaround would be to add another Jenkins job to do that regularly but it's not perfect. Also note that you absolutely must authenticate against GitHub as otherwise you'll quickly reach the GitHub API request limit (when authenticated you are allowed 5000 requests per hour).
Anyway it's great and I love it.
