Having a CI tool set up is great. However if your CI starts sending a lot of build failure emails and a portion of these are false positives (i.e. for example CI infrastructure issues or flickering tests) then what will happen is that the developers are going to trust less and less your CI and builds will stay failing more and more till your release day when you're going to spend several days trying to stabilize your build and your tests before you can release. Thus delaying the release and starting on a downward slope...
It's thus very important that your CI only sends real build failures to the developers.
I've been looking for a way to do with Jenkins and I've found a solution that works (although I would have liked something simpler - If Jenkins experts read this and there's a better please let me know! 
).
So my solution needs 3 Jenkins plugins:
- The Mail Ext plugin
- The Groovy Postbuild plugin
- The Scriptler plugin, for automation.
Without further ado, here's the script that you can run in Scriptler and which will modify the email notification of all your jobs:
import jenkins.model.*
import hudson.plugins.emailext.*
import hudson.plugins.emailext.plugins.trigger.*
import hudson.tasks.*
import hudson.maven.reporters.*
import org.jvnet.hudson.plugins.groovypostbuild.*
def instance = jenkins.model.Jenkins.instance
def jobNames = [
"xwiki-commons.*",
"xwiki-rendering.*",
"xwiki-platform.*",
"xwiki-enterprise.*",
"xwiki-manager.*"
]
def deleteEmailConfiguration(item, publishers)
{
// Remove the Maven Mailer if any since we want to use the Mail Ext plugin exclusively
def reporters = item.reporters
def mavenMailer = reporters.get(MavenMailer)
if (mavenMailer) {
println " - Removing default mailer reporter with recipients [${mavenMailer.recipients}]"
reporters.remove(mavenMailer)
}
// Remove any default Mailer Publisher
def mailer = publishers.get(Mailer)
if (mailer) {
println " - Removing ${Mailer.class.name} publisher"
publishers.remove(Mailer)
}
// Remove any default ExtendedEmailPublisher Publisher
def extMailer = publishers.get(ExtendedEmailPublisher)
if (extMailer) {
println " - Removing ${ExtendedEmailPublisher.class.name} publisher"
publishers.remove(ExtendedEmailPublisher)
}
// Remove any Groovy Postbuild Plugin
def groovyPostbuild = publishers.get(GroovyPostbuildRecorder)
if (groovyPostbuild) {
println " - Removing ${groovyPostbuild.class.name} publisher"
publishers.remove(GroovyPostbuildRecorder)
}
}
instance.items.each { item ->
println "Checking ${item.name}..."
def match = false
jobNames.each {
if (item.name.matches(it)) {
match = true
}
}
if (match) {
if (configureEmailForMatching.equals("true")) {
println " - Modifying ${item.name}..."
def publishers = item.publishersList
deleteEmailConfiguration(item, publishers)
// Add the Groovy Postbuild Plugin
def script = '''
import hudson.model.*
def messages = [
[".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
[".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
[".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
[".*hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
[".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
[".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
[".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"],
[".*com.jcraft.jsch.JSchException: java.net.UnknownHostException: maven.xwiki.org.*", "maven.xwiki.org unavailable", "maven.xwiki.org is not reachable!"],
[".*Fatal Error: Unable to find package java.lang in classpath or bootclasspath.*", "Compilation issue", "Compilation issue!"]
]
def shouldSendEmail = true
messages.each { message ->
if (manager.logContains(message.get(0))) {
manager.addWarningBadge(message.get(1))
manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
manager.buildUnstable()
shouldSendEmail = false
}
}
if (!shouldSendEmail) {
def pa = new ParametersAction([
new BooleanParameterValue("noEmail", true)
])
manager.build.addAction(pa)
}
'''
def gp = new GroovyPostbuildRecorder(script, [], 0)
println(" - Adding new ${gp.class.name} publisher")
publishers.replace(gp)
// Add an Email Ext Publisher and configure it
def ep = new ExtendedEmailPublisher()
ep.defaultSubject = '$DEFAULT_SUBJECT'
ep.defaultContent = '$DEFAULT_CONTENT'
ep.recipientList = "notifications@xwiki.org"
// Add script to not send email for false positives
ep.presendScript = '''
import hudson.model.*
build.actions.each { action ->
if (action instanceof ParametersAction) {
if (action.getParameter("noEmail")) {
cancel = true
}
}
}
'''
ep.configuredTriggers.add(new FailureTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
ep.configuredTriggers.add(new UnstableTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
// We don't want en email for fixed builds now since it means build that have failed because of infra reasons will generate an email when they work again
/*
ep.configuredTriggers.add(new FixedTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
*/
println(" - Adding new ${ep.class.name} publisher")
publishers.replace(ep)
item.save()
} else {
println " - Skipping ${item.name}"
}
} else {
if (removeEmailForNonMatching.equals("true")) {
println " - Removing email configuration for ${item.name}..."
def publishers = item.publishersList
deleteEmailConfiguration(item, publishers)
item.save()
} else {
println " - Skipping ${item.name}"
}
}
}
import hudson.plugins.emailext.*
import hudson.plugins.emailext.plugins.trigger.*
import hudson.tasks.*
import hudson.maven.reporters.*
import org.jvnet.hudson.plugins.groovypostbuild.*
def instance = jenkins.model.Jenkins.instance
def jobNames = [
"xwiki-commons.*",
"xwiki-rendering.*",
"xwiki-platform.*",
"xwiki-enterprise.*",
"xwiki-manager.*"
]
def deleteEmailConfiguration(item, publishers)
{
// Remove the Maven Mailer if any since we want to use the Mail Ext plugin exclusively
def reporters = item.reporters
def mavenMailer = reporters.get(MavenMailer)
if (mavenMailer) {
println " - Removing default mailer reporter with recipients [${mavenMailer.recipients}]"
reporters.remove(mavenMailer)
}
// Remove any default Mailer Publisher
def mailer = publishers.get(Mailer)
if (mailer) {
println " - Removing ${Mailer.class.name} publisher"
publishers.remove(Mailer)
}
// Remove any default ExtendedEmailPublisher Publisher
def extMailer = publishers.get(ExtendedEmailPublisher)
if (extMailer) {
println " - Removing ${ExtendedEmailPublisher.class.name} publisher"
publishers.remove(ExtendedEmailPublisher)
}
// Remove any Groovy Postbuild Plugin
def groovyPostbuild = publishers.get(GroovyPostbuildRecorder)
if (groovyPostbuild) {
println " - Removing ${groovyPostbuild.class.name} publisher"
publishers.remove(GroovyPostbuildRecorder)
}
}
instance.items.each { item ->
println "Checking ${item.name}..."
def match = false
jobNames.each {
if (item.name.matches(it)) {
match = true
}
}
if (match) {
if (configureEmailForMatching.equals("true")) {
println " - Modifying ${item.name}..."
def publishers = item.publishersList
deleteEmailConfiguration(item, publishers)
// Add the Groovy Postbuild Plugin
def script = '''
import hudson.model.*
def messages = [
[".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
[".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
[".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
[".*hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
[".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
[".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
[".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"],
[".*com.jcraft.jsch.JSchException: java.net.UnknownHostException: maven.xwiki.org.*", "maven.xwiki.org unavailable", "maven.xwiki.org is not reachable!"],
[".*Fatal Error: Unable to find package java.lang in classpath or bootclasspath.*", "Compilation issue", "Compilation issue!"]
]
def shouldSendEmail = true
messages.each { message ->
if (manager.logContains(message.get(0))) {
manager.addWarningBadge(message.get(1))
manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
manager.buildUnstable()
shouldSendEmail = false
}
}
if (!shouldSendEmail) {
def pa = new ParametersAction([
new BooleanParameterValue("noEmail", true)
])
manager.build.addAction(pa)
}
'''
def gp = new GroovyPostbuildRecorder(script, [], 0)
println(" - Adding new ${gp.class.name} publisher")
publishers.replace(gp)
// Add an Email Ext Publisher and configure it
def ep = new ExtendedEmailPublisher()
ep.defaultSubject = '$DEFAULT_SUBJECT'
ep.defaultContent = '$DEFAULT_CONTENT'
ep.recipientList = "notifications@xwiki.org"
// Add script to not send email for false positives
ep.presendScript = '''
import hudson.model.*
build.actions.each { action ->
if (action instanceof ParametersAction) {
if (action.getParameter("noEmail")) {
cancel = true
}
}
}
'''
ep.configuredTriggers.add(new FailureTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
ep.configuredTriggers.add(new UnstableTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
// We don't want en email for fixed builds now since it means build that have failed because of infra reasons will generate an email when they work again
/*
ep.configuredTriggers.add(new FixedTrigger(
email : new EmailType(sendToRecipientList : true,
body : ExtendedEmailPublisher.PROJECT_DEFAULT_BODY_TEXT,
subject : ExtendedEmailPublisher.PROJECT_DEFAULT_SUBJECT_TEXT)))
*/
println(" - Adding new ${ep.class.name} publisher")
publishers.replace(ep)
item.save()
} else {
println " - Skipping ${item.name}"
}
} else {
if (removeEmailForNonMatching.equals("true")) {
println " - Removing email configuration for ${item.name}..."
def publishers = item.publishersList
deleteEmailConfiguration(item, publishers)
item.save()
} else {
println " - Skipping ${item.name}"
}
}
}
After you run this script, if you check a job's configuration you'll find a Groovy postbuild step with the following script, which will set up a variable to tell the Email Ext plugin whether to skip sending an email or not:
import hudson.model.*
def messages = [
[".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
[".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
[".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
[".*hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
[".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
[".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
[".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"],
[".*com.jcraft.jsch.JSchException: java.net.UnknownHostException: maven.xwiki.org.*", "maven.xwiki.org unavailable", "maven.xwiki.org is not reachable!"],
[".*Fatal Error: Unable to find package java.lang in classpath or bootclasspath.*", "Compilation issue", "Compilation issue!"]
]
def shouldSendEmail = true
messages.each { message ->
if (manager.logContains(message.get(0))) {
manager.addWarningBadge(message.get(1))
manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
manager.buildUnstable()
shouldSendEmail = false
}
}
if (!shouldSendEmail) {
def pa = new ParametersAction([
new BooleanParameterValue("noEmail", true)
])
manager.build.addAction(pa)
}
def messages = [
[".*A fatal error has been detected by the Java Runtime Environment.*", "JVM Crash", "A JVM crash happened!"],
[".*Error: cannot open display: :1.0.*", "VNC not running", "VNC connection issue!"],
[".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*", "VNC issue", "VNC connection issue!"],
[".*hudson.plugins.git.GitException: Could not fetch from any repository.*", "Git issue", "Git fetching issue!"],
[".*Error communicating with the remote browser. It may have died..*", "Browser issue", "Connection to Browser has died!"],
[".*Failed to start XWiki in .* seconds.*", "XWiki Start", "Failed to start XWiki fast enough!"],
[".*Failed to transfer file.*nexus.*Return code is:.*ReasonPhrase:Service Temporarily Unavailable.", "Nexus down", "Nexus is down!"],
[".*com.jcraft.jsch.JSchException: java.net.UnknownHostException: maven.xwiki.org.*", "maven.xwiki.org unavailable", "maven.xwiki.org is not reachable!"],
[".*Fatal Error: Unable to find package java.lang in classpath or bootclasspath.*", "Compilation issue", "Compilation issue!"]
]
def shouldSendEmail = true
messages.each { message ->
if (manager.logContains(message.get(0))) {
manager.addWarningBadge(message.get(1))
manager.createSummary("warning.gif").appendText("<h1>${message.get(2)}</h1>", false, false, false, "red")
manager.buildUnstable()
shouldSendEmail = false
}
}
if (!shouldSendEmail) {
def pa = new ParametersAction([
new BooleanParameterValue("noEmail", true)
])
manager.build.addAction(pa)
}
And the Email Ext plugin will also have been configured and it will contain a prerun script that verifies if it should send an email or not.
import hudson.model.*
build.actions.each { action ->
if (action instanceof ParametersAction) {
if (action.getParameter("noEmail")) {
cancel = true
}
}
}
build.actions.each { action ->
if (action instanceof ParametersAction) {
if (action.getParameter("noEmail")) {
cancel = true
}
}
}
Of course you should adapt the messages the script is looking for in your build log and adapt it to your cases.
For our need we catch the following:
- Check if the JVM has crashed
- Check if VNC is down (we run Selenium tests)
- Check for Browser crash (we run Selenium tests)
- Check for Git connection issue
- Check for machine slowness (if XWiki cannot start under 2 minutes then it means the machine has some problems)
- Check if Nexus is up (we deploy our artifacts to a Nexus reporitory)
- Check if our CI server has an issue publishing built artifacts to our remote Maven repository
- Check for some strange issue with compilation that happen from time to time with the "Fatal Error: Unable to find package java.lang in classpath or bootclasspath" error message.
Enjoy!
