Flaky tests are a plague because the lower the credibility in your CI strategy, by sending false positive emails.
In a previous blog post, I detailed a solution we use on the XWiki project to handle false positives due to the environment on which the CI build is running. However this solution wasn't handling flaky tests. Let's fix this!
So the strategy we're implementing for Flaky tests is the following:
- When a Flaky test is discovered, create a JIRA issue (we currently have the following open issues related to Flaky tests)
- The JIRA issue is marked as containing a flaky test by filling a custom field called "Flickering Test", using the following format: <package name of test class>.<test class name>#<test method name>. There can be several entries separated by commas.
- In our Pipeline script, after the tests have executed, review the failing ones and check if they are in the list of known flaky tests in JIRA. If so, indicate it in the Jenkins test report. If all failing tests are flickers, don't send a notification email.
