Test Collection and Reporting

Using the TestCollationService APIs its possible to write your own Tasks that parse files or logs that contain test results and have the results display in Bamboo.

Parsing Test Results

Example test report format

To make this tutorial simple we have developed a file format for a fictional testing framework called My Unit. In MyUnit, each test result is written to a new line and each field is separated by the pipe character (|).

The test result line in the MyUnit format has the following fields:

• Suite - the name of the suite of tests that the test belongs to
• Test Name - the name of the test
• Duration - time the test has taken to run in seconds
• State - SUCCESS or FAIL

Heres what the typical output looks like from MyUnit:

1
2
FirefoxTests|TestThatLoginAppears|2.5|SUCCESS
FirefoxTests|TestUserProfileIsShown|1.1|SUCCESS
FirefoxTests|CanLogout|1.5|FAIL
SafariTests|TestThatLoginAppears|2.5|SUCCESS
SafariTests|TestUserProfileIsShown|1.1|SUCCESS
SafariTests|CanLogout|1.5|FAIL

In this example we assume that MyUnit test files use the .result file extension.

Parsing Test results

TestCollationService supports parsing custom test formats defined by either a custom TestReportCollector (file based) or TestReportProvider (non-file based) implementation. Since MyUnit writes its files out to the file system this example implements the TestReportCollector interface.

File Patterns

The collateTestResults method on TestCollationService accepts Ant-style file glob pattern matching. This makes it possible to select single files or files with certain names that match an expression. For example the pattern /foo/bar.result will only select the file /foo/bar.result relative to the working directory of the server. For selection of multiple .result files in the same directory one could use the pattern /some/sub/directory/*.result.

When a file has been found that matches the configured expression the collect method is called on the TestReportCollector and the results are then added to the build result.

Be aware that if multiple matches are found, the same instance of the TestReportCollector will be used on multiple threads to reduce the parsing time, so it is important that your implementation is stateless.

Files that do not match the extensions in the getSupportedFileExtensions method of the TestReportCollector are automatically excluded from the pattern matching. This is so the user does not specify a pattern such as "/*" (which selects all files recursively in the current directory) and attempt to parse them as test results.

TestReportCollector Example

MyUnitTestReportCollector

1
2
public class MyUnitTestReportCollector implements TestReportCollector
{
    public TestCollectionResult collect(File file) throws Exception
    {
        TestCollectionResultBuilder builder = new TestCollectionResultBuilder();

        Collection<TestResults> successfulTestResults = Lists.newArrayList();
        Collection<TestResults> failingTestResults = Lists.newArrayList();

        List<String> lines = Files.readLines(file, Charset.forName("UTF-8"));

        for (String line : lines)
        {
            String[] atoms = StringUtils.split(line, '|');
            String suiteName = atoms[0];
            String testName = atoms[1];
            String durationInSeconds = atoms[2];
            String status = atoms[3];

            Double duration = (Double.parseDouble(durationInSeconds) * 1000);

            TestResults testResults = new TestResults(suiteName, testName, duration.toString());
            if ("SUCCESS".equals(status))
            {
                testResults.setState(TestState.SUCCESS);
                successfulTestResults.add(testResults);
            }
            else
            {
                testResults.setState(TestState.FAILED);
                failingTestResults.add(testResults);
            }
        }

        return builder
                .addSuccessfulTestResults(successfulTestResults)
                .addFailedTestResults(failingTestResults)
                .build();
    }

    public Set<String> getSupportedFileExtensions()
    {
        return Sets.newHashSet("result"); // this will collect all *.result files
    }
}

Using a TestReportCollector in a Task

All that needs to be done now is to create a new TaskType that injects the TestCollationService in the constructor and have it use the MyUnitTestReportCollector to collate the test results for the user configured file pattern.

Example

MyUnitTestParserTask

1
2
public class MyUnitTestParserTask implements TaskType
{
    private final TestCollationService testCollationService;

    public MyUnitTestParserTask(TestCollationService testCollationService)
    {
        this.testCollationService = testCollationService;
    }

    public TaskResult execute(TaskContext taskContext) throws TaskException
    {
        TaskResultBuilder taskResultBuilder = TaskResultBuilder.create(taskContext);

        final String testFilePattern = taskContext.getConfigurationMap().get("testPattern");

        testCollationService.collateTestResults(taskContext, testFilePattern, new MyUnitTestReportCollector());

        return taskResultBuilder.checkTestFailures().build();
    }
}