Consider adopting the Open Test Reporting format

[marcphilipp]

Is your feature request related to a problem?

You've probably come across the "JUnit XML" format for test reporting. This format did not originate from the JUnit project but was initially introduced by the Ant build tool and then adopted by other Java build tools like Maven and Gradle. Many build servers know how to parse the XML-based format, and even non-Java tools often support it. However, it’s based on the concept of test classes and methods, so using it for frameworks and tools where those elements are not present is awkward at best. Moreover, it does not support nested structures beyond a simple parent-child relationship. Finally, it is not extensible: no additional attributes can be added without the risk of breaking existing tools.

For those reasons, many testing frameworks (including Spock) have defined their own reporting formats. This has given them the flexibility they need, but the number of tools that can parse, display, or transform their custom formats is very limited.

Describe the solution you'd like

To overcome these limitations, the JUnit team is defining a new format for test reporting. Its goal is to be platform-agnostic so that as many testing frameworks as possible can benefit from it. Moreover, it is designed to be extensible so new data can be added as needed, without breaking consumers. However, all well-known attributes are properly defined so it remains consumable by downstream reporting tools.

Of course, it will take a while for downstream tools to support the new format. However, as the number of testing frameworks that have adopted it increases, the more likely downstream tools are to do so as well.

The new format is based on XML because it provides more expressive ways to define schemas. Moreover, XML has typed extensions built-in via the use of multiple schemas. If a testing framework provides a listener mechanism, it should be possible to write an Open Test Reporting XML file from an extension.

Benefits

• Easy to write event-based XML format (with optional Java API)
  • Supports infrastructure information, Git metadata, tags, data/file attachments
  • Custom data can be added by defining an extension schema
• Language-agnostic
  • No Java-specific elements in the core schema
  • Not tied to the concept of test classes and methods
  • Full support for nested structures
• Extensible HTML report generator

Describe alternatives you've considered

No response

Additional context

The JUnit team would be happy to get your feedback on this initiative. We can discuss here or you're welcome to start a thread or open an issue in the Open Test Reporting repo. Should you consider adopting the new format, we'd be happy to provide guidance but we won't have the resources to actually contribute an implementation.

[Vampire]

What custom Spock reporting format are you referring to?
From the top of my head I'm not aware of any custom Spock report format.

Spock 1.x was just a JUnit 4 runner, Spock 2.x a JUnit 5 Platform Engine.
The report generation is up to JUnit or the tool driving it.
How / where do we currently provide an own reporting format?

Or are you talking about https://github.com/renatoathaydes/spock-reports?
That is a separate 3rd party Spock extension that provides a custom HTML report and provides means for generating any text-based report through a templating mechanism.

I personally think a new reporting format that is more flexible and can properly report nested results and is not bound to classes and methods is great, as soon as it is widely adopted by according consumers like TeamCity, SonarQube, and so on.
Currently even Gradle is very bound to classes and methods and does not really properly support nested structures.

[marcphilipp]

Or are you talking about renatoathaydes/spock-reports?
That is a separate 3rd party Spock extension that provides a custom HTML report and provides means for generating any text-based report through a templating mechanism.

Yes, that's what I had in mind. Thanks for the clarification!

Currently even Gradle is very bound to classes and methods and does not really properly support nested structures.

I think a Gradle plugin that makes the new reporting format easy-to-use and generates an HTML page from it would be a good start.

[Vampire]

I don't think a Gradle plugin can nicely customize which reports a Test task produces.
Well, a test-engine specific plugin could surely produce another output file and an associate Gradle plugin could add the file as additional output to the Test task.
The Gradle plugin could of course also bring in the dependency for the test-engine specific plugin to the test runtime classpath.

But wouldn't it make more sense if Gradle would be enhanced, so that it can produce such a report for any JUnit Platform Engine?

[marcphilipp]

I don't think a Gradle plugin can nicely customize which reports a Test task produces.

It can set the system properties that enable and configure the report (e.g. junit.platform.reporting.open.xml.enabled=true) on all Test task and define a custom task to generate the reports as a "finalizer" task.

Well, a test-engine specific plugin could surely produce another output file and an associate Gradle plugin could add the file as additional output to the Test task.
The Gradle plugin could of course also bring in the dependency for the test-engine specific plugin to the test runtime classpath.

Why would it have to be test-engine specific if the JUnit Platform writes the report?

But wouldn't it make more sense if Gradle would be enhanced, so that it can produce such a report for any JUnit Platform Engine?

Yes, in the long run but I doubt they'd jump on it at this stage.

[Vampire]

Why would it have to be test-engine specific if the JUnit Platform writes the report?

Hopefully it does not have to be.
But you opened this issue in the Spock engine repository. :-)

[marcphilipp]

Fair enough. I should probably have been more precise in what I was asking for. At this stage, I'm mainly interested in feedback about the format. Is there anything you're missing? As you'll be using this via JUnit Platform, is there anything we should add to JUnit's listener infrastructure so it can get included in the report?

[leonard84]

What custom Spock reporting format are you referring to?
From the top of my head I'm not aware of any custom Spock report format.

There has been an unreleased spock-report module, which supported more Spock-specific information and was based on JSON. It was removed with the move to 2.0 to reduce the maintenance burden.

As you'll be using this via JUnit Platform, is there anything we should add to JUnit's listener infrastructure so it can get included in the report?

Do you have examples of how platform engines will use this new format, what comes out of the box, and what has to done by the engine?

But wouldn't it make more sense if Gradle would be enhanced, so that it can produce such a report for any JUnit Platform Engine?

Yes, in the long run but I doubt they'd jump on it at this stage.

The Gradle HTML report hasn't received much love in a long time, and I wouldn't hold my breath for this to change. If we want to be able to evolve and move quickly, having a pluggable reporter seems to be the better way.

At this stage, I'm mainly interested in feedback about the format

Is there anything in place to model cross-version attributes? So that you could merge reports to generate something like this.

[Vampire]

The Gradle HTML report hasn't received much love in a long time, and I wouldn't hold my breath for this to change. If we want to be able to evolve and move quickly, having a pluggable reporter seems to be the better way.

Imho that is a separate topic, isn't it?
Such a pluggable approach should be provided by Gradle.
And actually it kind of has, you can enable the junit xml report, or with the new JUnit capabilities this new kind of report and then have a task that creates an HTML report out of that.

[marcphilipp]

As you'll be using this via JUnit Platform, is there anything we should add to JUnit's listener infrastructure so it can get included in the report?

Do you have examples of how platform engines will use this new format, what comes out of the box, and what has to done by the engine?

There's a TestExecutionListener in junit-platform-reporting that writes the Open Test Reporting XML file. Currently, the only way engines can contribute is by publishing report and (new in 5.12) file entries to the EngineExecutionListener they get as part of the execution request.

At this stage, I'm mainly interested in feedback about the format

Is there anything in place to model cross-version attributes? So that you could merge reports to generate something like this.

I think the version combination could be modeled as metadata of the execution (the root XML element) and a downstream tool could read that metadata and generate such a table from it. Currently, there's no way for engines to contribute custom metadata but I'm open to discuss how we could achieve that.

[leonard84]

The EngineExecutionListener.reportingEntryPublished is too limited.

We want to be able to publish rich events:

• Engine
  • Configuration values
  • Execution level data, e.g., random seed.
• Spock Metadata Block
  • Spock Tags (link with name), e.g., @Issue (don't confuse with Platform test-tags)
• Specification
  • Narrative
• Feature
  • Repeat narrative + feature + Block texts for full context as a rich text element (markdown)
• Iteration (incl. implicit iteration for non-parameterized features)
  • Block via Block listener
  • File attachments should belong to the block