Define a new XML report schema that supports the JUnit Platform

[sbrannen]

Overview

The XmlReportWriter in the junit-platform-console currently generates XML that is compliant with the de facto standard for JUnit 4; however, this XSD is very limiting and does not support many of the features of the JUnit Platform.

Goals

Define a new XML report schema that supports the features of the JUnit Platform.

The following is only a partial list of features needed beyond the JUnit 4 de facto standard.

• unique IDs
• display names
• tags
• test sources
• custom reporting entries
• nested test classes
• skipped vs. aborted execution

Additional Considerations

• Provide a mechanism for converting between the JUnit 4 XSD and JUnit Platform XSD (e.g., via XSLT).
• Provide a JSON based format compatible with the XML format.

Related Discussions

• Define a standard for test reports ota4j-team/opentest4j#9
• Create rudimentary XML test report #86
• Support UTF-16 in generated XML test reports #350
• XML report uses display names as class names and lacks hierarchy structure #372

[tobymurray]

I've never looked at Allure before (assuming it's this). I'll see what it offers for what I'm looking at - thanks for the tip. I'll update if it comes together with something usable that might be workable for others.

[baev]

I've never looked at Allure before (assuming it's this).

yep

We even have a plugin for XRay https://github.com/allure-framework/allure2/tree/master/plugins/xray-plugin

As a bonus you'll not only get support for JUnit5 tests, but we also provide integrations for majority of test frameworks (not only for Java, but also for Python, JavaScript, C# and others).

Advertising

Allure has paid version https://qameta.io/ that extends open source report and integrations, and it also provides XRay integration.

[adrian-skybaker]

Please consider a clear mechanism to link to attachments. This is supported in NUnit, as well as Allure 2.

[bitcoder]

Unfortunately, this keeps getting postponed :( we have many customers asking for this kind of capability

[marcphilipp]

Any arguments against using JSON instead of XML for the new format?

[bitcoder]

Hi @marcphilipp , we don't see any problems with adopting JSON instead of XML. For us, it will be a new format we have to support and if it's more flexible/easier in the future to make changes in it, ok.
As long as JUnit team provides a kinda of schema for it, it's ok 👍 We've seen some JSON formats out there that don't have a clear definition which in the end leads to some confusion around implementations.

[signed]

Would be nice to have support for #271 in the report as well.

Any arguments against using JSON instead of XML for the new format?

If you think about changing the format did you consider yaml?

[bitcoder]

@signed , yaml is not suited for reporting IMO.

[aslakhellesoy]

I'd like to chime in with our experience defining a new report format in Cucumber a couple of years ago. Cucumber has been able to produce a JSON report for several years, but we're phasing this out and replacing it with a stream of messages, encoded as NDJSON.

Here is a minimal example.

Due to the cross-platform nature of Cucumber, we implemented this using protobuf, but in hindsight that was a mistake, and we're in the process of replacing it with JSON Schema.

The main reason we went for multiple messages instead of one large JSON document is to reduce potentially huge memory consumption. If the report contains several image attachments (common when used in conjunction with browser testing tools), the report can get very big. The lack of streaming support in JSON means all those attachments must be kept in memory until the final document is written. So we went with individual messages instead.

I'm not suggesting JUnit adopt the Cucumber message format as-is. But I do think an NDJSON based message stream format could work well.

You can read more about Cucumber Messages here.

[sbrannen]

@aslakhellesoy, thanks for sharing your insight regarding individual JSON messages. We'll take that into consideration.

[brabenetz]

Any update on this ticket?
Would be nice to have readable test-names for Parameterized Tests in the surfire-reports.
Or does someone know a workaround?

[marcphilipp]

I'm working on it but in the meantime, have you tried configuring Surefire?

[benhumphrey-okta]

If a test report XML output testcase entry looks like this for my JUnitParamsRunner JUnit4 test, Is there any workaround ? I assume configuring statelessTestsetReporter is for Junit5 tests (I attempted to add it to my pom but it seemed to have no effect on a Junit4 test. Our organization has a large number of JUnitParamsRunner tests. Conversion isn't straigtforward (that I'm aware) given thousands of legacy tests.

<testcase name="" classname="com.

Also wanted to confirm this is in the scope of this isse.

[marcphilipp]

If a test report XML output testcase entry looks like this for my JUnitParamsRunner JUnit4 test, Is there any workaround ?

I'm not a Surefire expert. I'm afraid, you'll have to ask on StackOverflow or the Maven users list.

Also wanted to confirm this is in the scope of this isse.

Yes, that should be addressed by the new format as well.

[marcphilipp]

Please see ota4j-team/opentest4j#9 (comment)