First experience using .NET Core console runner

[runehalfdan]

First, thank you for deciding to support running .NET Core tests using console runner. Since .NET Framework has stopped evolving, we have decided to move our code-base to .NET Core. But since our test-projects still must build against .NET Framework 4.8, we build all our dlls as .NET Standard 2.0, which in turn prevents us from using all new .NET Core features. (We have thin entrypoint-assemblies that we build target .NET Core 2.2)

Issues encountered while trying out the new Console Runner (wasn't much information I could find on this):

• .nunit projects files apparently not supported. Not a big issue, but we really like to have a dedicated file listing our 40+ test-assemblies.
• Had to add <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> to test-projects for the console-runner to find dependences.
• Some tests do assembly-scanning, which breaks when running under the new Console Runner. We haven't been able to figure out how to fix this. This works as it should when running tests in Visual Studio

I hope others can fill in some information on the best path to migrate to using the .NET Core console runner.

Have not tested the .NET Core Console Runner on our CI server yet (Bamboo)

[runehalfdan]

Some updates:
In our nunit-file, se set processModel="Multiple" and domainUsage="single" for things to work. (I have no idea why we need this, but our code-base is old and large).

Also since our CI-server, Bamboo, don't support NUnit3 test-result, we need .NET Core Console Runner to support NUnit2 output format.

[ChrisMaddock]

Thanks for trying this @runehalfdan - this is everything I hoped people would do with the beta! Let's break this down...

Had to add true to test-projects for the console-runner to find dependences.

Interesting, this was a problem I thought we'd solved. Do your test-projects have a *.deps.json file? Random stab-in-the-dark, could you try temporarily upgrading your 'thin entry point' to .NET Core 3.1, and see if that helps?

Some tests do assembly-scanning, which breaks when running under the new Console Runner. We haven't been able to figure out how to fix this. This works as it should when running tests in Visual Studio

Breaks in what sort of way, sorry? This sounds like it would need a little more debugging - let us know if you manage to pin it down!

nunit projects files apparently not supported. Not a big issue, but we really like to have a dedicated file listing our 40+ test-assemblies.
Also since our CI-server, Bamboo, don't support NUnit3 test-result, we need .NET Core Console Runner to support NUnit2 output format.

Both the nunit project loader and v2 result writer come from extensions to the engine. The .NET Core Engine should be able to load extensions, but nobody has so far created a .NET Core build of either of those two extensions. Neither extension has an active maintainer at the moment - so someone would need to take on the task of converting those. If that's something you/your company would be interested in - I'd be happy to advise on what things should look like, and I'm sure we can co-ordinate releasing a new NuGet package between us. (@CharliePoole, would you mind?) I don't expect either would be too difficult a task - hopefully just a case of adding a new target and sorting out the packaging.

On the project loader, if your main need is just a file to list your test assemblies in, have you looked at using the @file argument?

In our nunit-file, se set processModel="Multiple" and domainUsage="single" for things to work. (I have no idea why we need this, but our code-base is old and large).

domainUsage will never be available in .NET Core as there's no real concept of App Domains - see the note halfway down this page. The good news is that using these two options together is contradictory anyway - there's no way (that I know of, at least!) of sharing an app domain across multiple processes.

I'm not too sure which option will "win" when you specify both - I suspect that you project actually ends up running with multiple processes and also multiple app domains. Multiple processes is on the roadmap for the .NET Core Engine but there's some pretty chunky obstacles in the way.

You say this is needed to work - what in particular doesn't work without these options?

(wasn't much information I could find on this):

Yeah, sorry. Documentation is another thing on the todo list... 🙂

[runehalfdan]

Thanks for trying this @runehalfdan - this is everything I hoped people would do with the beta! Let's break this down...

Had to add true to test-projects for the console-runner to find dependences.

Interesting, this was a problem I thought we'd solved. Do your test-projects have a *.deps.json file? Random stab-in-the-dark, could you try temporarily upgrading your 'thin entry point' to .NET Core 3.1, and see if that helps?

Cannot easily update to .NET Core 3.1 (too many other dependencies that break), but I'll investigate this a bit more.

Some tests do assembly-scanning, which breaks when running under the new Console Runner. We haven't been able to figure out how to fix this. This works as it should when running tests in Visual Studio

Breaks in what sort of way, sorry? This sounds like it would need a little more debugging - let us know if you manage to pin it down!

After more trials and errors I suspect this is on our side. We have some static initializers that must be run in a specific order, followed by some setup in each test-fixture. This was solved earlier by running each test-assembly isolated. But need more investigation.
But the good news is that when I run tests in assemblies one-by-one, they pass

nunit projects files apparently not supported. Not a big issue, but we really like to have a dedicated file listing our 40+ test-assemblies.
Also since our CI-server, Bamboo, don't support NUnit3 test-result, we need .NET Core Console Runner to support NUnit2 output format.

On the project loader, if your main need is just a file to list your test assemblies in, have you looked at using the @file argument?

Thanks for the @file-file (not sure we I didn't see that before...).
I'll look into converting the extensions when I get some time for external tasks

[runehalfdan]

Both the nunit project loader and v2 result writer come from extensions to the engine. The .NET Core Engine should be able to load extensions, but nobody has so far created a .NET Core build of either of those two extensions. Neither extension has an active maintainer at the moment - so someone would need to take on the task of converting those. If that's something you/your company would be interested in - I'd be happy to advise on what things should look like, and I'm sure we can co-ordinate releasing a new NuGet package between us. (@CharliePoole, would you mind?) I don't expect either would be too difficult a task - hopefully just a case of adding a new target and sorting out the packaging.

I forked v2 result-writer to check things out. Updating project-files and targets was easy, and I can do the same for the project-loader. But I need some guidance on how to enable multi-targeted extensions in the runner.

[ChrisMaddock]

Cool! So from memory, you need to structure the NuGet package like this:

tools    > net20          > nunit-project-loader.dll
         > netcoreapp2.1  > nunit-project-loader.dll
         > .addins

And .addins should be a text file that looks like this:

net20/
netcoreapp2.1/

(The purpose of that file is basically to list all directories that a valid extension can be found in.)

Package that up, install it alongside a NuGet installed NUnit Console, and I think things should just work!

Use the --list-extensions option on the console to check what's installed. If it's not working - it's probably because I'm remembering the logic incorrectly. I'd suggest running with --trace=Debug and seeing what's logged by the Extension service. That combined with the code in the ExtensionService class on the NUnit Engine should not be too complex to track down.

Good luck!

[ChrisMaddock]

ref nunit/docs#506 and nunit/docs#507 to improve these things. 🙂

[ChrisMaddock]

@runehalfdan There's also more info here that might help: #394. Turns out it was nearly two years ago we implemented that part!

[kennethac]

Interesting, this was a problem I thought we'd solved. Do your test-projects have a *.deps.json file? Random stab-in-the-dark, could you try temporarily upgrading your 'thin entry point' to .NET Core 3.1, and see if that helps?

I am still encountering this error with my .NET Core 3.1 application which has a *.deps.json.

[ChrisMaddock]

Great, thanks for confirming, @webman1. By all means we're open to ideas here, if you get a chance to debug further. 🙂

[runehalfdan]

I finally found time to test the updates addins, nunitprojectloader and nunitv2resultwriter.

I had to tweak some addins-configs, but the .NET Core console runner loaded the addins as expected, and both .nunit project file worked, and output for Nunit v2 worked.

One potential issue though; the processModel settings from the .nunit-file was ignored (was set to Multiple, which isn't supported in the .NET Core runner yet)

[CharliePoole]

Regarding multiple process issue, see #745

[ChrisMaddock]

Thanks for the feedback on all this @runehalfdan - glad you've got the extensions up and running too!

I've broken the dependency loading issue out into #828. On the extensions, I believe @CharliePoole has now officially taken over maintaining those projects - I expect he'll be looking at .NET Standard compatible releases at some point. (And appreciate any contributions! 😄)

Thanks for finding these issues - appreciate the help!