Custom analyzer silently fails builds, only "csc.exe exited with code 1" is printed

[EikeSchwass]

Analyzer package

Custom analyzer published to private NuGet feed. Works for any open files and with the setting "Enable full solution analysis" also for closed files, as long as the project is not build.

Repro steps

1. Analyzer is installed via nuget package from private artifact nuget feed in Azure DevOps
2. Analyzer works as expected for open files (errors get displayed in the Error List window)
3. Building the project removes the error from the custom analyzer from the Error List and no output is generated (e.g. AnalyzerXYZ reported: "warning xyzxyz" or something similar, build just fails)
4. After the build has failed only "csc.exe exited with code 1" appears in the Error List with no further information or reason.

Expected behavior

The error from the analyzer should be reported in either the output window (especially for CI) or at least in the Error List window - preferable both. The error should not disappear after a build has failed.

Actual behavior

No information/reason for the failed build is given anywhere

[jcouv]

Unfortunately, to investigate this compiler misbehavior, we need a repro that we can step through. Could you share steps to repro this issue? I can share my direct microsoft email address if this involves sensitive details.

If you can repro from the command-line, that is even better. Then you could try adding the /bl (binary log) option to the MSBuild invocation and see if any more details about the failure are logged there.

[jcouv]

Tagging @mavasani, in case he recognizes this as a typical failure mode with analyzers that do certain bad things (in which case I'd recommend debugging your custom analyzer)

[mavasani]

Agreed, we need a repro here to investigate and root cause this.

[EikeSchwass]

@jcouv @mavasani Here is a redacted output log from "msbuild -bl -verbosity:diag". I packed it as a zip because the file is roughly 45 MB,

output.zip

[jcouv]

@EikeStein I see the following in the log. Seems to point to the root of the problem. Is this the problem?

  Microsoft (R) Visual C# Compiler Version 3.4.1-beta4-19610-02 (c4e5d138) (TaskId:695)
  Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten. (TaskId:695)
  Setting\SettingsManagerUnitTests.cs(37,21): error VFD001: The verb 'Shoul' of the expectation 'ShoulThrowException' is not allowed
  Use: Should (TaskId:695)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Roslyn\Microsoft.CSharp.Core.targets(59,5): error MSB6006: "csc.exe" exited with code 1. [C:\Users\eike.stein\source\repos\ProjectRootName\ProjectNameX\BBB.ProjectName.Database.Tests\BBB.ProjectName.Database.Tests.csproj]
  Output Item(s): CscCommandLineArgs= (TaskId:695)
Done executing task "Csc" -- FAILED. (TaskId:695)

[mavasani]

Ah is this the issue where analyzer is reporting a multi-line message? I think I filed a compiler bug on it not handling this scenario gracefully and crashing, but I cannot find that bug anymore.

[EikeSchwass]

@jcouv This is the error message that is generated by the analyzer, but it doesn't show up when building with verbosity set to default. So the error message being multiline is whats causing the problem? If thats the case, that should definitaly be documented somewhere.

[EikeSchwass]

I think this is a duplicate of #1455.

[mavasani]

Thanks @EikeStein - can you please point us to the analyzer NuGet package and/or source code to help us root cause this?

[EikeSchwass]

@mavasani This Analyzer should reproduce the issue:

    [DiagnosticAnalyzer(LanguageNames.CSharp)]
    public class TestAnalyzer : DiagnosticAnalyzer
    {
        private static DiagnosticDescriptor TestRule { get; } = new DiagnosticDescriptor("Test001", "Method naming", "{0}", "Naming", DiagnosticSeverity.Error, true, "This diagnostic will not be shown");

        public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics => ImmutableArray.Create(TestRule);

        public override void Initialize(AnalysisContext context)
        {
            context.EnableConcurrentExecution();
            context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.Analyze | GeneratedCodeAnalysisFlags.ReportDiagnostics);
            context.RegisterSymbolAction(AnalyzeSymbol, SymbolKind.NamedType);
            context.RegisterSymbolAction(AnalyzeSymbol, SymbolKind.Method);
        }

        private static void AnalyzeSymbol(SymbolAnalysisContext context)
        {
            var symbol = context.Symbol;
            if (symbol is IMethodSymbol methodSymbol)
            {
                const string forbiddenMethodName = "FaultyMethod";
                if (methodSymbol.Name.Contains(forbiddenMethodName))
                {
                    var diagnostic = Diagnostic.Create(TestRule, methodSymbol.Locations[0],
                        $"The method '{methodSymbol.Name}' contains forbidden content '{forbiddenMethodName}'\nConsider renaming your method");
                    context.ReportDiagnostic(diagnostic);
                }
            }
        }
    }

The problematic part is the "\n" in the message. If I remove it everything works as expected.

[mavasani]

@EikeStein Awesome. @jcouv - does the compiler team have enough to investigate this issue further? I presume something in the command line compiler error reporting OR parsing is not handling multi-line diagnostic messages. We can either fix the command line compiler here OR detect this case in Diagnostic.Create method and throw a reasonable exception so the analyzer author is aware about this issue?

[jcouv]

@mavasani Yes, I think we have enough information to proceed. I'm not sure where the issue is located though. The build from command-line did output the diagnostic, which makes it look like the analyzer and diagnostic handled the situation okay.
So I wonder if the issue is rather in the displaying in the Error List. I'll try a simple repro and let you know.

[mavasani]

@jcouv - I though @EikeStein initial screenshot shows csc.exe exited with code 1

[jcouv]

Ah, okay. Probably a compiler issue then ;-)

[EikeSchwass]

@jcouv Any updates on this? It is not super urgent, though.

[jcouv]

@EikeStein Sorry, no update yet.