Introduction
Testing is one of the most important aspect in (modern) software development, it is like having insurance in our lives, we are not getting insurances because everything will be bad in our life, we get them in case if something goes south, then there is something to helps us survive from the potential damages. The same is with tests, we write them to ensure our application(Code Analyzer), First works as expected Second) later on by refactoring and enhancement of it, we are not breaking the expected behavior, and finally they are specifications of our system, describing the expected behavior.
With all that in mind, let's jump into the bits and bytes of how to test our previously created code analyzer.
Setup the test project
The first thing to do is to setup the project, create a test project in a way you like and with your preferred testing framework, I am using xUnit for as many years as I could remember and in this post will use it as well; however, most of the concepts and classes that will be discussed here have similar equivalents in either of testing frameworks of your choice. I am using Rider.
In order to start testing, it is required to add certain NuGet packages to our project, since I am using xUnit, the NuGet packages are for that testing framework, if you are using NUnit or MSTest then consider the equivalents there! For example, Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.NUnit or Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.MSTest.
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.XUnit" Version="1.1.2" />
</ItemGroup>
PS: I am using version 1.1.2, since some of the classes the we will see later are obsolete from the earlier versions.
It is also required to add a project reference to the project containing the code analyzers we want to test:
<ItemGroup>
<ProjectReference Include="..\Sample.Analyzers\Sample.Analyzers.csproj"/>
</ItemGroup>
Writing the first Test
Now create a class called MaybeSemanticAnalyzerSpecs.cs. The fist test is about checking whenever a throw statement exists in a method that returns some sort of Maybe<T> then we expect the test to check if the Diagnostic API has reported an issue or not, and at the expected location.
Before writing that it is good to learn about a couple of helper classes, one of which is CSharpAnalyzerVerifier, it provides us with factory methods to create DiagnosticResults representing an expected diagnostic for the single DiagnosticDescriptor supported by the provided analyzer:
var expectedDiagnostic = CSharpAnalyzerVerifier<MaybeSemanticAnalyzer, DefaultVerifier>
.Diagnostic(MaybeSemanticAnalyzer.DiagnosticId) // "SHG001"
.WithLocation(8, 9);
The first type parameter is the analyzer under test, in our case the MaybeSemanticAnalyzer from the previous post, and as the second type parameter it accepts a verifier; you might have seen test framework dependent verifiers, like XUnitVerifier which were available from Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.XUnit version 1.1.1 but with version 1.1.2 they are now deprecated and it is better off to use the DefaultVerifier which is obviously the default implementation of the IVerifier interface. The Diagnostic method indicates the ID of the diagnostic to be reported and the WithLocation points to the location in the source file that this diagnostic should have been reported.
The next class is CSharpAnalyzerTest in which we define the test state and all the dependencies that are required, for instance, in our example the Maybe type is in a separate project, so we have to add a project reference to that in the TestState. Speaking of TestState which is of type SolutionState and requires properties that more or less we are familiar with from the first post in this series( or the video ) when creating a compilation using CSharpCompilation.Create. AdditionalReferences property is exactly for the purpose of adding reference to other assemblies; Source property accepts the sources that we want to run our analyzer against them,and finally, we also need to add a reference to the .NET SDK, in our case net9.0, which is possible via ReferenceAssemblies using RefereneAssemblies.Net static class.
var testAnalyzer = new CSharpAnalyzerTest<MaybeSemanticAnalyzer, DefaultVerifier>()
{
TestState =
{
Sources = { code },
ReferenceAssemblies = ReferenceAssemblies.Net.Net90,
AdditionalReferences = { MetadataReference.CreateFromFile(typeof(None).Assembly.Location) }
},
ExpectedDiagnostics = { expectedDiagnostic }
};
In the code above, the last property, ExpectedDiagnostics, as the name suggests, accepts the expected diagnostics for this test analyzer to verify against; at the end, calling RunAsync from testAnalyzer instance ensures that all the expected diagnostics are met for the test to pass and be green.
await testAnalyzer.RunAsync();
The whole Test
[Fact]
public async Task When_MethodWithReturnTypeMaybe_ContainsThrow_Then_ReportDiagnostic()
{
//lang=c#
const string code = """
using System;
using Sample.Fx;
public class Program
{
public Maybe<int> GetValue(string number)
{
throw new InvalidOperationException("Could not parse the number");
}
}
""";
var expectedDiagnostic = CSharpAnalyzerVerifier<MaybeSemanticAnalyzer, DefaultVerifier>
.Diagnostic(MaybeSemanticAnalyzer.DiagnosticId)
.WithLocation(8, 9);
var testAnalyzer = new CSharpAnalyzerTest<MaybeSemanticAnalyzer, DefaultVerifier>()
{
TestState =
{
Sources = { code },
ReferenceAssemblies = ReferenceAssemblies.Net.Net90,
AdditionalReferences = { MetadataReference.CreateFromFile(typeof(None).Assembly.Location) },
},
ExpectedDiagnostics = { expectedDiagnostic }
};
await testAnalyzer.RunAsync();
}
PS: ReferenceAssemblies static class provides other runtime references as well, depending on the scenarios, you might want to use another one.
Another test case would be if the return type of the method is not Maybe<T> and there is a throw statement, then no diagnostics should be reported, I suggest you try it yourself and then check the rest of the blog for the answer 😉
[Fact]
public async Task When_MethodWithReturnTypeNotMaybe_ContainsThrow_Then_ReportNoDiagnostic()
{
//lang=c#
const string code =
"""
using System;
using Sample.Fx;
public class Program
{
public int GetValue(string number)
{
throw new InvalidOperationException("Could not parse the number");
}
}
""";
var testAnalyzer = new CSharpAnalyzerTest<MaybeSemanticAnalyzer, DefaultVerifier>()
{
TestState =
{
Sources = { code },
ReferenceAssemblies = ReferenceAssemblies.Net.Net90,
AdditionalReferences = { MetadataReference.CreateFromFile(typeof(None).Assembly.Location) },
},
};
await testAnalyzer.RunAsync();
}
Conclusion
To write tests for code analysis we need to use certain classes provided by the Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.* which * depends on which testing framework you are using (xUnit, NUnit, or MSTest); Two of those classes are 1) CSharpAnalyzerVerifier<TAnalyzer, TVerifier> providing us with factories for creating the expected DiagnosticResult, if you are using versions before 1.1.2 you could use verifiers like XUnitVerifier but from this version they are obsolete and one could use DefaultVerifier. 2) is CSharpAnalyzerTest which acts as the primary underlying API for testing the analyzers, it requires certain properties to be set, such as Sources, ReferenceAssemblies, AdditionalAssemblies, and ExpectedDiagnostics.
At the end, thanks for reading, enjoy coding and Dametoon Garm [2]
