Origional post (Mon, 13 Dec 2004)
Test coverage is a key testing metric that every project that is serious about testing should be producing. The .net profiler API has opened the door for open source profilers, and either nCover or CoverageEye seems to be the tool of choice, at least until Whidbey ships.
After some initial investigation I decided to use CoverageEye, primarily because we were using nAnt to perform our builds, and changing the build process to explicitly call nCover, and then have nCover call nunit-console for all our tests was going to be a non-trivial task.
Unlike nCover, CoverageEye can be started and stopped independently of the assembly that is being profiled, and some initial tests using the CoverageEye GUI produced the results we were after, so I rolled up my sleeves and implemented CoverageEye into our build process.
The results I got were not what I expected. CoverageEye would work intermittently, sometimes it would work, but most of the time it wouldn’t. A friend and colleague, David Every was following a parallel path, utilising Visual Build and experiencing the same lack of results.
Stuart Richardson, the Microsoft UK SDC based author of the tool was trying to help Dave, but without success. Over the last couple of days, I have finally managed to understand exactly what is going on, and that nCover has to be called the way it is to work around the exact issue that CoverageEye suffers from.
When the .net Common Language Runtime (CLR) spins up, it checks for the existence of a system environment variable; COR_ENABLE_PROFILING. If this environment variable exists and is set to a non-zero value, then the CLR looks at the value of the COR_PROFILER which can be either a CLSID (Class ID) or a ProgID (Program ID) of the profiler that the runtime is to use.
The implication here is that these values need to be set before the CLR instance is started, if profiling is to occur. If any managed application is started before these environment variables are set, then profiling will not occur.
nAnt, Visual Studio.net 2003 and IIS hosting .net sites, all contain or host managed code, so any attempt to start the profiler once these are up and running will fail, as the environment variables were not set when the CLR is started.
nCover solves this issue by setting the environment variables and then spawning a specific process with a new instance of the CLR that has debugging enabled, as shown in this abridged and slightly modified for clarity code from nCover shows.
ProfilerSettings ps;
Process proc=null;
ProcessStartInfo psi;
psi = new ProcessStartInfo(ps.CommandLineExe, ps.CommandLineArgs);
psi.UseShellExecute = false;
psi.ErrorDialog = false;
if (ps.WorkingDirectory != null)
{
psi.WorkingDirectory = ps.WorkingDirectory;
}
psi.EnvironmentVariables["CorEnableProfiling"] = "1";
psi.EnvironmentVariables["CorProfiler"] = "{4d56495b-0799-4ede-898f-7f07637d2dfc}";
proc = Process.Start(psi);In the example above, you can see that nCover explicitly sets the required environment variables on a new process to ensure that the profiler is activated, and this is the reason that nCover has to explicitly execute the application that you want to test, which is the exact situation I was trying to avoid.
A possible solution
So what is the answer? For myself, I set up a command shell (or a batch file), that sets the two environment variables explicitly before nAnt is invoked. This does mean that EVERYTHING will be passed to the profiler, including nAnt itself and then I rely on the CoverageEye configuration to explicitly manage which assemblies are profiled.
On V2 of the CLR, this behaviour appears to have been changed and MSBuild does no suffer the same challenges as a result.
So to summarise, if you want to be able to explicitly profile an assembly, as part of a build process then nCover is probably the way to go, otherwise CoverageEye is a good solution, and you can’t beat the price.
Update (Fri, 7 Jan 2005)
I emailed Stuart Richardson to provide some feedback, as all good users should, and Stuart pointed out that whilst I was on the right track I still didn’t have the whole story …
“The way I think this works is as follows:
Processes get their environments in the usual manner. Some from inheriting them, some have them set in code etc. The environment is a merge of the user / system and application specific settings.
Processes that launch another process, like say cmd.exe, will launch a new process with a copy of their environment variables.
So if you launch a copy of cmd.exe and then add a system environment variable, and then type SET you will see it is not present.
If you launch a copy of explorer (the shell) and launch a copy of cmd.exe and type SET – add the system environment variable and do the same again launching a new copy of cmd.exe, you will see it is present. This is because explorer responds to the broadcast windows message that tells all listening processes the environment has changed, and it gives the processes the chance to update them.
When the coverageeye UI sets these environment variables is sends the broadcast message, and any processes that are written to respond to these changes in environment variables will pick them up.
So I believe that the issue you face is actually that the process that starts your tests does not update itself when this message is broadcast.
This is also not the entire story. Windows services – including ASP.NET apps and normal windows services get there environment set as machine start up. You will never be able to influence these environment variables. CoverageEye does have a solution to this and writes a custom environment for any configured services.
If you try getting coverage manually setting these environment variables for web services you will find that you do not get any results.”
Thanks Stuart !
Update (Sat, 19 March 2005)
Chris Burrows has posted an update to the code coverage challenges we have been having.
References
GotDotNet User Sample: CoverageEye.NET 2.0
CoverageEye.NET: Releases: Home
nCover Source Code
Alan Dean’s Blog post – [Tool] NCover or CoverageEye
Mark Levison’s Blog post – CoverageEye.Net tested and dropped
Microsoft Patterns and Practices – How To: Use CLR Profiler
Matt Pietrek’s Blog post – Cool Whidbey debugger feature: Merge Environment
Matt Pietrek’s MSDN Article – The .NET Profiling API and the DNProfiler Tool
.NET Framework Developer’s Guide – Enabling Profiling
