Home » 2011 » August

Monthly Archives: August 2011

Hasty Impressions: OpenCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my third candidate: OpenCover.
OpenCover is developed by Shaun Wilde. He was a developer on (and is the only remaining maintainer of) PartCover. He’s used what he learned working on PartCover to develop OpenCover, but OpenCover is a new implementation, not a port.

I tried OpenCover 1.0.514. Since I downloaded a couple weeks ago there have been 3 more releases, with the 1.0.606 release promising a big performance improvement.

The Cost

Free! And you can get the source.

VS integration

None that I can find.

Command Line Execution

Covering an application from the command line is easy, and reminiscent of using PartCover the same way. I used this command to see what code my BookFinder unit tests exercised:

OpenCover.Console.exe -arch:64 -register target:nunit-console.exe -targetargs:bin\debug\BookFinder.Tests.dll -output:..\..\opencover.xml -filter:+[BookFinder.Core]*

Let’s look at that.

  • -arch:64 – I’m running on a 64-bit system. I didn’t get any results without this.
  • -register – I’m auto-deploying OpenCover. More on that later.
  • -target:nunit-console.exe – I like NUnit
  • -targetargs:bin\debug\BookFinder.Tests.dll – arguments to NUnit to tell it what assembly to test, and how.
  • -output:..\..\opencover.xml – where to put the coverage results. This file is not a report – it’s intended for machines to read, not humans.
  • -filter:+[BookFinder.Core]* – BookFinder.Core is the only assembly I was interested in – it holds the business logic.

GUI Runner

There isn’t one, but I have to wonder if there won’t be. Otherwise, why call the command line coverer OpenCover.Console.exe?

XML Report

OpenCover doesn’t generate a human-readable report. Instead, you can postprocess the coverage output. ReportGenerator is the recommended tool, and it works like a charm.

ReportGenerator.exe .\opencover.xml XmlReport Xml

generates an XML report in the Xml directory. The summary looks like this:

<?xml version="1.0" encoding="utf-8"?>
<CoverageReport scope="Summary">
  <Summary>
    <Generatedon>2011-08-05-2011-08-05</Generatedon>
    <Parser>OpenCoverParser</Parser>
    <Assemblies>1</Assemblies>
    <Files>5</Files>
    <Coverage>71.6%</Coverage>
    <Coveredlines>126</Coveredlines>
    <Coverablelines>176</Coverablelines>
    <Totallines>495</Totallines>
  </Summary>
  <Assemblies>
    <Assembly name="BookFinder.Core.DLL" coverage="71.6">
      <Class name="BookFinder.BookDepository" coverage="85.7" />
      <Class name="BookFinder.BookListViewModel" coverage="50" />
      <Class name="BookFinder.BoolProperty" coverage="50" />
      <Class name="BookFinder.BoundPropertyStrategy" coverage="0" />
      <Class name="BookFinder.ListProperty" coverage="75" />
      <Class name="BookFinder.Property" coverage="100" />
      <Class name="BookFinder.StringProperty" coverage="100" />
      <Class name="BookFinder.ViewModelBase" coverage="81" />
    </Assembly>
  </Assemblies>
</CoverageReport>

ReportGenerator also generates Html and LaTeX output, with a “summary” variant for each of the three output types.

The XML report would be most useful for inclusion in build result reports, but I found the HTML version easy to use to examine coverage results down to the method level.
HTML Coverage Summary HTML Coverage Detail
I appreciate the coverage count by each of the lines – not as fancy as dotCover’s “which tests cover this”, but it could be a helpful clue when you’re trying to decide what you need to do to improve your coverage.

Joining Coverage Runs

Perhaps your test are scattered in space or time and you want to get an overview of all the code that’s covered by them. OpenCover doesn’t really do anything special for you, but ReportGenerator has your back. Specify multiple input files on the command line, and the results will be aggregated and added to a comprehensive report:

ReportGenerator.exe output1.xml;output2.xml;output3.xml XmlReport Xml

DIY Auto-Deploy

There’s no built-in auto-deploy for OpenCover. However, I made my own auto-deployable package like so:

  1. install OpenCover
  2. copy the C:\Program Files (x86)\OpenCover directory somewhere – call this your package directory
  3. uninstall OpenCover – you won’t need it any more

Then I just made sure my coverage build step

  • knew where the OpenCover package directory was (for the build system at the Day Job, I added it to our “subscribes”)
  • used the -register flag mentioned above to register OpenCover before running the tests

That’s it. No muss, no fuss. I did a similar (but easier, since there’s no registration needed) trick with ReportGenerator, and all of a sudden I have a no-deploy system.

In less than an hour’s work, I could upgrade a project so the build servers and all the developers could run a coverage target, with no action on their part, other than pulling the updated source tree and building. (Which is pretty much what the build server does all day long anyhow…)

DIY (for now) Coverage with Isoloator

Isoloator and OpenCover don’t work together out of the box, but thanks to advice I got from Igal Tabachnik, Typemock employee, it was not hard to change this.

Isolator’s supported coverage tools are partly configurable. There is a typemockconfig.xml under the Isolator install directory – typically %ProgramFiles (x86)%\Typemock\Isoloator\6.0 (or %ProgramFiles%, I suppose). Mr. Tabachnik had me add

 
<Profiler Name="OpenCover" Clsid="{1542C21D-80C3-45E6-A56C-A9C1E4BEB7B8}" DirectLaunch="false">
  <EnvironmentList />
</Profiler>

to the ProfilerList element, and everything meshed. His StackOverflow answer provides full details and suggests that official support for OpenCover will be added to Isolator.

IIS

I can’t find any special IIS support. I’m not saying OpenCover can’t be used to cover an application running in IIS, only that I didn’t find any help for it. I may investigate this later.

Sequence Point coverage

OpenCover counts sequence points, not statements. Yay!

Conclusion

Pros:

  • free
  • open source
  • active project
  • XML/HTML/LaTeX reports (via ReportGenerator)
  • report merging (via ReportGenerator)
  • Isolator support is easy to add (and may be included in future Isolators)
  • auto-deploy package is easy to make

Cons:

  • no IDE integration
  • no help with IIS profiling

I really like OpenCover. It’s easy to use, relatively full-featured, and free. In a work environment, where there’s a tonne of developers who want the in-IDE profiling experience, it may not be the best bet, but I’d use it for my personal .NET projects in a flash.

Advertisements

Hasty Impressions: PartCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my second candidate: PartCover.

Technical stuff

PartCover has a GUI runner as well as a command-line mode. It integrates with Isolator, but doesn’t offer any help for those wanting to profile IIS-hosted applications.
There are some XSL files provided that allow one to generate HTML reports, but probably the better way is to use ReportGenerator to make HTML or XML reports.
PartCover claims to be auto-deployable, but I did not try this.

Project Concerns

The hardest thing about working with PartCover is learning about PartCover – finding definitive information about the project’s state is quite difficult. Searching with Google finds the SourceForge project which contains a note to see latest news on the PartCover blog, which hasn’t been updated since 17 June 2009. Back at SourceForge, you can download a readme written by Shaun Wilde, which says that he’s the last active developer and has moved development to a GitHub project.

At last! A project with recent (26 June 2011) updates. Unfortunately, my trials did not end here. I tried a number of versions, each with their own quirks. Unfortunately, I did not keep as careful track of which version had which problem as I should, and can’t say which version (from either GitHub or SourceForge) had which problems, but I can describe the problems.

At first I thought things were working really well, but then noticed that I had abnormally high coverage levels on my projects – one legacy project that I knew had about 5% coverage was registering as over 20%!
I looked at one assembly’s summary and found 6 classes with 0% coverage and one with 80%, and the assembly was registering an 80%. It turns out that completely uncovered classes were not counting against the total.

I tried other versions, with either the same results, or failures to run altogether. Ultimately, I gave up.

A Successor

It turns out that PartCover has a successor of sorts – Shaun Wilde, the last surviving maintainer of PartCover, has started his own coverage tool – OpenCover. It already seems be a viable PartCover replacement, and is in active development, so I’ll be checking it out as a free, non-IDE-integrated coverage tool.

Conclusion

Pros:

  • free!
  • XML/HTML via ReportGenerator
  • report merging via ReportGenerator
  • Isolator support
  • auto-deployable (reported)
  • sequence point coverage

Cons:

  • no IDE integration
  • no special IIS support
  • forked implementations, each with their own warts
  • not quite abandoned, but not a lot of interest behind the project

Until I noticed the high coverage levels, I didn’t mind PartCover. I figured its lack of price and its Isolator support made it a viable candidate. Unfortunately, the high coverage reports and other problems soured me on the deal, as did the lack of maintenance. I’m going to look at OpenCover instead.