Home » Development » Testing » Hasty Impressions: dotCover 1.1

Hasty Impressions: dotCover 1.1

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at the first candidate: JetBrains dotCover.

I tried dotCover 1.1, integrated with ReSharper 5.1 running in VS2008.

The Cost

A lifetime license, with 1 year of free upgrades is $199 $149 – a special introductory price.

This isn’t usurious, but considering that ReSharper C# edition, a tool that changes the way I work every single day, is $249, it’s enough.

VS integration

cover with dotCover

This is where I expected dotCover to shine, and it didn’t disappoint – the integration with Visual Studio (and with ReSharper) was excellent. The first thing I noticed was an extra “Cover with dotCover” item in the ReSharper test menu (triggered from the yellow and green ball things). I clicked it, and it ran my tests, bringing up the familiar Unit Test results window.

Once the tests ran, there was pause while dotCover calculated the coverage info, and then the bottom pane filled in with coverage results: green/red bars by every method in the covered assemblies. Clicking on the methods warps to the source code, which is also highlighted – covered statements have a green background, and uncovered statements have red. In fact, every source file opened in the IDE has the highlighting.

dotCover BookFinder tests

Finding tests that cover code

The most interesting feature that dotCover has is the ability to identify which tests covered which lines of code. I’m not entirely sold on this, thinking it more of a gimmick than anything else. When I first heard about it, I thought “I don’t care which test covered which line, so long as the lines are covered. I’m here to see what isn’t covered.”. Yes, I think in italics sometimes.

Still, I gave it a go. Right-clicking on a line of code (once coverage has been run) brought up a tiny menu full of covered lines of code. I don’t know why, but it made me happy. I suppose one could use this from time to time to make sure a new test case is exercising what it’s supposed to, but normally I can tell that by how a new test fails, or by what I’ve typed just before the test starts working. Worst case, I could always debug through a single test – something made very easy by the ReSharper test runner.


There was one aspect of this feature that I could imagine someone using – the ability to run the tests that cover a line of code. All that’s needed is to hit the “play” button on the “Show Covering Tests” popup. If the full suite of tests takes a very long time to run, this could be useful. Still, it doesn’t do much for me personally – if my tests took that long to run, I’d try speed them up. If nothing else, I would probably just run the test fixture designed to test the class or method in question, instead of my entire bolus of tests.

So, running tests that cover some code is a cool feature, but it’s not that useful. I’d rather see something like the automatic test runs and really cool “what’s covered” information provided by Mighty-Moose.

Command Line Execution

Covering an application from the command line is pretty straightforward. I used this command to see what code my BookFinder unit tests exercised:

dotcover cover /TargetExecutable=nunit-console.exe /TargetArguments=.\BookFinder.Tests.dll /Output=dotCoverOutput /Filters=+:BookFinder.Core

BookFinder.Core is the only assembly I was interested in – it holds the business logic. “cover” takes multiple include and exclude filters, even using wildcards for assemblies, classes, and methods.

One quite cool feature is to use the help subcommand to generate an XML configuration file, which can be used to specify the parameters for the cover command:

dotCover help cover coverSettings.xml

will create a coverSettings.xml file that can be edited to specify the executable, arguments, and filters. Then use it like so:

dotCover cover coverSettings.xml

without having to specify the same batch of parameters all the time.

Joining Coverage Runs

Multiple coverage snapshots – perhaps from running tests on different assemblies, or just from performing different test runs on the same application – can be merged together into a comprehensive snapshot:

dotCover merge /Source snapshot1;snapshot2 /Output mergedsnapshot

Just include all the snapshots, separated by semicolons.

XML Report

After generating snapshots and optionally merging them, they can be turned into an XML report using the report command:

dotcover report /Source=.\dotCoverOutput /Output=coverageReport.xml

There are options to generate HTML and JSON as well.

Note that if there’s only one snapshot, the “merge” step is not needed. In fact, there’s even a separate analyse command that will cover and generate a report in one go.

No Auto-Deploy

There’s no auto-deploy for dotCover – it needs to be installed. And since it’s a plugin, Visual Studio is a requirement. This is a small inconvenience for developers and our build servers. Having to put VS on all our test machines is a bit of a bigger deal – definitely a strike against dotCover.

TypeMock Isolator support in the future

The dotCover 1.1 doesn’t integrate with Isolator 6. Apparently dotCover’s hooks are a little different than many other profiles (nCover, PartCover, …). I’ve been talking to representatives from both TypeMock and JetBrains, though, and they tell me that the problem is solved, and an upcoming release of Isolator will integrate with dotCover. Even better, a pre-release version that supports the latest dotCover EAP is available now.

IIS

dotCover covers IIS, but only by using the plugin – this means that the web server has to have Visual Studio and dotCover installed, and it’s a manual step to invoke the coverage. In the JetBrains developer community there’s a discussion about command-line IIS support, but no word from JetBrains staff on when this might come.

Statement-level coverage

As Kevin Jones notes, dotCover reports coverage of statements coverage, not sequence points. This means that a line like this:

return value > 10
      ? Colors.Red
      : Colors.White;

Will report as completely covered, even if it’s executed only once – in order to ensure an accurate coverage report for this idea, the ?: would have to be replaced by an if-else block.
This isn’t necessarily a major strike against the tool, but it’s worth knowing, as it will skew the results somewhat.

Conclusion

Pros:

  • awesome IDE integration
  • XML/HTML/JSON reports
  • report merging
  • IIS profiling

Cons:

  • moderate price
  • no auto-deploy
  • no Isolator support—yet

Overall, I like the tool. I’m a little disappointed by the lack of auto-deploy and the inability to run IIS coverage from the command line, but those problems can be worked around. I was very impressed with the in-IDE support as well as the automatically generated configuration files using the “help” subcommand.
Ordinarily, the I’d say the current lack of Isolator support is a deal-breaker, but I recently demoed the product to some colleagues, and they went bonkers for the IDE integration. I guess I’ll be writing JetBrains and TypeMock looking for the betas.

About these ads

3 Comments

  1. Hi Blair,

    Thanks for a comprehensive and fair review of dotCover.
    Although it remains to be seen whether dotCover wins in your coverage tooling challenge, I’ll make sure to read your further posts which will probably help me better understand the competition.
    I thought I’d add a point or two:
    * dotCover coverage engine is bundled with TeamCity (jetbrains.com/teamcity/), including the free professional edition, and doesn’t require Visual Studio installed on the build server. We do sell dotCover as a Visual Studio plug-in but its core used in TeamCity is free.
    * The next major version of dotCover is expected to bundle ReSharper unit test runner, which means that dotCover users will be equally equipped in terms of unit test coverage whether they’re using ReSharper or not.
    * As to the issues with covering ternary expressions, I don’t know if this is something that we can expect to be changed in foreseeable future but regardless, if you want a way to get more accurate coverage results, you can replace ternary conditions with traditional if-else blocks with a pair of ReSharper SSR patterns that goes like this:
    Search pattern:
    return $expression$ ? $true$ : $false$;
    Replace pattern:
    if ($expression$)
    {return $true$;}
    else
    {return $false$;}
    where all placeholders are expressions.

    Good luck with your further reviews of code coverage tools!

  2. Blair Conrad says:

    Hi, Jura.

    Thanks for the note. I appreciate you taking the time to read my post and help out.

    I know almost nothing about TeamCity, but will take a look in the very near future – we’re already using CruiseControl for CI (mostly because we have to support Windows and Unix platforms), and had been considering moving to Jenkins, but if there’s no harm in looking at TeamCity.

    I think it’s great that dotCover will come with ReSharper’s test runner, but don’t expect it to affect me much, since I refuse to give up my ReSharper.

    Good note on the Structural Search and Replace! I hadn’t thought of that. Truth be told I was more concerned about all the notes that ReSharper’s been showing us, saying how we can turn our if/elses into ?:s. Still, it’s worth a try. And just when I was learning to love ??…

  3. Blair Conrad says:

    Regarding my comment about building on multiple platforms. I didn’t mean to suggest that TeamCity doesn’t support that (which is appears to do), but that back when we picked CruiseControl, multiple platform support was a factor. Of course, I doubt there was a TeamCity then…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 34 other followers

%d bloggers like this: