Home » Posts tagged 'TypeMock'

# Tag Archives: TypeMock

## Hasty Impressions: NCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my fourth candidate: NCover.

I tried NCover 3.4.18.6937.

## The Cost

NCover Complete is $479 plus$179 for a 1-year subscription (which gives free version updates). I thought this was a little steep. NCover Classic is $199/$99. I looked at NCover Complete, because that’s the kind of trial version they give out. Also, the feature set for Classic was too similar to that offered by other tools that cost less. Check out the feature comparison, if you like.

## Support

I haven’t had enough problems to really stress the support network, but I will say this – the NCover chaps are really keen on keeping in touch with people who have trial copies of the program. I’ve received 3 separate e-mails from my assigned NCover rep in the 2 weeks since I first installed my trial copy. I replied to one of these, asking for a clarification on the VS integration (see below), and got a speedy response.
It’s nice to see such a high level of customer support, but I do feel just a little bit smothered…

## VS integration

The best advice from the NCover folks is to create an external tool to launch NCover. That’s an okay solution if you want to run all the unit tests in a project and profile them, but it lacks flexibility. Then to actually look at the report, you have to launch the NCover Explorer and load the report.

## VS integration

This is where I expected dotCover to shine, and it didn’t disappoint – the integration with Visual Studio (and with ReSharper) was excellent. The first thing I noticed was an extra “Cover with dotCover” item in the ReSharper test menu (triggered from the yellow and green ball things). I clicked it, and it ran my tests, bringing up the familiar Unit Test results window.

Once the tests ran, there was pause while dotCover calculated the coverage info, and then the bottom pane filled in with coverage results: green/red bars by every method in the covered assemblies. Clicking on the methods warps to the source code, which is also highlighted – covered statements have a green background, and uncovered statements have red. In fact, every source file opened in the IDE has the highlighting.

### Finding tests that cover code

The most interesting feature that dotCover has is the ability to identify which tests covered which lines of code. I’m not entirely sold on this, thinking it more of a gimmick than anything else. When I first heard about it, I thought “I don’t care which test covered which line, so long as the lines are covered. I’m here to see what isn’t covered.”. Yes, I think in italics sometimes.

Still, I gave it a go. Right-clicking on a line of code (once coverage has been run) brought up a tiny menu full of covered lines of code. I don’t know why, but it made me happy. I suppose one could use this from time to time to make sure a new test case is exercising what it’s supposed to, but normally I can tell that by how a new test fails, or by what I’ve typed just before the test starts working. Worst case, I could always debug through a single test – something made very easy by the ReSharper test runner.

There was one aspect of this feature that I could imagine someone using – the ability to run the tests that cover a line of code. All that’s needed is to hit the “play” button on the “Show Covering Tests” popup. If the full suite of tests takes a very long time to run, this could be useful. Still, it doesn’t do much for me personally – if my tests took that long to run, I’d try speed them up. If nothing else, I would probably just run the test fixture designed to test the class or method in question, instead of my entire bolus of tests.

So, running tests that cover some code is a cool feature, but it’s not that useful. I’d rather see something like the automatic test runs and really cool “what’s covered” information provided by Mighty-Moose.

## Command Line Execution

Covering an application from the command line is pretty straightforward. I used this command to see what code my BookFinder unit tests exercised:

dotcover cover /TargetExecutable=nunit-console.exe /TargetArguments=.\BookFinder.Tests.dll /Output=dotCoverOutput /Filters=+:BookFinder.Core


BookFinder.Core is the only assembly I was interested in – it holds the business logic. “cover” takes multiple include and exclude filters, even using wildcards for assemblies, classes, and methods.

One quite cool feature is to use the help subcommand to generate an XML configuration file, which can be used to specify the parameters for the cover command:

dotCover help cover coverSettings.xml


will create a coverSettings.xml file that can be edited to specify the executable, arguments, and filters. Then use it like so:

dotCover cover coverSettings.xml


without having to specify the same batch of parameters all the time.

## Joining Coverage Runs

Multiple coverage snapshots – perhaps from running tests on different assemblies, or just from performing different test runs on the same application – can be merged together into a comprehensive snapshot:

dotCover merge /Source snapshot1;snapshot2 /Output mergedsnapshot


Just include all the snapshots, separated by semicolons.

## XML Report

After generating snapshots and optionally merging them, they can be turned into an XML report using the report command:

dotcover report /Source=.\dotCoverOutput /Output=coverageReport.xml

There are options to generate HTML and JSON as well.

Note that if there’s only one snapshot, the “merge” step is not needed. In fact, there’s even a separate analyse command that will cover and generate a report in one go.

## No Auto-Deploy

There’s no auto-deploy for dotCover – it needs to be installed. And since it’s a plugin, Visual Studio is a requirement. This is a small inconvenience for developers and our build servers. Having to put VS on all our test machines is a bit of a bigger deal – definitely a strike against dotCover.

## TypeMock Isolator support in the future

The dotCover 1.1 doesn’t integrate with Isolator 6. Apparently dotCover’s hooks are a little different than many other profiles (nCover, PartCover, …). I’ve been talking to representatives from both TypeMock and JetBrains, though, and they tell me that the problem is solved, and an upcoming release of Isolator will integrate with dotCover. Even better, a pre-release version that supports the latest dotCover EAP is available now.

## IIS

dotCover covers IIS, but only by using the plugin – this means that the web server has to have Visual Studio and dotCover installed, and it’s a manual step to invoke the coverage. In the JetBrains developer community there’s a discussion about command-line IIS support, but no word from JetBrains staff on when this might come.

## Statement-level coverage

As Kevin Jones notes, dotCover reports coverage of statements coverage, not sequence points. This means that a line like this:

return value > 10
? Colors.Red
: Colors.White;


Will report as completely covered, even if it’s executed only once – in order to ensure an accurate coverage report for this idea, the ?: would have to be replaced by an if-else block.
This isn’t necessarily a major strike against the tool, but it’s worth knowing, as it will skew the results somewhat.

## Conclusion

Pros:

• awesome IDE integration
• XML/HTML/JSON reports
• report merging
• IIS profiling

Cons:

• moderate price
• no auto-deploy
• no Isolator support—yet

Overall, I like the tool. I’m a little disappointed by the lack of auto-deploy and the inability to run IIS coverage from the command line, but those problems can be worked around. I was very impressed with the in-IDE support as well as the automatically generated configuration files using the “help” subcommand.
Ordinarily, the I’d say the current lack of Isolator support is a deal-breaker, but I recently demoed the product to some colleagues, and they went bonkers for the IDE integration. I guess I’ll be writing JetBrains and TypeMock looking for the betas.

## Can you cover me? Looking for a .NET coverage tool

Recently at the Day Job, my boss’s boss has been on a “code confidence” kick. We’ve always done various levels of automated and manual unit, integration, issue, system, and regression testing, but he’s looking to improve the process. Part of this push involves getting better at measuring which tests exercise what parts of the code. We want to know this for the usual reasons: we can identify gaps in our testing, or more likely find opportunities to cover some areas earlier in the testing cycle. It’d be nice to know that a particularly critical section of code has been adequately exercised by the per-build unit tests, without having to wait for nightly integration testing or wait even longer for a human to get their grubby mitts on it.

To that end, I’m looking for a .NET coverage tool to dazzle us with tales of test runs. Over the next little while, I’ll look at a few candidates, summarize my findings, and hopefully come up with a winner.

## Considerations

Here are some factors that will influence me. Some of these may be negotiable, if a candidate really shines in other areas.

• We’d like to see coverage information in our build reports, so the tool should run from the command line.
• It’d be easier to put the coverage info our our build reports if the coverage reports were in XML.
• I really prefer a product that has an auto-deploy, so it can be bundled with the source tree and new developers or build servers just work. You may remember the pains I went to to auto-deploy TypeMock Isolator.
• While I’m on the subject, one of our products uses Isolator as its mocking framework, so the coverage tool should be able to link with TypeMock Isolator.
• We have a web services layer, which will be exercised by unit tests, but if we could gather stats on the layer as it’s being exercised by the client-facing portion, that would be gravy. To that end, it should be possible to cover IIS.
• When I used TestDriven.NET + NCover a few years ago, I enjoyed being able to quickly see what my tests covered. This isn’t a requirement of our current initiative, but IDE integration would be a bonus.
• Price is a factor. Money’s available, but why spend if you don’t have to? Or at least, why not pay less for an equivalent product.

## The Candidates

Googling has lead me to these candidates, which I’ll be examining in the next little while:

Update: I picked one.

## Auto-deploying TypeMock Isolator Without Trashing the Installation

At the Day Job, we use TypeMock Isolator as the isolation framework for the client portion of our flagship product. Historically we’d used version 3, but recently I had the opportunity to upgrade the code and build system to use the 2010 (or “version 6”) edition.

#### Backward Compatibility

I was very pleased to see that no code changes were required with the upgrade. Sure, we’d like to start using the new Arrange-Act-Assert API, and to trade in the method name strings for the type-safe lambda expressions, but I didn’t want to have to run back and convert everything today. And I didn’t. Typemock Isolator appears to be backward compatible (at least as far as the feature set we use goes).

#### Auto-Deployment

In fact, the whole exercise of moving up to 2010 would’ve been over in almost no time were it not for one thing—we need to auto-deploy Isolator. The reasons are several:

• we have many dozen people working on the product, spread across four teams and three offices all over the world, so coordinating the installation is tricky
• some people have a need to occasionally build the product, but don’t actively develop it – imposing an install on them seems rude
• some of our developers actively oppose unit testing, and I didn’t want to give them any more ammunition than I had to

We’d had a home-grown auto-deploy solution working with Isolator 3, but it was a little clunky and some of the details of the Isolator install had changed, so it wasn’t really up to auto-deploying 6. Fortunately, I found a Typemock Insider blog post about auto-deploying.

We use Apache Ant for our builds, but it was no trouble to shell out to an MSBuild task to auto-deploy Isolator:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<TypeMockLocation>path\to\TypeMock\Isolator\files</TypeMockLocation>
<NUNIT>path\to\nunit-console.exe</NUNIT>
</PropertyGroup>

<Import Project="$(TypeMockLocation)\TypeMock.MSBuild.Tasks"/> <Target Name="RegisterTypeMock"> <TypeMockRegister Company="MyCompany" License="XXX-XXX" AutoDeploy="true"/> <TypeMockStart/> <Exec ContinueOnError="false" Command="$(NUNIT) $(TestAssembly)"/> <TypeMockStop Undeploy="true"/> </Target> </Project>  #### Build Server Licenses This worked really well – I was testing the tests on my local machine, watching Isolator auto-deploy and auto-undeploy. Everything was great, until I realized: we have two licenses—one for developers, and one for build servers. It only seemed right to use the appropriate one depending on whether we were building on a developer’s machine or a build server. Fortunately, all our build servers set a specific environment variable, so it was a simple matter to have MSBuild pick the correct one. #### Undeploying Isolator Mangles the Installed Instance Even though we’re providing a mechanism for auto-deploying Isolator, some developers will prefer to install it in order to use the Visual Studio AddIn to aid debugging. I’d heard that undoing the auto-deployment could wreak havoc with the installed version of Typemock Isolator, and that it’s sometime necessary to repair the installed instance. A little testing, with the help of a coworker, showed this to be the case. Worse, it appeared that the auto-deploy/undeploy broke his ability to run the product in the IDE – as soon as the process started, it would end, with a “CLR error 80004005”. Disabling the Isolator AddIn made the error go away. So it looked like we’d need to figure out how not to break installed Isolator instances while still supplying auto-deployment when it’s needed. Searching found nothing promising, so I resorted to Registry spelunking. Unfortunately, the installed Isolator and auto-deployed Isolator make very similar Registry entries – there was nothing that I felt confident basing “Is Isolator installed?” on. After poking around and coming up short, I fell back to using the filesystem. By default, Isolator is installed in %ProgramFiles%\TypeMock\Isolator\6.0, so I decided to use that as the determinant. I’d feel dirty doing this for code destined for a customer’s site, but I can live with telling our own developers that if they choose to install Isolator, they should install it in the default location or face the consequences. Still, if anyone comes up with a more reliable way to determine if Isolator is installed, please post it in the comments. #### Putting it all Together Here’s the MSBuild file I ended up with. It uses the correct license based on machine type, and only auto-deploys/undeploys when Isolator isn’t installed – existing installations are left alone. <Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <PropertyGroup> <TypeMockLocation>path\to\TypeMock\Isolator\files</TypeMockLocation> <NUNIT>path\to\nunit-console.exe</NUNIT> <!-- Used to detect TypeMock installs. --> <UsualTypeMockInstallDir>$(ProgramFiles)\TypeMock\Isolator\6.0</UsualTypeMockInstallDir>

<!--
Only deploy Typemock if it's not already in the usual install dir.

If developers install Typemock, they should install it in the
default location in order to help the build system decide
whether or not we need to auto-deploy (since auto-deploy and
undeploy can corrupt the TypeMock VisusalStudio Add-In, and
interfere with the ability to run programs in the IDE.
-->
<DeployTypeMock>false</DeployTypeMock>
<DeployTypeMock Condition="!Exists('$(UsualTypeMockInstallDir)')">true</DeployTypeMock> <License>XXX-XXX</License> <License Condition="'$(BuildServer)' != ''">YYY-YYY</License>
</PropertyGroup>
<Import Project="$(TypeMockLocation)\TypeMock.MSBuild.Tasks"/> <Target Name="RegisterTypeMock"> <TypeMockRegister Company="MyCompany" License="$(License)" AutoDeploy="$(DeployTypeMock)"/> <TypeMockStart/> <Exec ContinueOnError="false" Command="$(NUNIT) $(TestAssembly)" /> <TypeMockStop Undeploy="$(DeployTypeMock)"/>
</Target>
</Project>