Home » Posts tagged 'HastyImpressions'

Tag Archives: HastyImpressions

Hasty Impressions: NCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my fourth candidate: NCover.

I tried NCover 3.4.18.6937.

The Cost

NCover Complete is $479 plus $179 for a 1-year subscription (which gives free version updates). I thought this was a little steep. NCover Classic is $199/$99. I looked at NCover Complete, because that’s the kind of trial version they give out. Also, the feature set for Classic was too similar to that offered by other tools that cost less. Check out the feature comparison, if you like.

Support

I haven’t had enough problems to really stress the support network, but I will say this – the NCover chaps are really keen on keeping in touch with people who have trial copies of the program. I’ve received 3 separate e-mails from my assigned NCover rep in the 2 weeks since I first installed my trial copy. I replied to one of these, asking for a clarification on the VS integration (see below), and got a speedy response.
It’s nice to see such a high level of customer support, but I do feel just a little bit smothered…

VS integration

The best advice from the NCover folks is to create an external tool to launch NCover. That’s an okay solution if you want to run all the unit tests in a project and profile them, but it lacks flexibility. Then to actually look at the report, you have to launch the NCover Explorer and load the report.

There’s additional advice at the end of the Running NCover from Visual Studio video – if you want a more integrated Visual Studio experience, you should obtain TestDriven.Net. That probably works well enough, but I’m not wild about paying an additional $189 per head (roughly) for a test runner that (in my opinion, and excepting the NCover integration of course) is a less robust solution than the one that comes bundled with ReSharper.

Oh. There’s one more feature that I found – once you are examining a coverage report, you can Edit in VS.NET, which opens the appropriate file in Visual Studio. This is somewhat convenient, but doesn’t warp you to the correct line, which is a bit of a letdown.

Command Line Execution

The command line offers many and varied options for configuring the coverage run. Here’s a sample invocation:

NCover.Console.exe //exclude-assemblies BookFinder.Tests //xml ..\..\coverage.nccover nunit-console.exe bin\debug\BookFinder.Tests.dll

Upon execution, NCover tells me this:

Adding the '/noshadow' argument to the NUnit command line to ensure NCover can gather coverage data.
To prevent this behavior, use the //literal argument.

I really like that it defaults to passing the recommended /noshadow to NUnit. The // switches are also a good touch – it makes providing arguments to the executable being covered a lot easier. These features make the command line invocation the best I’ve seen among coverage tools.

GUI Runner

NCover options

The GUI runner looks just like a GUI wrapper on top of the command line options – they appear to support the same level of configuration. After the tests have been run, the NCoverExplorer allows one to browse the results and to save a report as XML or HTML.

XML Report

Reports are generated either from the GUI runner or by using the NCover.Reporting executable, which has a plethora of options for choosing XML or HTML reports of various flavours.
XML reports contain all the information you might want to summarize for inclusion in build output, but they’re hard to understand. Witness:

<stats acp="95" afp="80" abp="95" acc="20" ccavg="1.5" ccmax="5" ex="0" ei="1" ubp="12" ul="40" um="10" usp="39" vbp="63" vl="89" vsp="105" mvc="18" vc="2" vm="22" svc="120">

If you stare at this long enough (and correlate with a matching HTML report), you figure out that this means that there are

  • 39 unvisited sequence points, and
  • 105 visited sequence points

along with various other stats, so using attribute extraction and Math, we could see that 105/144 or 72.9% of the sequence points are covered.

It’s odd that there are many more reports available for HTML than XML. Notably absent from the XML offering: “Summary”. What is it about summaries that make them unsuitable for rendering as XML when HTML is fine?

Reports of Auto-Deploy

My Support Guy explained that you can xcopy deploy NCover using the //reg flag, but I did not find any documentation on how to do this. Support Guy claims there is an “honour system” kind of licensing model that supports this, but the trial copy I had did not work this way. I eventually abandoned this line of investigation.

Mature Isolator Support

From Visual Studio, under the Typemock menu, configure Typemock Isolator to Link with NCover&nsbsp;3.0.
When using the TypeMockStart MSBuild task, use

<TypeMockStart Link="NCover3.0" ProfilerLaunchedFirst="true"/>

and it just works, assuming you have TypeMock Isolator installed or set to auto-deploy.

IIS

IIS coverage is available, simply by selecting it from the GUI runner options or from the command line using the //iis switch. Other Windows Services can be covered in the same manner. Note though, that these features are only available in the Complete flavour of NCover 3.0.

Sequence Point coverage

Supported, as well as branch point coverage and other metrics, including cyclomatic complexity. Nice options to have, although probably a little advanced for my team’s current needs and experience.

Conclusion

Pros:

  • sequence point and branch coverage
  • large feature set, including trends, cyclomatic complexity analysis, and much much more
  • commercial product with strong support
  • report merging
  • easy IIS profiling
  • supports Isolator

Cons:

  • costly
  • weak IDE integration
  • inconsistent (comparing XML to HTML) report offerings
  • confusing auto-deploy

I expected to be blown away by NCover—from all reports, it’s the Cadillac of .NET coverage tools. After demoing it, I figured I’d end up desperately trying to make a case to the Money Guy to shell out hundreds of dollars per developer (and build server), but this did not happen.
While NCover definitely has lots of features, it’s lacking some pretty important ones as well, notably IDE integration. Other features just weren’t as I expected – the cornucopia of report types is impressive, but overkill for a team just starting out, and many of the report types aren’t available in XML and/or are very minor variations on other report types.
Ultimately, I don’t see what NCover offers to justify its price tag, especially across a large team. If ever I felt a need to have one of the specialized report, I’d consider obtaining a single license for tactical use, but I can’t imagine any more than that.

Advertisements

Hasty Impressions: OpenCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my third candidate: OpenCover.
OpenCover is developed by Shaun Wilde. He was a developer on (and is the only remaining maintainer of) PartCover. He’s used what he learned working on PartCover to develop OpenCover, but OpenCover is a new implementation, not a port.

I tried OpenCover 1.0.514. Since I downloaded a couple weeks ago there have been 3 more releases, with the 1.0.606 release promising a big performance improvement.

The Cost

Free! And you can get the source.

VS integration

None that I can find.

Command Line Execution

Covering an application from the command line is easy, and reminiscent of using PartCover the same way. I used this command to see what code my BookFinder unit tests exercised:

OpenCover.Console.exe -arch:64 -register target:nunit-console.exe -targetargs:bin\debug\BookFinder.Tests.dll -output:..\..\opencover.xml -filter:+[BookFinder.Core]*

Let’s look at that.

  • -arch:64 – I’m running on a 64-bit system. I didn’t get any results without this.
  • -register – I’m auto-deploying OpenCover. More on that later.
  • -target:nunit-console.exe – I like NUnit
  • -targetargs:bin\debug\BookFinder.Tests.dll – arguments to NUnit to tell it what assembly to test, and how.
  • -output:..\..\opencover.xml – where to put the coverage results. This file is not a report – it’s intended for machines to read, not humans.
  • -filter:+[BookFinder.Core]* – BookFinder.Core is the only assembly I was interested in – it holds the business logic.

GUI Runner

There isn’t one, but I have to wonder if there won’t be. Otherwise, why call the command line coverer OpenCover.Console.exe?

XML Report

OpenCover doesn’t generate a human-readable report. Instead, you can postprocess the coverage output. ReportGenerator is the recommended tool, and it works like a charm.

ReportGenerator.exe .\opencover.xml XmlReport Xml

generates an XML report in the Xml directory. The summary looks like this:

<?xml version="1.0" encoding="utf-8"?>
<CoverageReport scope="Summary">
  <Summary>
    <Generatedon>2011-08-05-2011-08-05</Generatedon>
    <Parser>OpenCoverParser</Parser>
    <Assemblies>1</Assemblies>
    <Files>5</Files>
    <Coverage>71.6%</Coverage>
    <Coveredlines>126</Coveredlines>
    <Coverablelines>176</Coverablelines>
    <Totallines>495</Totallines>
  </Summary>
  <Assemblies>
    <Assembly name="BookFinder.Core.DLL" coverage="71.6">
      <Class name="BookFinder.BookDepository" coverage="85.7" />
      <Class name="BookFinder.BookListViewModel" coverage="50" />
      <Class name="BookFinder.BoolProperty" coverage="50" />
      <Class name="BookFinder.BoundPropertyStrategy" coverage="0" />
      <Class name="BookFinder.ListProperty" coverage="75" />
      <Class name="BookFinder.Property" coverage="100" />
      <Class name="BookFinder.StringProperty" coverage="100" />
      <Class name="BookFinder.ViewModelBase" coverage="81" />
    </Assembly>
  </Assemblies>
</CoverageReport>

ReportGenerator also generates Html and LaTeX output, with a “summary” variant for each of the three output types.

The XML report would be most useful for inclusion in build result reports, but I found the HTML version easy to use to examine coverage results down to the method level.
HTML Coverage Summary HTML Coverage Detail
I appreciate the coverage count by each of the lines – not as fancy as dotCover’s “which tests cover this”, but it could be a helpful clue when you’re trying to decide what you need to do to improve your coverage.

Joining Coverage Runs

Perhaps your test are scattered in space or time and you want to get an overview of all the code that’s covered by them. OpenCover doesn’t really do anything special for you, but ReportGenerator has your back. Specify multiple input files on the command line, and the results will be aggregated and added to a comprehensive report:

ReportGenerator.exe output1.xml;output2.xml;output3.xml XmlReport Xml

DIY Auto-Deploy

There’s no built-in auto-deploy for OpenCover. However, I made my own auto-deployable package like so:

  1. install OpenCover
  2. copy the C:\Program Files (x86)\OpenCover directory somewhere – call this your package directory
  3. uninstall OpenCover – you won’t need it any more

Then I just made sure my coverage build step

  • knew where the OpenCover package directory was (for the build system at the Day Job, I added it to our “subscribes”)
  • used the -register flag mentioned above to register OpenCover before running the tests

That’s it. No muss, no fuss. I did a similar (but easier, since there’s no registration needed) trick with ReportGenerator, and all of a sudden I have a no-deploy system.

In less than an hour’s work, I could upgrade a project so the build servers and all the developers could run a coverage target, with no action on their part, other than pulling the updated source tree and building. (Which is pretty much what the build server does all day long anyhow…)

DIY (for now) Coverage with Isoloator

Isoloator and OpenCover don’t work together out of the box, but thanks to advice I got from Igal Tabachnik, Typemock employee, it was not hard to change this.

Isolator’s supported coverage tools are partly configurable. There is a typemockconfig.xml under the Isolator install directory – typically %ProgramFiles (x86)%\Typemock\Isoloator\6.0 (or %ProgramFiles%, I suppose). Mr. Tabachnik had me add

 
<Profiler Name="OpenCover" Clsid="{1542C21D-80C3-45E6-A56C-A9C1E4BEB7B8}" DirectLaunch="false">
  <EnvironmentList />
</Profiler>

to the ProfilerList element, and everything meshed. His StackOverflow answer provides full details and suggests that official support for OpenCover will be added to Isolator.

IIS

I can’t find any special IIS support. I’m not saying OpenCover can’t be used to cover an application running in IIS, only that I didn’t find any help for it. I may investigate this later.

Sequence Point coverage

OpenCover counts sequence points, not statements. Yay!

Conclusion

Pros:

  • free
  • open source
  • active project
  • XML/HTML/LaTeX reports (via ReportGenerator)
  • report merging (via ReportGenerator)
  • Isolator support is easy to add (and may be included in future Isolators)
  • auto-deploy package is easy to make

Cons:

  • no IDE integration
  • no help with IIS profiling

I really like OpenCover. It’s easy to use, relatively full-featured, and free. In a work environment, where there’s a tonne of developers who want the in-IDE profiling experience, it may not be the best bet, but I’d use it for my personal .NET projects in a flash.

Hasty Impressions: PartCover

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at my second candidate: PartCover.

Technical stuff

PartCover has a GUI runner as well as a command-line mode. It integrates with Isolator, but doesn’t offer any help for those wanting to profile IIS-hosted applications.
There are some XSL files provided that allow one to generate HTML reports, but probably the better way is to use ReportGenerator to make HTML or XML reports.
PartCover claims to be auto-deployable, but I did not try this.

Project Concerns

The hardest thing about working with PartCover is learning about PartCover – finding definitive information about the project’s state is quite difficult. Searching with Google finds the SourceForge project which contains a note to see latest news on the PartCover blog, which hasn’t been updated since 17 June 2009. Back at SourceForge, you can download a readme written by Shaun Wilde, which says that he’s the last active developer and has moved development to a GitHub project.

At last! A project with recent (26 June 2011) updates. Unfortunately, my trials did not end here. I tried a number of versions, each with their own quirks. Unfortunately, I did not keep as careful track of which version had which problem as I should, and can’t say which version (from either GitHub or SourceForge) had which problems, but I can describe the problems.

At first I thought things were working really well, but then noticed that I had abnormally high coverage levels on my projects – one legacy project that I knew had about 5% coverage was registering as over 20%!
I looked at one assembly’s summary and found 6 classes with 0% coverage and one with 80%, and the assembly was registering an 80%. It turns out that completely uncovered classes were not counting against the total.

I tried other versions, with either the same results, or failures to run altogether. Ultimately, I gave up.

A Successor

It turns out that PartCover has a successor of sorts – Shaun Wilde, the last surviving maintainer of PartCover, has started his own coverage tool – OpenCover. It already seems be a viable PartCover replacement, and is in active development, so I’ll be checking it out as a free, non-IDE-integrated coverage tool.

Conclusion

Pros:

  • free!
  • XML/HTML via ReportGenerator
  • report merging via ReportGenerator
  • Isolator support
  • auto-deployable (reported)
  • sequence point coverage

Cons:

  • no IDE integration
  • no special IIS support
  • forked implementations, each with their own warts
  • not quite abandoned, but not a lot of interest behind the project

Until I noticed the high coverage levels, I didn’t mind PartCover. I figured its lack of price and its Isolator support made it a viable candidate. Unfortunately, the high coverage reports and other problems soured me on the deal, as did the lack of maintenance. I’m going to look at OpenCover instead.

Hasty Impressions: dotCover 1.1

This post is part of a continuing series chronicling my search for a .NET coverage tool.

Today I’m looking at the first candidate: JetBrains dotCover.

I tried dotCover 1.1, integrated with ReSharper 5.1 running in VS2008.

The Cost

A lifetime license, with 1 year of free upgrades is $199 $149 – a special introductory price.

This isn’t usurious, but considering that ReSharper C# edition, a tool that changes the way I work every single day, is $249, it’s enough.

VS integration

cover with dotCover

This is where I expected dotCover to shine, and it didn’t disappoint – the integration with Visual Studio (and with ReSharper) was excellent. The first thing I noticed was an extra “Cover with dotCover” item in the ReSharper test menu (triggered from the yellow and green ball things). I clicked it, and it ran my tests, bringing up the familiar Unit Test results window.

Once the tests ran, there was pause while dotCover calculated the coverage info, and then the bottom pane filled in with coverage results: green/red bars by every method in the covered assemblies. Clicking on the methods warps to the source code, which is also highlighted – covered statements have a green background, and uncovered statements have red. In fact, every source file opened in the IDE has the highlighting.

dotCover BookFinder tests

Finding tests that cover code

The most interesting feature that dotCover has is the ability to identify which tests covered which lines of code. I’m not entirely sold on this, thinking it more of a gimmick than anything else. When I first heard about it, I thought “I don’t care which test covered which line, so long as the lines are covered. I’m here to see what isn’t covered.”. Yes, I think in italics sometimes.

Still, I gave it a go. Right-clicking on a line of code (once coverage has been run) brought up a tiny menu full of covered lines of code. I don’t know why, but it made me happy. I suppose one could use this from time to time to make sure a new test case is exercising what it’s supposed to, but normally I can tell that by how a new test fails, or by what I’ve typed just before the test starts working. Worst case, I could always debug through a single test – something made very easy by the ReSharper test runner.


There was one aspect of this feature that I could imagine someone using – the ability to run the tests that cover a line of code. All that’s needed is to hit the “play” button on the “Show Covering Tests” popup. If the full suite of tests takes a very long time to run, this could be useful. Still, it doesn’t do much for me personally – if my tests took that long to run, I’d try speed them up. If nothing else, I would probably just run the test fixture designed to test the class or method in question, instead of my entire bolus of tests.

So, running tests that cover some code is a cool feature, but it’s not that useful. I’d rather see something like the automatic test runs and really cool “what’s covered” information provided by Mighty-Moose.

Command Line Execution

Covering an application from the command line is pretty straightforward. I used this command to see what code my BookFinder unit tests exercised:

dotcover cover /TargetExecutable=nunit-console.exe /TargetArguments=.\BookFinder.Tests.dll /Output=dotCoverOutput /Filters=+:BookFinder.Core

BookFinder.Core is the only assembly I was interested in – it holds the business logic. “cover” takes multiple include and exclude filters, even using wildcards for assemblies, classes, and methods.

One quite cool feature is to use the help subcommand to generate an XML configuration file, which can be used to specify the parameters for the cover command:

dotCover help cover coverSettings.xml

will create a coverSettings.xml file that can be edited to specify the executable, arguments, and filters. Then use it like so:

dotCover cover coverSettings.xml

without having to specify the same batch of parameters all the time.

Joining Coverage Runs

Multiple coverage snapshots – perhaps from running tests on different assemblies, or just from performing different test runs on the same application – can be merged together into a comprehensive snapshot:

dotCover merge /Source snapshot1;snapshot2 /Output mergedsnapshot

Just include all the snapshots, separated by semicolons.

XML Report

After generating snapshots and optionally merging them, they can be turned into an XML report using the report command:

dotcover report /Source=.\dotCoverOutput /Output=coverageReport.xml

There are options to generate HTML and JSON as well.

Note that if there’s only one snapshot, the “merge” step is not needed. In fact, there’s even a separate analyse command that will cover and generate a report in one go.

No Auto-Deploy

There’s no auto-deploy for dotCover – it needs to be installed. And since it’s a plugin, Visual Studio is a requirement. This is a small inconvenience for developers and our build servers. Having to put VS on all our test machines is a bit of a bigger deal – definitely a strike against dotCover.

TypeMock Isolator support in the future

The dotCover 1.1 doesn’t integrate with Isolator 6. Apparently dotCover’s hooks are a little different than many other profiles (nCover, PartCover, …). I’ve been talking to representatives from both TypeMock and JetBrains, though, and they tell me that the problem is solved, and an upcoming release of Isolator will integrate with dotCover. Even better, a pre-release version that supports the latest dotCover EAP is available now.

IIS

dotCover covers IIS, but only by using the plugin – this means that the web server has to have Visual Studio and dotCover installed, and it’s a manual step to invoke the coverage. In the JetBrains developer community there’s a discussion about command-line IIS support, but no word from JetBrains staff on when this might come.

Statement-level coverage

As Kevin Jones notes, dotCover reports coverage of statements coverage, not sequence points. This means that a line like this:

return value > 10
      ? Colors.Red
      : Colors.White;

Will report as completely covered, even if it’s executed only once – in order to ensure an accurate coverage report for this idea, the ?: would have to be replaced by an if-else block.
This isn’t necessarily a major strike against the tool, but it’s worth knowing, as it will skew the results somewhat.

Conclusion

Pros:

  • awesome IDE integration
  • XML/HTML/JSON reports
  • report merging
  • IIS profiling

Cons:

  • moderate price
  • no auto-deploy
  • no Isolator support—yet

Overall, I like the tool. I’m a little disappointed by the lack of auto-deploy and the inability to run IIS coverage from the command line, but those problems can be worked around. I was very impressed with the in-IDE support as well as the automatically generated configuration files using the “help” subcommand.
Ordinarily, the I’d say the current lack of Isolator support is a deal-breaker, but I recently demoed the product to some colleagues, and they went bonkers for the IDE integration. I guess I’ll be writing JetBrains and TypeMock looking for the betas.

Hasty impressions: Continuous testing using AutoTest.NET

Rinat Abdullin recently posted about Mighty Moose and AutoTest.NET, two projects for continuous testing in the .NET/Mono space. My interest was immediately piqued, as I’m a huge fan of continuous testing. I’ve been using py.test to run my Python unit tests for years now, almost solely because it offers this feature.

I’m taking a look at AutoTest.Net first. Mostly because it’s free. If I’m going to use something at home, it won’t be for-pay, and the Day Job has been notoriously slow at shelling out for developer tools.

Update: there was a bug that had been fixed on trunk, but not in the installer that I used. AutoTest.Net is better at detecting broken builds than I report below.

Setting up AutoTest.NET

Download and installation were straightforward. I opted to use the Windows installer package, AutoTest.Net-v1.0.1beta (Windows Installer).zip. I just unzipped, ran the MSI, let it install both VS 2008 and VS 2010 Add-Ins (the other components are required, it seems), and that was that.

Then I cracked open the configuration file (at c:\Program Files\AutoTest.Net\AutoTest.config). I just changed two entries:

  • BuildExecutable, and
  • NUnitTestRunner

That’s it. Well, for the basic setup.

Running the WinForms monitor

I opened a command prompt to the root of a small project and ran the WinForms monitor, telling it to look for changes in the current directory.

& 'C:\Program Files\AutoTest.Net\AutoTest.WinForms.exe' .

The application started, presenting me with a rather frightening window

I mean, it makes sense. I have neither built nor run yet, so what did I expect? Still, I was taken aback by the plainness of it. Only temporarily daunted, I then hit the tiny unlabelled button in the northeast corner and got a new window. This was less scary.

Everything seemed to be in order. I hadn’t specified MS Test or XUnit runners, nor a code editor. It says it’s watching my files. So let’s test it.

Mucking with the source

It’s supposed to watch my source changes and Do The Right Thing. Let’s see about that.

A benign modification to one test file

I changed the text in one of my test files. No functionality was changed – it was purely cosmetic. AutoTest.Net noticed, rebuilt the solution, and ran the tests! Pretty slick. Things moved quickly, but here’s what I saw from the application:

A benign modification to one “core” file

Next I changed the text in one of the core files – this file is part of a project that’s referenced by the BookFinder GUI project, and the test project. Again, this was a cosmetic change only, just to see what AutoTest.NET would do.
It did what it should – built the three projects and ran the tests. See?

A core change that breaks a test

So, now I’ll modify the core code in a way that breaks a test.
It picks up the change, builds, tests, and does a really nice job of showing me the failure. I see the test that failed, and when I click it, am presented with the stack trace, including hyperlink to the source.

Unfortunately, clicking the hyperlink didn’t go so well:

That was a little disappointing. On the brighter side, hitting “Continue” did continue, with no seeming ill-effects.

Redemption

Confession time. I hadn’t checked the CodeEditor section of the configuration file. As it turns out, it had a slightly different path to my devenv than the correct one. I fixed up the path and tried again. This time, clicking on the hyperlink opened devenv at the right spot.

So the problems was ultimately my fault, but I can’t help but wish for more graceful behaviour – how about a “I couldn’t find your editor” dialogue? Ah, well. The product’s young. Polish will no doubt come.

I repaired the code that broke the tests, and AutoTest.Net was happy again after rebuilding and rerunning the tests.

Syntax Error

For my last test, I decided to actually break the compile. This was kind of disappointing. It claimed to run the 3 builds and the tests, and said that everything passed. I’m not sure why this would be – I was really hoping for an indication that the compilation failed, but nope. Everything was rainbows and puppies. Spurious rainbows and puppies.

The VS Add-In

There’s an add-in. You can activate it under the “Tools” menu. It looks and behaves like the WinForms app.

The Console Monitor

I am used to running py.test in the console, so I thought I’d check out AutoTest’s console monitor next. I started it up, made a benign change, and then made a test-breaking change. Here’s what I saw:

[Info] 'Default' Starting up AutoTester
[Info] 'AutoTest.Console.ConsoleApplication' Starting AutoTest.Net and watching "." and all subdirectories.
[Warn] 'AutoTest.Console.ConsoleApplication' XUnit test runner not specified. XUnit tests will not be run.
[Info] 'AutoTest.Console.ConsoleApplication' Tracker type: file change tracking
[Warn] 'AutoTest.Console.ConsoleApplication' MSTest test runner not specified. MSTest tests will not be run.
[Info] 'AutoTest.Console.ConsoleApplication'
[Info] 'AutoTest.Console.ConsoleApplication' Preparing build(s) and test run(s)
[Info] 'AutoTest.Console.ConsoleApplication' Ran 3 build(s) (3 succeeded, 0 failed) and 2 test(s) (2 passed, 0 failed, 0 ignored)
[Info] 'AutoTest.Console.ConsoleApplication'
[Info] 'AutoTest.Console.ConsoleApplication' Preparing build(s) and test run(s)
[Info] 'AutoTest.Console.ConsoleApplication' Ran 3 build(s) (3 succeeded, 0 failed) and 2 test(s) (1 passed, 1 failed, 0 ignored)
[Info] 'AutoTest.Console.ConsoleApplication' Test(s) failed for assembly BookFinder.Tests.dll
[Info] 'AutoTest.Console.ConsoleApplication'     Failed -> BookFinder.Tests.BookListViewModelTests.FindClick_WithTitleG_FindsEndersGame:
[Info] 'AutoTest.Console.ConsoleApplication'

Not bad, but I have no stack trace for the failed test. Just the name. I’m a little sad to lose functionality relative the WinForms runner. I know I wouldn’t be able to click on source code lines, but still.

Gravy – Hooking up Growl

Undeterred by the disappointing performance in the Syntax Error test, I soldiered on. I use Growl for Windows for notifications, and I was keen to see the integration. I went back to the configuration file and input the growlnotify path. While I was there, I set notify_on_run_started to false (after all, I know when I hit “save”), and notify_on_run_completed to true. Then I fixed my compile error and saved the file.
In addition to the usual changes to the output window, I saw some happy toast:

Honestly, with a GUI or text-based component around, I’m not sure how much benefit this will be, but I guess I can minimize the main window and so long as tests keep passing, I can get some feedback. Still it’s kind of fun.

Impressions

I really like the idea of this tool. I love the idea of watching my code and continuously running the tests. The first steps are very good – I like the clickonable line numbers to locate my errors, and I think the Growl support is cute, but probably more of a toy than an actual useful feature.

Will I Use It?

Not now, and probably never at the Day Job. The inability to detect broken builds is pretty disappointing.
Also, at work, I have ReSharper to integrate my unit tests. I’ve bound “rerun the previous test set” to a key sequence, so it’s just as easy for me to trigger as it is to save a file.

At home? Maybe. If AutoTest.Net starts noticing when builds fail, then I probably will use it when I’m away from ReSharper and working in .NET.