Home » Posts tagged 'OpenCover'
Tag Archives: OpenCover
[Update on 2013-06-22: I should’ve mentioned this a while ago, but the issue and patch I submitted were accepted and built into ReportGenerator 220.127.116.11, so if you have anything newer, you should be good.]
Recently I was working on a project at the Day Job, using OpenCover 18.104.22.168 and ReportGenerator 4.0.804 to report my test coverage, as is my wont, when the report generation started taking figuratively forever. Investigating, I saw something like
found report files: D:/sandbox/project/src/buildlogs/temp_test_coverage/Project.UnitTest.coverage.xml Loading report 'D:\sandbox\project\src\buildlogs\temp_test_coverage\Project.UnitTest.coverage.xml' Preprocessing report Indexing classes in directory 'D:\sandbox\project\src\Module1\SubPath\' Added coverage information of 370/370 auto properties to module 'Module1' Indexing classes in directory 'D:\'
My D: drive isn’t the hugest, but it’s big enough, so that explained the delay. And of course, I certainly didn’t want anything above D:\sandbox\project\src indexed.
I took a peek at my .coverage.xml file and the ReportGenerator code and until I found the offending lines
<Module hash="9A-A3-0A-C0-1D-57-BA-2A-C2-D4-5B-9E-08-DE-BD-2D-46-04-AF-32"> <FullName>D:\Sandbox\project\src\Module\UnitTest\bin\Release\Module.dll</FullName> <ModuleName>Module</ModuleName> <Files> … <File uid="803" fullPath="D:\sandbox\project\src\Module\File1.cs" /> <File uid="806" fullPath="D:\Sandbox\project\src\Module\File2.cs" /> <File uid="808" fullPath="D:\sandbox\project\src\Module\File3.cs" /> …
Note the “Latin capital letter S” at the beginning of “Sandbox” on line 7. All the other lines had a “Latin small letter S”.
When ReportGenerator goes looking for *.cs files to scan, it starts at the directory whose name is the longest common prefix of all the fullPaths. Because “S” isn’t “s”, it came up with “D:\”.
I submitted an issue on the ReportGenerator CodePlex project, so maybe we’ll see a fix soon.
Of course I wondered “Why does the S differ for that entry?” but I figured I’d look at one thing at a time, and locating the fix for ReportGenerator was quicker.
This is the gala awards show, where my chosen coverage tool is announced.
If you’ve come this far, you’ve probably already read the title, and it won’t surprise you to learn that I’ve chosen OpenCover. It offered the best fit for my requirements – the only areas where I found it lacking were in the “nice to haves”. Witness:
- OpenCover is pretty easy to run from the command line – second only to NCover.
- It can (with the help of ReportGenerator) generated coverage reports in XML and HTML.
- OpenCover has an integrated auto-deploy, so it can be bundled with the source tree and new developers or build servers just work – dotCover has no such option, and I was not able to use NCover this way.
- I’ve been able to link with TypeMock Isolator with little trouble, and the new Isolator may obviate the need for my small workaround.
- It’s free. Aside from the obvious benefit, it’s nice to not have to count licenses when adding developers and/or build server nodes.
- There’s no GUI integration, but this was a nice to have. If some developer is absolutely dying to have this, my boss’s boss has indicated that money could be available for individual licenses of something like dotCover.
- There’s no support for integrating with IIS. We don’t need this right now, so that’s okay. Again, if we one or two developers find a need, we have the option of buying a license of some other tool. Even better, support may be coming soon.
After considering OpenCover’s strengths in the areas I absolutely needed, and its weaknesses, which all appear to be in areas that I care a little less about, I recommended it the boss’s boss, who was agreed with the assessment and was happy to keep a little money in his pocket for now.
So, I grabbed 2.0.802, incorporated it into one product’s build, and out popped coverage numbers. Very exciting. I did notice a few things, though:
- Branch coverage has been added since I last evaluated the product!
- One fairly complicated integration-style testfixture is not runnable under OpenCover – the class tested creates a background thread and starting the thread results in a
System.AccessViolationException. I was unable to determine the cause of this, and have temporarily removed the test from coverage, instead executing it with NUnit directly. I’m going to continue investigating this problem.
- Since I’m XCopy deploying, I was bitten by the dependency on the Microsoft Visual C++ 2010 Redistributable Package – I ended up including the DLLs in my imported bundle, and all was well, but I worry a little about the stability of this solution.
- The time taken to execute our tests (there are over 5000, and many hit a database) increased from about 7 minutes to about 8. This is an acceptable degradation, since the test run isn’t the bottleneck in our build process.
- The number of “Cannot instrument as no PDB could be loaded” messages is daunting. I’m hoping that things will be improved once I get a build that contains a fix for issue 40.
Today I’m looking at my second candidate: PartCover.
PartCover has a GUI runner as well as a command-line mode. It integrates with Isolator, but doesn’t offer any help for those wanting to profile IIS-hosted applications.
There are some XSL files provided that allow one to generate HTML reports, but probably the better way is to use ReportGenerator to make HTML or XML reports.
PartCover claims to be auto-deployable, but I did not try this.
The hardest thing about working with PartCover is learning about PartCover – finding definitive information about the project’s state is quite difficult. Searching with Google finds the SourceForge project which contains a note to see latest news on the PartCover blog, which hasn’t been updated since 17 June 2009. Back at SourceForge, you can download a readme written by Shaun Wilde, which says that he’s the last active developer and has moved development to a GitHub project.
At last! A project with recent (26 June 2011) updates. Unfortunately, my trials did not end here. I tried a number of versions, each with their own quirks. Unfortunately, I did not keep as careful track of which version had which problem as I should, and can’t say which version (from either GitHub or SourceForge) had which problems, but I can describe the problems.
At first I thought things were working really well, but then noticed that I had abnormally high coverage levels on my projects – one legacy project that I knew had about 5% coverage was registering as over 20%!
I looked at one assembly’s summary and found 6 classes with 0% coverage and one with 80%, and the assembly was registering an 80%. It turns out that completely uncovered classes were not counting against the total.
I tried other versions, with either the same results, or failures to run altogether. Ultimately, I gave up.
It turns out that PartCover has a successor of sorts – Shaun Wilde, the last surviving maintainer of PartCover, has started his own coverage tool – OpenCover. It already seems be a viable PartCover replacement, and is in active development, so I’ll be checking it out as a free, non-IDE-integrated coverage tool.
- XML/HTML via ReportGenerator
- report merging via ReportGenerator
- Isolator support
- auto-deployable (reported)
- sequence point coverage
- no IDE integration
- no special IIS support
- forked implementations, each with their own warts
- not quite abandoned, but not a lot of interest behind the project
Until I noticed the high coverage levels, I didn’t mind PartCover. I figured its lack of price and its Isolator support made it a viable candidate. Unfortunately, the high coverage reports and other problems soured me on the deal, as did the lack of maintenance. I’m going to look at OpenCover instead.