Better formatter auto-discovery in FakeItEasy 1.13.0

A few weeks ago, I wrote about the problems that FakeItEasy’s assembly scanning was causing while it was looking for user-defined extensions. To recap, FakeItEasy was scanning all assemblies in the AppDomain and the working directory, looking for types that implemented IArgumentValueFormatter, IDummyDefinition, or IFakeConfigurator. This process was quite slow. Worse, it raised LoaderLock exceptions when debugging, and Runtime errors anytime I ran my tests using the ReSharper test runner.

At that time, I’d opened issue 130, intended to allow configuration of the scanning procedure. I’m happy to say that the issue has been closed “no fix”. Instead, I’ve contributed the fix for Issue 133 — Improved performance of assembly scanning. It doesn’t introduce any configuration options, but streamlines the scanning process.

The original behaviour was:

  1. find all the DLLs in the application directory
  2. load all the found DLLs
  3. find the distinct assemblies among those loaded from the directory and those already in the AppDomain
  4. scan each assembly and add all the types to a list

The new behaviour, heavily inspired by Nancy‘s bootstrapper-finding code, is:

  1. find all the DLLs in the application directory
  2. discard DLLs that are already part of the AppDomain – We don’t even have to crack these files open again, since we already know everything about them. Note that this check examines the absolute paths to the DLL and the loaded assembly, and will be fooled by shadow copying. So, if your test runner makes shadow copies, this time won’t be saved. I turned off shadow copying with no ill effects (and a tremendous speedup), but your mileage may vary.
  3. load each remaining DLL for reflection only – This may be faster, and it may not, but it has another big advantage – it doesn’t cause any of the code in the assembly to execute. (It was the execution of the assembly code that caused my LoaderLock and Runtime errors.)
  4. for each assembly that references FakeItEasy, fully load it – If we don’t do this, we can’t scan for all the types in the assembly because
    When using the ReflectionOnly APIs, dependent assemblies must be pre-loaded or loaded on demand through the ReflectionOnlyAssemblyResolve event.

    according to the error I got when I tried it. Note that excluding assemblies that don’t reference FakeItEasy means we only examine assemblies that could possibly define formatting/dummy/configuration extensions, cutting down on the scanning time.

  5. scan each of the following, remembering all contained types:
    • the assemblies we just loaded from files,
    • the AppDomain assemblies that reference FakeItEasy, and
    • FakeItEasy – We need to include FakeItEasy explicitly because it defines its own formatter extensions, and since we’re otherwise only looking at assemblies that reference FakeItEasy, we’d miss it.

This new scanning behaviour has been released in the FakeItEasy 1.13.0 build, and has been a boon to me already. I’m enjoying the faster test runs (0.534 seconds for my first test, versus 1.822 (or more)) and the improved stability of the test runner. NuGet it now.

Watch your spaces – HTTP Error 500.19 – Internal Server Error

Late last week at the Day Job, a colleague came to me with a problem. The web service he was trying to hit was throwing an error he’d never seen before:

HTTP Error 500.19 – Internal Server Error
The requested page cannot be accessed because the related configuration data for the page is invalid.

I’d never seen it before either, at least not in this exact incarnation. Take a look

screenshot of 500.19 error

In case the text isn’t so clear, here are the details:

Module IpRestrictionModule
Notification BeginRequest
Handler WebServiceHandlerFactory-Integrated-4.0
Error Code 0x80072af9
Requested URL http://localhost:80/My.Virtual.Directory/Service.asmx
Physical Path C:\inetpub\wwwroot\My.Virtual.Directory\Service.asmx
Logon Method Not yet determined
Logon User Not yet determined

The errors suggested that we have problems with the configuration file, but the web.config was present (and well-formed), and there were no obvious permission problems, so it seems the file was being read. There was nothing in the event logs. Web searches yielded nothing that matched the 0x80072af9 error code or the description of the error. Even ERR.exe, recommended by Troubleshooting HTTP 500.19 Errors in IIS 7, failed me.

Fortunately, there were sibling virtual directories on the server, and they were working fine, even under the same App Pool. I knew that this virtual directory, unlike the others, restricted access to a whitelist of IP addresses. So, I changed the security/ipSecurity node’s allowUnlisted to true, just in case for some reason the clients’ IP addresses weren’t being detected properly. No change.

Frustrated, I removed the whole security node. The service worked!

So I took a closer look at the node:

<security>
  <ipSecurity allowUnlisted="false">
    <add ipAddress="127.0.0.1" allowed="true" />
    <add ipAddress="1.2.3.4 " allowed="true" />
  </ipSecurity>
</security>

Check out that “1.2.3.4″ ipAddress. Now check it again. It’s actually “1.2.3.4 “, with a space at the end. (I bolded the space there, just so you wouldn’t miss it.) It seems that this messes up the IP parsing, and IIS is completely flummoxed. Remove the space, and all is well.

Fixated on Fixie – the birth of a new unit test runner

I enjoy reading about how software is made, and I like unit testing frameworks. So, when I heard about Patrick Plioi‘s new project Fixie, I rushed to check it out.

In this case, “check it out” doesn’t mean “clone the repo and dig around the source code”. Nor does it mean “install the NuGet package and build something”. Although I may do those things in the future.

Nope. It means I read Mr. Plioi’s articles about Fixie and its development. And I am having a great time. Moreso than hearing about Fixie’s features (or more often lack of features), I’m enjoying seeing Mr. Plioi’s approach to setting up a new project, including:

  • prototyping the scariest integration points first
  • the importance of starting out with a one-click build, for himself and for potential future contributors
  • streamlining, automating, or eliminating as much ceremony as possible
  • bootstrapping, and more!

The articles are well-written and articulate, and mildly funny. They’re trending a little more into the implementation of Fixie itself, rather than guiding philosophies, but I still find them interesting. And it’s worth noting that all the while I was enjoying the articles, I was thinking in the back of my head “this is a great exercise, and very instructive, but I’ve no interest in actually using Fixie—I’m content with NUnit“. Until I read DRY Test Inheritance. I really liked the low-ceremony way conventions are used to locate test setups and teardowns. It hooked me. Even though I am usually not a fan of test class inheritance and the scheme described in this article has more weight than the Default Convention.

Of course, we’ll probably never switch at the Day Job, at least not until the ReSharper test runner supports Fixie, but it might be fun to use for a small home project.

FakeItEasy’s argument formatter auto-discovery – boon and inconvenience

Hi again. At the Day Job, we’ve recently dropped Typemock Isolator and NMock2 as the mocking frameworks of choice in the products that I work on. We’ve jumped on the FakeItEasy bandwagon. So far, we’re enjoying the change. FakeItEasy is powerful enough and the concepts and syntax fit the mind pretty well. Today I’m going to focus on one feature that I’ve really enjoyed but that has been an occasional thorn in the side.

This is a feature that Patrik Hägne has blogged about before, but that I think is still not well known. I found it accidentally, and have benefited from it. You can provide custom argument renderers to improve the messages you get when FakeItEasy detects an error due to missing or mismatched calls. Check out Mr. Hägne’s post for the full details, but if I may be so bold as to rip off some of his examples, here’s the gist (original meaning, not fancy github one).

Define a class that extends ArgumentValueFormatter<Person> (where Person is a class in your project), override GetStringValue with something that renders a Person, and FakeItEasy errors that need to talk about a Person change from this

Assertion failed for the following call:
    'FakeItEasy.Examples.IPersonRepository.Save()'
  Expected to find it exactly never but found it #1 times among the calls:
    1.  'FakeItEasy.Examples.IPersonRepository.Save(
            personToSave: FakeItEasy.Examples.Person)'

to

Assertion failed for the following call:
    'FakeItEasy.Examples.IPersonRepository.Save()'
  Expected to find it exactly never but found it #1 times among the calls:
    1.  'FakeItEasy.Examples.IPersonRepository.Save(
            personToSave: Person named Patrik Hägne,
                          date of birth 1977-04-05 (12227,874689919 days old).)'

It’s very easy to use, and quite helpful. However, lately I’ve had a few difficulties with some test projects and have tracked it back to an aspect of this feature. Specifically, for certain very large projects

  • My test fixtures are taking a long time to start up – several extra seconds while waiting for the first test to run. Specifically, the delay was happening in my first A.Fake call.
  • During this delay, several “LoaderLock was detected” popups appear, which have no obvious ill effect, but are very annoying, and
  • Finally, after a recent upgrade of dependent libraries, when I run the tests using the Resharper test runner, I see a “Microsoft Visual C++ Runtime Library Runtime Error!” in JetBrains.ReSharper.TestRunner.CLR4.exe. It claims that I’m trying to “use MSIL code from this assembly during native code initialzation”. The tests continue to run, but the TestRunner process never exits, and needs to be killed before test can be run again.

The reasons all these things are happening during the first FakeItEasy call is due to the way that FakeItEasy finds the custom ArgumentValueFormatter implementations. It scans all available assemblies, looking for any implementations. In this case, “all available assemblies” means every assembly in the AppDomain as well as all *.dll files in the current directory. This actually makes the feature a little more powerful than Mr. Hägne indicated—you can define your extensions in other assemblies than the test project’s. In fact, this is how FakeItEasy finds its own built-in ArgumentValueFormatters (one for null, one for System.String, and one for any System.Object that doesn’t have its own extensions). FakeItEasy is in the AppDomain, so its extensions are located by the scan. One benefit of doing such a wide scan is that it’s possible to define the formatter extension classes in a shared library that can be used across test projects.

It’s the scanning that’s causing my pain. First, some of the solutions at the Day Job are quite large, with dozens of assemblies in the test project’s AppDomain and build directory. Even if everything went well, it would take seconds to load and scan all those assemblies. Second, some of the DLLs in the directory aren’t under our control. Some aren’t managed. Some don’t play well with others. It’s these ones that are causing the other problems I mentioned above. Loading these assemblies causes them to be accessed in ways that they were never planned to be, which causes the LoaderLocks and Runtime Error.

What now? We’re investigating the assemblies we’re using to see if we can’t access them in a better way, but that’s probably going to be a slow operation, and one that may not bear fruit. In the meantime, I’ve forked FakeItEasy and am using the custom build in the one project that it was causing the most pain. The custom version only loads extensions from the FakeItEasy assembly. It’s kind of a terrible hack, and means that we can’t define custom extensions, but we hadn’t for that project anyhow, so it’s not yet causing pain. On the brighter side, there are no more errors or popups, and the tests start much more quickly.

Longer term, I’ve created FakeItEasy issue 130 to make the extension location a little more flexible. Once accepted and implemented, it will give the user control over how extension classes are located during FakeItEasy startup. (Then I can resume using the vanilla FakeItEasy at the Day Job.) If you’re curious, pop on over and take a look.

ReportGenerator indexing your whole drive? Check the case of your fullPaths.

[Update on 2013-06-22: I should've mentioned this a while ago, but the issue and patch I submitted were accepted and built into ReportGenerator 1.7.3.0, so if you have anything newer, you should be good.]

Recently I was working on a project at the Day Job, using OpenCover 1.7.1.0 and ReportGenerator 4.0.804 to report my test coverage, as is my wont, when the report generation started taking figuratively forever. Investigating, I saw something like

found report files: D:/sandbox/project/src/buildlogs/temp_test_coverage/Project.UnitTest.coverage.xml
Loading report 'D:\sandbox\project\src\buildlogs\temp_test_coverage\Project.UnitTest.coverage.xml'
 Preprocessing report
  Indexing classes in directory 'D:\sandbox\project\src\Module1\SubPath\'
  Added coverage information of 370/370 auto properties to module 'Module1'
  Indexing classes in directory 'D:\'

My D: drive isn’t the hugest, but it’s big enough, so that explained the delay. And of course, I certainly didn’t want anything above D:\sandbox\project\src indexed.

I took a peek at my .coverage.xml file and the ReportGenerator code and until I found the offending lines

<Module hash="9A-A3-0A-C0-1D-57-BA-2A-C2-D4-5B-9E-08-DE-BD-2D-46-04-AF-32">
  <FullName>D:\Sandbox\project\src\Module\UnitTest\bin\Release\Module.dll</FullName>
  <ModuleName>Module</ModuleName>
  <Files>
    …
    <File uid="803" fullPath="D:\sandbox\project\src\Module\File1.cs" />
    <File uid="806" fullPath="D:\Sandbox\project\src\Module\File2.cs" />
    <File uid="808" fullPath="D:\sandbox\project\src\Module\File3.cs" />
    …

Note the “Latin capital letter S” at the beginning of “Sandbox” on line 7. All the other lines had a “Latin small letter S”.
When ReportGenerator goes looking for *.cs files to scan, it starts at the directory whose name is the longest common prefix of all the fullPaths. Because “S” isn’t “s”, it came up with “D:\”.

I submitted an issue on the ReportGenerator CodePlex project, so maybe we’ll see a fix soon.

Of course I wondered “Why does the S differ for that entry?” but I figured I’d look at one thing at a time, and locating the fix for ReportGenerator was quicker.

Moving LibraryHippo to Python 2.7 – OpenID edition

Now that Google has announced that Python 2.7 is fully supported on Google App Engine, I figured I should get my act in gear and make convert LibraryHippo over. I’d had a few aborted attempts earlier, but this time things are going much better.

How We Got Here – Cloning LibraryHippo

One of the requirements for moving to Python 2.7 is that the app must use the High Replication Datastore, and LibraryHippo did not. Moreover, the only way to convert to the HRD is to copy your data to a whole new application. So I bit the bullet, and made a new application from the LibraryHippo source.

When you set up a new application, you have the option of allowing federated authentication via OpenID. I’d wanted to do this for some time, so I thought, “While I’m changing the datastore, template engine, and version of Python under the hood, why not add a little complexity?”, and I picked it.

The Simplest Thing That Should Work – Google as Provider

In theory, LibraryHippo should be able to support any OpenID provider, but I wanted to start with Google as provider for a few reasons:

  • concentrating one one provider would get the site running quickly and I could add additional providers over time
  • I need to support existing users – they’ve already registered with App Engine using Google, and I want things to keep working for them, and
  • I wanted to minimize my headaches – I figure, if an organization supports both an OpenID client feature and an OpenID provider, they must work together as well as any other combination.

Even though there’s been official guidance around using OpenID in App Engine since mid-2010, I started with Nick Johnson’s article for an overview – he’s never steered me wrong before. And I’m glad I did. While the official guide is very informative, Nick broke things down really well. To quote him,

Once you’ve enabled OpenID authentication for your app, a few things change:

  • URLs generated by create_login_url without a federated_identity parameter specified will redirect to the OpenID login page for Google Accounts.
  • URLs that are protected by “login: required” in app.yaml or web.xml will result in a redirect to the path “/_ah/login_required”, with a “continue” parameter of the page originally fetched. This allows you to provide your own openid login page.
  • URLs generated by create_login_url with a federated_identity provider will redirect to the specified provider.

That sounded pretty good – the existing application didn’t use login: required anywhere, just create_login_url (without a federated_identity, of course).
So, LibraryHippo should be good to go – every time create_login_url is used to generate a URL, it’ll send users to Google Accounts. I tried it out.

It just worked, almost. When a not-logged-in user tried to access a page that required a login, she was directed to the Google Accounts page. There were cosmetic differences, but I don’t think they’re worth worrying about:

standard Google login page

standard Google login page

federated Google login page

federated Google login page

Approve access to e-mail address

After providing her credentials, the user was redirected to a page that asked her if it was okay for LibraryHippo to know her e-mail address. After that approval was granted, it was back to the LibaryHippo site and everything operated as usual.

However, login: admin is still a problem. I really shouldn’t have been surprised by this, but login: admin seems to do the same thing that login: required does – redirect to /_ah/login_required, which is not found.

Login Required Not Found

This isn’t a huge problem – it only affects administrators (me), and I could workaround by visiting a page that required any kind of login first, but it still stuck in my craw.
Fortunately, the fix is very easy – just handle /_ah/login_required. I ripped off Nick’s OpenIdLoginHandler, only instead of offering a choice of providers using users.create_login_url, this one always redirects to Google’s OpenId provider page. With this fix, admins are able to go directly from a not-logged-in state to any admin required page.

class OpenIdLoginHandler(webapp2.RequestHandler):
    def get(self):
        continue_url = self.request.GET.get('continue')
        login_url = users.create_login_url(dest_url=continue_url)

        self.redirect(login_url)        

...

handlers = [ ...
    ('/_ah/login_required$', OpenIdLoginHandler),
    ... ]

Using Other Providers

With the above solution, LibraryHippo’s authentication system has the same functionality as before – users can login with a Google account. It’s time to add support for other OpenID providers.

username.myopenid.com

I added a custom provider picker page as Nick suggested, and tried to login with my myOpenID account, with my vanity URL as provider – blair.conrad.myopenid.com. The redirect to MyOpenID worked just as it should, and once I was authenticated, I landed back at LibraryHippo, at the “family creation” page, since LibraryHippo recognized me as a newly-authenticated user, with no history.

myopenid.com

Buoyed by my success, I tried again, this time using the “direct provider federated identity” MyOpenID url – myopenid.com. It was a complete disaster.

Error: Server Error  The server encountered an error and could not complete your request. If the problem persists, please report your problem and mention this error message and the query that caused it.

Once MyOpenID had confirmed my identity, and I was redirected back to the LibraryHippo application, App Engine threw a 500 Server Error. There’s nothing in the logs – just the horrible error on the screen. In desperation, I stripped down my login handler to the bare minimum, using the example at Using Federated Authentication via OpenID in Google App Engine as my guide. I ended up with this class that reproduces the problem:

class TryLogin(webapp2.RequestHandler):
    def get(self):
        providers = {
            'Google'   : 'www.google.com/accounts/o8/id',
            'MyOpenID' : 'myopenid.com',
            'Blair Conrad\'s MyOpenID' : 'blair.conrad.myopenid.com',
            'Blair Conrad\'s WordPress' : 'blairconrad.wordpress.com',
            'Yahoo' : 'yahoo.com',
            'StackExchange': 'openid.stackexchange.com',
            }
        
        user = users.get_current_user()
        if user:  # signed in already
            self.response.out.write('Hello <em>%s</em>! [<a href="%s">sign out</a>]' % (
                user.nickname(), users.create_logout_url(self.request.uri)))
        else:     # let user choose authenticator
            self.response.out.write('Hello world! Sign in at: ')
            for name, uri in providers.items():
                self.response.out.write('[<a href="%s">%s</a>]' % (
                    users.create_login_url(dest_url= '/trylogin', federated_identity=uri), name))

...

handlers = [
    ('/trylogin$', TryLogin),
    ('/_ah/login_required$', OpenIdLoginHandler),
    ...
]

Interestingly, both Yahoo! and WordPress work, but StackExchange does not. If it weren’t for Yahoo!, I’d guess that it’s the direct provider federated identities that give App Engine problems (yes, Google is a direct provider, but I consider it to be an exception in any case).

Next steps

For now, I’m going to use the simple “just Google as federated ID provider” solution that I described above. It seems to work, and I’d rather see if I can find out why these providers fail before implementing an OpenID selector that excludes a few providers. Also, implementing the simple solution will allow me to experiment with federated IDs on the side, since I don’t know how e-mail will work with federated IDs, or how best to add federated users as families’ responsible parties. But that’s a story for another day.

Best all-around .NET coverage tool – OpenCover

This post is the conclusion of a series chronicling my search for a .NET coverage tool.

This is the gala awards show, where my chosen coverage tool is announced.

If you’ve come this far, you’ve probably already read the title, and it won’t surprise you to learn that I’ve chosen OpenCover. It offered the best fit for my requirements – the only areas where I found it lacking were in the “nice to haves”. Witness:

  • OpenCover is pretty easy to run from the command line – second only to NCover.
  • It can (with the help of ReportGenerator) generated coverage reports in XML and HTML.
  • OpenCover has an integrated auto-deploy, so it can be bundled with the source tree and new developers or build servers just work – dotCover has no such option, and I was not able to use NCover this way.
  • I’ve been able to link with TypeMock Isolator with little trouble, and the new Isolator may obviate the need for my small workaround.
  • It’s free. Aside from the obvious benefit, it’s nice to not have to count licenses when adding developers and/or build server nodes.
  • There’s no GUI integration, but this was a nice to have. If some developer is absolutely dying to have this, my boss’s boss has indicated that money could be available for individual licenses of something like dotCover.
  • There’s no support for integrating with IIS. We don’t need this right now, so that’s okay. Again, if we one or two developers find a need, we have the option of buying a license of some other tool. Even better, support may be coming soon.

After considering OpenCover’s strengths in the areas I absolutely needed, and its weaknesses, which all appear to be in areas that I care a little less about, I recommended it the boss’s boss, who was agreed with the assessment and was happy to keep a little money in his pocket for now.

So, I grabbed 2.0.802, incorporated it into one product’s build, and out popped coverage numbers. Very exciting. I did notice a few things, though:

  1. Branch coverage has been added since I last evaluated the product!
  2. One fairly complicated integration-style testfixture is not runnable under OpenCover – the class tested creates a background thread and starting the thread results in a System.AccessViolationException. I was unable to determine the cause of this, and have temporarily removed the test from coverage, instead executing it with NUnit directly. I’m going to continue investigating this problem.
  3. Since I’m XCopy deploying, I was bitten by the dependency on the Microsoft Visual C++ 2010 Redistributable Package – I ended up including the DLLs in my imported bundle, and all was well, but I worry a little about the stability of this solution.
  4. The time taken to execute our tests (there are over 5000, and many hit a database) increased from about 7 minutes to about 8. This is an acceptable degradation, since the test run isn’t the bottleneck in our build process.
  5. The number of “Cannot instrument as no PDB could be loaded” messages is daunting. I’m hoping that things will be improved once I get a build that contains a fix for issue 40.

Categories

Follow

Get every new post delivered to your Inbox.

Join 34 other followers