Thursday, April 21, 2011

Getting Started with AutoTest.Net

I recently came across a new tool called AutoTest.Net.  This is a great tool if you are a TDD/BDD practitioner.  Basically AutoTest.Net can sit in the background and watch your source tree.  Whenever you make changes to a file and save it, AutoTest will build your solution and run all unit tests that it finds.  When it has run the tests it will display the failed tests in the AutoTest window (there is also a console runner if you are a hardcore console user). 

This takes away some of the friction that comes with unit testing because now you don’t have to do the step of build and run tests during development.  Simply write a test and save.  AutoTest runs your tests and you see a failed test.  Now implement the code that should make the test and save again.  AutoTest again detects the change and does a build and test run.  This cycle becomes automatic quite quickly and I’m finding I really like not having to explicitly build and run tests. 

Features

AutoTest.Net sports a nice set of features.  You can see the full list of features on the main GitHub repo page (scroll down) but here’s a few.  It supports multiple test frameworks including MSTest, NUnit, MSpec, and XUnit.net (I believe other frameworks can be supported by writing a plugin).  It can also monitor your project in two different ways.  One is by watching for changes to all files in your project (ie. watching the source files).  The other is by watching only assemblies.  Using this second method it would only run tests when you recompile your code manually.  I’m not sure where this would be preferred but I might find a use for it yet.

Edit: Svein (the author of AutoTest.Net) commented that one other compelling feature is Growl/Snarl support. If you have either one installed you can get test pass/fail notifications through Growl/Snarl, which means you don't have to flip back to the AutoTest window to check that status of your code. Nice!

This leads me to one other feature that I missed and that is that AutoTest.Net is cross-platform. It currently supports both the Microsoft .NET framework as well as Mono. This means you can develop on OS X or Linux.

Getting AutoTest.Net

So, to get started, download AutoTest.Net from GitHub.  The easiest way is to click the “Downloads” link on the main AutoTest.Net project page and click one of the AutoTest zipfile links (at this time AutoTest.Net-v1.1.zip is the latest). 

AutoTest.Net Download

Once downloaded, unzip to a directory that you’ll run it from.  I’ve put it in my “utilities” directory.  At this point you can start using AutoTest.Net by running AutoTest.WinForms.exe or AutoTest.Console.exe.  If you run the WinForms version, AutoTest will start by asking you what directory you’d like to monitor for changes.  Typically you would select the root directory of your .Net solution (where your .sln file resides).  Once you’ve selected a directory and clicked OK, you’re ready to start development.

image

The screenshot above is the main AutoTest.Net window.  You can see that it ran 1 build and executed a total of 13 unit tests.  The neat thing is that now as I continue to work, all I need to do to cause AutoTest.Net to recompile and re-run all tests is to make a code change and save the file.  This sounds like a small change in the regular flow of code, save, build, run tests.  However, once I worked with AutoTest.Net for a while it started to feel very natural and going back to the old flow will feel like adding friction.

Configuration

Alright, so we have AutoTest.Net running and that’s good.  As with most tools, it comes with a config file, AutoTest.config (which is shared between the WinForms and Console apps).  This config file is well-documented so for the most part you can just open it up and figure out what knobs you can adjust.

Overriding Options

One nice feature that AutoTest.Net has is that you can have a base configuration in the directory where the AutoTest binaries reside but then override it by dropping an AutoTest.config file in the root directory that you are monitoring.  This allows you to keep a sensible base config in your AutoTest application directory and then override per-project as needed.  Typically I’ve been setting the “watch” directory as the directory that my .sln file is in and so you’d drop the AutoTest.config file in the same directory as the .sln file. 

image

In the above screenshot I have the AutoTest.config file which overrides a few settings, one of which is the ignore file: _ignorefile.txt

IgnoreFile Option

One option that is useful is called the IgnoreFile.  This is well-documented in the config file but it didn’t click with me initially.  The IgnoreFile option specifies a file that contains a list of files and folders that AutoTest.Net should ignore when monitoring the configured directories for changes.  The config file mentions the .gitignore file as an similar example so if you are familiar with git, it should make sense.  The one piece that took me a while to figure out was where this ignore file should go.  I finally figured out that if you put it in the root of the monitored directory (ie. beside your .sln file), it will pick it up (see above screenshot). 

Wrapping up

AutoTest.Net is a great tool to help with the Red, Green, Refactor flow.  It removes some of the friction in my day-to-day work by eliminating the need to manually invoke a compile and test run. 

Lastly, the observant readers will have noticed that I said it builds and runs all tests any time it notices a file change.  Greg Young is working on a tool called Mighty Moose that builds on top of AutoTest.Net as an add-in to Visual Studio (I think there’s a standalone version too).  Mighty Moose ups the ante by figuring out what code changed and what tests would be affected by that change.  It then only runs the affected tests, significantly cutting down the test run time. 

3 comments:

Svein Arne Ackenhausen said...

Great post! One feature that should be mentioned though is the Growl/Snarl support which will give you run notifications.
About Mighty-Moose I was working on the standalone runner yesterday so it's on the way :)

Dave Harris said...

Without having used the product, my gut tells me that I wouldn't want to build and run tests every time I save a file. I save files far more frequently than I would want to test them, and a build is hugely intensive from a CPU/time standpoint. Wouldn't this slow a developer down considerably throughout the day?

Jeremy Wiebe said...

@Dave It partly depends on the size of the project. On my current project I don't think it'd work well to be building and running all tests whenever I save. However, for smaller and medium sized projects I don't think it'd be as impactful as you're thinking.

AutoTest.Net provides another option though. You can configure AutoTest.Net to monitor a set of assemblies instead of an entire source tree. In this way it will never build anything. It will only run tests when it sees that any of the monitored assemblies have changed. Using this option it would wait till _you_ recompile your project and then it would start up a test run.