Dear VSTS Test: I Hate You

I cannot even count the ways that VSTS Test is inferior to NUnit (or MbUnit).

I’ve been running my tests via TestDriven.NET (right-click, Run Test) and it has been working just fine.  The tests run, I get my results.  The reason I do this is because the built-in test runner sucks leaves something to be desired.

I went to a teammates machine the other day and he ran the test suite and most of the tests failed.  In fact, any test that hit the database failed.  All of them had the same error message: 

ConfigurationException:  Could not load the configSource ‘local-connectionStrings.config’.

This file is copied (via pre-build event) from a root location, to the output directory.  This file is, by nature, different for each developer.  It contains connection strings that are specific to the machine.

Using FileMon I was able to see that the VSTS Test Runner was looking in the ..\..\TestResults\{TIMESTAMP FOLDER HERE}\Out\ directory.  The VSTS product team claims that this is for remote running of tests, but I suspect that it is just a poor solution to the problem.  Anyway, there is no way that I can (via a batch file) copy the config file to this folder.  So Pre-Build events are out.

I read up on the issue and find that it is possible to have items copied to the Test Results folder.  You have to add this attribute to a test method:

[DeployItem(“local-connectionStrings.config”)]

…and yes, I said to a test method.  It will not work on a [TestInitialize] or [TestClass].  So basically if I want to use this feature, I have to retro fit around 80 tests to include this attribute, then tell my whole team to do it as well.  No thank you.

You can also add the Deployment Item to the localtestrun.testrunconfig file (under Solution Items).  This file says “local” but it is checked into source control, so it is not clear if this change will be applied to everyone’s working copy or my own.

This is a stupid, hacky solution to a problem that doesn’t even exist with the other unit test frameworks.  Since we aren’t really using any TFS test features yet (Web Tests will come) —I’m tempted to just rewrite them all for NUnit.  Of course I will get pushback from management, but these little issues are becoming a hinderance. 

If it is cumbersome to test, developers won’t write unit tests.

(Oh, and JetBrains:  when will you release support for the VSTS Tests?  I’m dying over here!)