CruiseControl.Net, or Is There A Limit to my Patience?

Continuing with my trials and tribulations in regards to CCNet

Building my projects. I needed to update the svn settings file we have with the path to the svn executable. Then, we are using Rodemeyer.MsBuildToCCNet.dll for the MSBuild tasks and the server needed that library. So I had to copy that to the CCnet server directory.

I ended up just installing VisualStudio 2010 on the build server though I wanted to avoid it. We are using utilities like resgen.exe as well as our unit tests using MSTest. I'm aware I can jump through hoops to install these separate of VisualStudio, but it ended up being a lot easier to just install VS.

I am setting up CCNet to run an NCover task to run our unit tests in MSTest (Apparently I was on drugs when I was thinking earlier that our tests were done in NUnit; I'm sort of wishing now that we had stuck with NUnit rather than using MSTest). The path to MSTest is in it's regular place after installing VisualStudio: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe. NCover is at its default installation place: C:\Program Files (x86)\NCover\ncover.console.exe. Hopefully this is as simple as putting the NCover profiler task into my ccNet project definition (http://www.cruisecontrolnet.org/projects/ccnet/wiki/NCover_Profiler_Task).

<ncoverprofile>
<executable>C:\Program Files (x86)\NCover\NCover.Console.exe</executable>
<program>C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe</program> <testproject>myproject.test.dll</testproject>
<workingdir>build\unittests</workingdir>
<includedassemblies>myproject.*.dll</includedassemblies>
</ncoverProfile>

Yeah... definitely not that easy. First off, the ncoverProfile tag has a bug and doesn't handle spaces in the paths. At least in the 3.3 version of NCover I was using to begin with. I got around that by using the DOS shortnames in the paths. CCNet just passes the testProject value as the first argument to MSTest, which also causes a problem. You get the error from MSTest: 'Invalid switch "c"'. The assembly name in the testProject node needs to have the /testcontainer switch appended. Then I had set the includedAssemblies to *.dll since I figured that would include all the dlls in my working directory. I got an error from NCover that the includedAssemblies included an invalid wildcard '*'. Really?!? Ok, so I just lef the includedAssemblies value blank figuring NCover would just be smart enough to include the dlls in the working directory, and it seems that maybe it was close anyway, because I finally got the MSTest command to at least run my unit tests. My task at this point looked like this:

<ncoverprofile>
<executable>C:\PROGRA~2\NCover\NCover.Console.exe</executable>
<program>C:\PROGRA~2\MICROS~2.0\Common7\IDE\MSTest.exe</program> <testproject>/testcontainer:MembershipTests.dll</testproject>
<workingdir>build\unittests</workingdir> <includedassemblies></includedassemblies>
</ncoverProfile>

MSTest was running but NCover was erroring saying that there was no coverage data: "NCover.Console is returning exit code #20000". Supposedly this can happen when you use NCover 64 bit to cover a 32 bit assembly or vice-versa. My assembly was built for AnyCPU, so I thought I should be ok. I tried running the tests through NCoverExplorer and was getting a similar error, so since only the 32-bit version of NCover was installed on this build machine (since that's the installer we had locally available before), I figured it wouldn't hurt to install the 64-bit version and see if that was indeed the problem.

Once I have the 64-bit version installed, my tests ran fine through NCoverExplorer even though I was still referencing the 32-bit installation of NCover on the machine! So, I was like... ok, let's try this out through CCNet now. After working through an issue with an old NCover task being in the MSBuild project (It suddenly reared its head since it HAD been looking for NCover in C:/Program Files/NCover instead of C:/Program Files (x86)/NCover and was simply failing to even load until I installed the 64-bit version on the new machine *sheepish*), I am getting 42 passing tests, 1 failure with 72% coverage. Not horrible considering it has been a while since we've gotten these tests to run at all. Just gotta go fix that one test, get my build to go green (it fails automatically on the failed unit test) and see if I can't get the coverage up at least to our 85% target. Then the more daunting task: to integrate this into the other VisualStudio projects that comprise the system. While the Membership providers were written in a TDD manner, most of the rest of the system was not. Last I looked, our unit test coverage was somewhere around 15%. Ugh. Finally our real goal here is to get some decent coverage with our acceptance testing. I still have to get Fitnesse running with CCNet on the new machine.

I frankly am wondering what sort of testing ThoughtWorks does before releasing some of their updates. I have to guess they aren't doing much testing on a Windows environment with Microsoft tools. I have to agree that it seems more and more that CCNet and MSBuild with MSTest simply don't mix well and that, though I have gotten this far, Continuous Integration shouldn't be this hard.