ContinueOnError, yet still failing the build?
I'm using MSBuild to run all the unit tests for our projects. The target looks like this:
| |
<Target Name="Test" DependsOnTargets="Build">
<CreateItem Include="FakeTestTarget\bin\$(Configuration)\*Test.dll;">
<Output ItemName="TestAssemblies" TaskParameter="Include"/>
</CreateItem> <Exec ContinueOnError="true" Command="mstest /detail:errormessage /detail:errorstacktrace /testcontainer:%(TestAssemblies.RelativeDir)%(TestAssemblies.Filename)%(TestAssemblies.Extension)">
</Exec>
</Target>
|
The reason I'm using ContinueOnError="true" is of course because I want all tests to run even if one test fails. However, this means the build as a whole will be marked as successful, even if some unit tests failed.
Is there a way around this?
The other option I have is to pass all the test DLLs to mstest as part of a single invocation. The problem with this is that the VSTestHost.exe process ends up using a *lot* of virtual memory, which eventually crashes the whole testing process.
The cleanest way would probably be to just write an MSBuild task that wraps mstest.exe. The task would loop over the unit test assemblies, keep track of failures, but finish all the unit test assemblies even if one failed.
Aside from that, the only thing I could come up with was the following. I haven't tested this, but I think it should work.
<Target Name="Test"
DependsOnTargets="
Build;
FindTestAssemblies;
RunAllTests;
CheckForTestErrors
"/><Target Name="FindTestAssemblies">
<CreateItem Include="FakeTestTarget\bin\$(Configuration)\*Test.dll;">
<Output ItemName="TestAssemblies" TaskParameter="Include"/>
</CreateItem>
</Target>
<!-- The Outputs attribute is a hack to force the whole target to loop over the list of items in the TestAssemblies list. -->
<Target Name="RunAllTests" Outputs="%(TestAssemblies.Identity)">
<Exec ContinueOnError="true"
Command="mstest /detail:errormessage /detail:errorstacktrace /testcontainer:%(TestAssemblies.Identity)"
>
<Output TaskParameter="ExitCode" PropertyName="MstestExitCode"/>
</Exec>
<CreateProperty Value="true"
Condition=" '$(MstestExitCode)' != '0' ">
<Output TaskParameter="Value" PropertyName="UnitTestFailed"/>
</CreateProperty>
</Target>
<Target Name="CheckForTestErrors">
<Error Text="One or more unit tests failed."
Condition="'$(UnitTestFailed)' == 'true'" />
</Target>
--Rajeev
Thanks, it looks like that might work. However, I ended doing something different: Writing a small app that used XPath against the XML test results files generated by mstest, looking for the failure codes. The build succeeds or fails depending on the exit code from this app.
Although, I can't help feeling that "fail the build, but continue anyway" is something that would be useful in general.
Arild,
You're
completely right that "fail the build but continue anyway" would be useful. It seems like there are three scenarios 1) You want any task failure to fail the target and the build. Normal default case.
2) You want any task failure to be ignored. E.g, you want to attempt to copy or delete a file, but don't care if it is locked. You can achieve this today with ContinueOnError="true". (Any errors from the task we change to warnings, because that seems more consistent, and sometimes hosts assume that logged errors implies a build failure)3) You want any task failure to fail the target and the build, but for execution to continue. E.g, you are building many projects with the MSBuild task (in a tree or a solution) and don't want to stop when the first one fails, but you do want the build to ultimately say 'failed'. Right now you can't easily do #3, except when you are passing a list of targets to the MSBuild task, which offers the "RunEachTargetSeparately" parameter.
Back in the spring the team talked about having an attribute like this on any tasks instead of ContinueOnError:
OnFailure="FailAndStop|SucceedAndWarn|FailAndContinue"
mapping to the 3 scenarios above respectively.
We liked the idea a lot, but it was too late to make the change at that point. We could perhaps add support for this in a future version though. (Clearly, ContinueOnError now has to be supported though.) What do you folks think? How important is this?
Dan
