Choosing a BDD framework for .Net

This was written in 2013 and isn’t current anymore.

tl;dr

Specs aren’t written by developers: choose between SpecFlow and NBehave.

Specs are written by developers: my personal recommendation NSpec. Runner up StoryQ.

Introduction

BDD or behavior driven development is the practice of writing an executable testable specification that describes the application’s behavior. This specification is often written in a fluent interface, a DSL or in plain English (or rather close to plain English).

While in TDD the focus is on writing tests that single out units of the application, BDD is focused on writing tests on the behavior of the application. This can also be tought as testing features of the application.

Before presenting the frameworks I will make a distinction between xBehave and xSpec style frameworks.

xBehave

xBehave frameworks are about writing user level stories in a form comprehensible by anyone. These stories can be written by the users themselves or by a group consisting of developers, users and testers.

These framework typically use a story defined in a DSL close to English and then map this story to a test written in code by the developers. Cucumber is a well known example of such a framework.

xSpec

xSpec frameworks are usually about developers writing tests in code using an approach that favors testing behavior and functionality. These tests are usually closer to unit tests in both appearance and granularity but feature some differences.

Both approaches could be combined on a single project. The user stories could be converted to xBehave tests, while the developers could adopt an xSpec framework to replace or complement xUnit tests.

Available frameworks

Here is a compilation of xSpec and xBehave BDD frameworks for .Net. This list is current as of the writing of this post and while I tried my best to find most frameworks, I realized while compiling this list that there are just too many, I am sure I must have missed a few.

By the way, it’s not an error in the table, I have found two frameworks named NSpec which appear to be totally independent. The first one listed is the most active and most popular. The second one is an older project that hasn’t seen any recent activity.

Name	Project development still active	Type	License	NuGet package
SpecFlow	Yes	xBehave	BSD style	Yes
specunit-net	No	xSpec	Apache License 2.0	No
Machine.Specifications / MSpec	Yes	xSpec	xUnit and MS-PL	Yes
NSpec	Yes	xSpec	MIT License	Yes
NSpec (older project)	No	xSpec	zlib/libpng license	No
NBehave	Yes	xBehave	BSD 3	Yes
StoryQ	No	***	MIT	Yes
NSpecify	No, seems never to have taken off.	xSpec	?	No
.NetSpec	No	xSpec	None	No
xbehave.net	Yes	xSpec	MIT	Yes
SubSpec	No	xSpec	MS-PL	Yes
NJasmine	Yes	xSpec	MIT	Yes
SpecsFor	Yes	xSpec	None	Yes
Behavioral	No	xSpec	BSD	Yes
NaturalSpec	Yes	xSpec	MIT / MS-PL	Yes
BDDish	No	xSpec	None	Yes

*** This framework is harder to classify. Based on my interpretation of xSpec/xBehave and my comprehension of the framework, I personally place it in a gray zone. They suggest to start with a plain text story but it isn’t directly used by the code. The specifications are directly in C# test code contrary to other more “pure” xBehave frameworks. It is a sort of hybrid and does tend to lean much more on the xSpec side than on the xBehave side.

Narrowing the selection

First of all, I would eliminate the following frameworks from investigation: specunit-net, NSpec (older project), NSpecify and .NetSpec as they are abandoned inactive projects with most never having really made it to “production status”.

NaturalSpec is in F# which is great for those doing F#. There is a link to a post on it’s GitHub page about how to use it with C# objects, but I would personally prefer a library that targets C# for a C# project.

I wasn’t able to see much activity for BDDish (last commit to GitHub more than 2 years ago, only 3 questions on StackOverflow, not much documentation or examples). I would also skip this one.

As an unrelated side-note, out of the remaining more serious contenders xbehave.net and SubSpec are strikingly similar. At first glance, both projects seem to be closely related. If you are considering one of these two, you might as well look into the other as well.

Making a selection for an xBehave framework

The first question you need to ask yourself is if you want to do xBehave style tests in a language closer to plain text, with specs that can be written by non developers.

If that is so, your choice is pretty slim. Your choice is between SpecFlow and NBehave.

StoryQ could also be considered if SpecFlow or NBehave are not satisfactory.

Both SpecFlow and NBehave use Gherkin (the DSL used by Cucumber) to write their specs. Both store specs in *.feature files. Both also use attributes on methods when writing tests.

[Given("I am not logged in")]
public void LogOut()
{

Both have documentation available on GitHub.

SpecFlow seems to have more documentation and it is also recommended in this StackOverflow question in the top two rated answers.

On the other hand the SpecFlow doc is pretty dry on code examples. I personally preferred NBehave’s documentation over SpecFlow.

Making a selection for an xSpec framework

If you do not need for your specs to be written in a plain text like format and want to save time and work use an xSpec framework. You won’t need to write the binding code and writing tests will be easier for those already familiar with an xUnit framework.

On the xSpec side, you have a lot of options.

I narrowed my search to the following frameworks: MSpec, NSpec, SpecsFor, StoryQ and xBehave.net.

The following table compares those frameworks by considering their total number of questions on StackOverflow. I also rated the official documentation on a scale of 1 to 3, with 1 being the poorest and 3 the best score. This rating is largely subjective and represents my appreciation of the available documentation. It should be noted I prefer a style which includes code samples and which gets you rapidly on your feet.

Name	Number of questions on StackOverflow	Documentation
MSpec	519*	1 / 3
NSpec	62	3 / 3
SpecsFor	5	3 / 3
StoryQ	36	3 / 3
xBehave.net	5	2 / 3

*: I added the number of questions for both MSpec and Machine.Specifications. Even if just looking at the number of questions for either MSpec or Machine.Specifications, it is the clear winner here.

My personal recommendation for an xSpec framework is NSpec.

After trying each of these frameworks for a simple two tests scenario here is my personal rating:

1. NSpec
2. StoryQ
3. SpecsFor

Here is a code sample of two simple tests using NSpec:

namespace NSpecTests
{
    class TicTacToe_specifications : nspec
    {
        void given_a_new_board()
        {
            it["A new tic tac toe board is empty"] = () => ticTacToeBoard.IsEmpty.should_be_true();

            context["When placing a first X"] = () =>
            {
                before = () => ticTacToeBoard.PlaceXat(1);
                it["Board should not be empty"] = () => ticTacToeBoard.IsEmpty.should_be_false();
            };
        }
        TicTacToeBoard ticTacToeBoard = new TicTacToeBoard();
    }
}

And here is the output that will be generated using NSpecRunner, NSpec default test runner.

Output :
TicTacToe specifications
  given a new board
    A new tic tac toe board is empty
    When placing a first X
      Board should not be empty

2 Examples, 0 Failed, 0 Pending

Here are the two same tests and their output in StoryQ:

namespace StoryQTests
{
    [TestClass]
    public class TicTacToe_specifications
    {
        TicTacToeBoard ticTacToeBoard;
        bool result;

        [TestMethod]
        public void given_a_new_board()
        {
            new Story("A new tic tac toe board is empty")
                .InOrderTo("to start a new game")
                .AsA("player")
                .IWant("to have an empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeTrue)
                .Execute();

            new Story("After placing a first X")
                .InOrderTo("to make my first move")
                .AsA("player")
                .IWant("to have a non empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .And(PlacingAFirstX)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeFalse)
                .Execute();
        }

        public void ANewBoard() { ticTacToeBoard = new TicTacToeBoard(); }
        public void PlacingAFirstX() { ticTacToeBoard.PlaceXat(1); }
        public void CheckingIfBoardIsEmpty() { result = ticTacToeBoard.IsEmpty; }
        public void ItShouldBeTrue() { Assert.IsTrue(result); }
        public void ItShouldBeFalse() { Assert.IsFalse(result); }
    }
}

And the output:

Story is A new tic tac toe board is empty
  In order to to start a new game
  As a player
  I want to have an empty board

      With scenario new board
        Given a new board                 => Passed
        When checking if board is empty   => Passed
        Then it should be true            => Passed
Story is After placing a first X
  In order to to make my first move
  As a player
  I want to have a non empty board

      With scenario new board
        Given a new board               => Passed
          And placing a first x         => Passed
        When checking if board is empty => Passed
        Then it should be false         => Passed

I tried to emulate the style in the StoryQ examples but I’m not sure if I am using StoryQ as efficiently as possible.

NSpec

Pros:

• concise
• easy to get started with
• makes me think of RSpec
• it kicks ass

Cons:

• can’t use MS test runner
• must manually rebuild test library before each NSpecRunner run
• tests will fail if some conventions aren’t followed (ie: underscores in method names)

StoryQ

Pros:

• uses MSTest or NUnit test runners and standard test attributes
• great tools
• focus on user stories

Cons:

• not sure if I am using it correctly
• very verbose

Conclusion

For my conclusion read the tl;dr section at the start of this post.