Choosing a BDD framework for .Net

tl;dr

Specs aren’t written by developers: choose between SpecFlow and NBehave.

Specs are written by developers: my personal recommendation NSpec. Runner up StoryQ.

Introduction

BDD or behavior driven development is the practice of writing an executable testable specification that describes the application’s behavior. This specification is often written in a fluent interface, a DSL or in plain English (or rather close to plain English).

While in TDD the focus is on writing tests that single out units of the application, BDD is focused on writing tests on the behavior of the application. This can also be tought as testing features of the application.

Before presenting the frameworks I will make a distinction between xBehave and xSpec style frameworks.

xBehave

xBehave frameworks are about writing user level stories in a form comprehensible by anyone. These stories can be written by the users themselves or by a group consisting of developers, users and testers.

These framework typically use a story defined in a DSL close to English and then map this story to a test written in code by the developers. Cucumber is a well known example of such a framework.

xSpec

xSpec frameworks are usually about developers writing tests in code using an approach that favors testing behavior and functionality. These tests are usually closer to unit tests in both appearance and granularity but feature some differences.

Both approaches could be combined on a single project. The user stories could be converted to xBehave tests, while the developers could adopt an xSpec framework to replace or complement xUnit tests.

Available frameworks

Here is a compilation of xSpec and xBehave BDD frameworks for .Net. This list is current as of the writing of this post and while I tried my best to find most frameworks, I realized while compiling this list that there are just too many, I am sure I must have missed a few.

By the way, it’s not an error in the table, I have found two frameworks named NSpec which appear to be totally independent. The first one listed is the most active and most popular. The second one is an older project that hasn’t seen any recent activity.

Name Project development still active Type License NuGet package
SpecFlow Yes xBehave BSD style Yes
specunit-net No xSpec Apache License 2.0 No
Machine.Specifications / MSpec Yes xSpec xUnit and MS-PL Yes
NSpec Yes xSpec MIT License Yes
NSpec (older project) No xSpec zlib/libpng license No
NBehave Yes xBehave BSD 3 Yes
StoryQ No *** MIT Yes
NSpecify No, seems never to have taken off. xSpec ? No
.NetSpec No xSpec None No
xbehave.net Yes xSpec MIT Yes
SubSpec No xSpec MS-PL Yes
NJasmine Yes xSpec MIT Yes
SpecsFor Yes xSpec None Yes
Behavioral No xSpec BSD Yes
NaturalSpec Yes xSpec MIT / MS-PL Yes
BDDish No xSpec None Yes

*** This framework is harder to classify. Based on my interpretation of xSpec/xBehave and my comprehension of the framework, I personally place it in a gray zone. They suggest to start with a plain text story but it isn’t directly used by the code. The specifications are directly in C# test code contrary to other more “pure” xBehave frameworks. It is a sort of hybrid and does tend to lean much more on the xSpec side than on the xBehave side.

Narrowing the selection

First of all, I would eliminate the following frameworks from investigation: specunit-net, NSpec (older project), NSpecify and .NetSpec as they are abandoned inactive projects with most never having really made it to “production status”.

NaturalSpec is in F# which is great for those doing F#. There is a link to a post on it’s GitHub page about how to use it with C# objects, but I would personally prefer a library that targets C# for a C# project.

I wasn’t able to see much activity for BDDish (last commit to GitHub more than 2 years ago, only 3 questions on StackOverflow, not much documentation or examples). I would also skip this one.

As an unrelated side-note, out of the remaining more serious contenders xbehave.net and SubSpec are strikingly similar. At first glance, both projects seem to be closely related. If you are considering one of these two, you might as well look into the other as well.

Making a selection for an xBehave framework

The first question you need to ask yourself is if you want to do xBehave style tests in a language closer to plain text, with specs that can be written by non developers.

If that is so, your choice is pretty slim. Your choice is between SpecFlow and NBehave.

StoryQ could also be considered if SpecFlow or NBehave are not satisfactory.

Both SpecFlow and NBehave use Gherkin (the DSL used by Cucumber) to write their specs. Both store specs in *.feature files. Both also use attributes on methods when writing tests.

[Given("I am not logged in")]
public void LogOut()
{

Both have documentation available on GitHub.

SpecFlow seems to have more documentation and it is also recommended in this StackOverflow question in the top two rated answers.

On the other hand the SpecFlow doc is pretty dry on code examples. I personally preferred NBehave’s documentation over SpecFlow.

Making a selection for an xSpec framework

If you do not need for your specs to be written in a plain text like format and want to save time and work use an xSpec framework. You won’t need to write the binding code and writing tests will be easier for those already familiar with an xUnit framework.

On the xSpec side, you have a lot of options.

I narrowed my search to the following frameworks: MSpec, NSpec, SpecsFor, StoryQ and xBehave.net.

The following table compares those frameworks by considering their total number of questions on StackOverflow. I also rated the official documentation on a scale of 1 to 3, with 1 being the poorest and 3 the best score. This rating is largely subjective and represents my appreciation of the available documentation. It should be noted I prefer a style which includes code samples and which gets you rapidly on your feet.

Name Number of questions on StackOverflow Documentation
MSpec 519* 1 / 3
NSpec 62 3 / 3
SpecsFor 5 3 / 3
StoryQ 36 3 / 3
xBehave.net 5 2 / 3

*: I added the number of questions for both MSpec and Machine.Specifications. Even if just looking at the number of questions for either MSpec or Machine.Specifications, it is the clear winner here.

My personal recommendation for an xSpec framework is NSpec.

After trying each of these frameworks for a simple two tests scenario here is my personal rating:

  1. NSpec
  2. StoryQ
  3. SpecsFor

Here is a code sample of two simple tests using NSpec:

namespace NSpecTests
{
    class TicTacToe_specifications : nspec
    {
        void given_a_new_board()
        {
            it["A new tic tac toe board is empty"] = () => ticTacToeBoard.IsEmpty.should_be_true();

            context["When placing a first X"] = () => 
            {
                before = () => ticTacToeBoard.PlaceXat(1);
                it["Board should not be empty"] = () => ticTacToeBoard.IsEmpty.should_be_false();
            };
        }
        TicTacToeBoard ticTacToeBoard = new TicTacToeBoard();
    }
}

And here is the output that will be generated using NSpecRunner, NSpec default test runner.

Output :
TicTacToe specifications
  given a new board
    A new tic tac toe board is empty
    When placing a first X
      Board should not be empty

2 Examples, 0 Failed, 0 Pending

Here are the two same tests and their output in StoryQ:

namespace StoryQTests
{   
    [TestClass]
    public class TicTacToe_specifications
    {
        TicTacToeBoard ticTacToeBoard;
        bool result;

        [TestMethod]
        public void given_a_new_board()
        {
            new Story("A new tic tac toe board is empty")
                .InOrderTo("to start a new game")
                .AsA("player")
                .IWant("to have an empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeTrue)
                .Execute();

            new Story("After placing a first X")
                .InOrderTo("to make my first move")
                .AsA("player")
                .IWant("to have a non empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .And(PlacingAFirstX)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeFalse)
                .Execute();
        }
        
        public void ANewBoard() { ticTacToeBoard = new TicTacToeBoard(); }
        public void PlacingAFirstX() { ticTacToeBoard.PlaceXat(1); }
        public void CheckingIfBoardIsEmpty() { result = ticTacToeBoard.IsEmpty; }
        public void ItShouldBeTrue() { Assert.IsTrue(result); }
        public void ItShouldBeFalse() { Assert.IsFalse(result); }
    }
}

And the output:

Story is A new tic tac toe board is empty
  In order to to start a new game
  As a player
  I want to have an empty board

      With scenario new board
        Given a new board                 => Passed
        When checking if board is empty   => Passed
        Then it should be true            => Passed
Story is After placing a first X
  In order to to make my first move
  As a player
  I want to have a non empty board

      With scenario new board
        Given a new board               => Passed
          And placing a first x         => Passed
        When checking if board is empty => Passed
        Then it should be false         => Passed

I tried to emulate the style in the StoryQ examples but I’m not sure if I am using StoryQ as efficiently as possible.

NSpec

Pros:

  • concise
  • easy to get started with
  • makes me think of RSpec
  • it kicks ass

Cons:

  • can’t use MS test runner
  • must manually rebuild test library before each NSpecRunner run
  • tests will fail if some conventions aren’t followed (ie: underscores in method names)

StoryQ

Pros:

  • uses MSTest or NUnit test runners and standard test attributes
  • great tools
  • focus on user stories

Cons:

  • not sure if I am using it correctly
  • very verbose

Conclusion

For my conclusion read the tl;dr section at the start of this post.

8 comments

  1. BDD is nice in theory but in practice (in the development of LOB application), in my opinion, it suffers from the same problems as tools generating code from UML and the likes. In essence, “BDD tests” only validate your comprehension of the specifications, and tests that in the end your code matches your specifications and not the client’s. So having to write more code first, before showing the result to the client and having to refactor more code every time the comprehension between the client and coder changes.

    I still believe that the most basic definition of agile means the shortest amount of time should be spent between receiving the specs from the client and showing them your resulting comprehension in the form of working software he can play with, while still producing quality code through practices like TDD. If the argument is turned around, saying that the client/analyst/business analyst should write the xSpec, I would only respond that these actors will start using BDD tools the same day they start using our Issues and Bugs Tracking system, which is never.

    Having said this, every software framework in existence should have it’s suite of BDD tests. When the specs come from the coder, the best tools for any coder to understand how and when to use your code should be written in a language that coders understand.

    Documentation, specifications, UML et al, xSpec etc… are all trying to unite and unify the intangible ideas of the client to the tangible world of the digital software and are all failing in the same manner. In the end they go contrary to the essence of agile which is to let go deterministic specifications, and adopt a more open ended world.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s