Bug reproduction steps

To properly fix a bug, you must be able to reproduce it. If the developers can’t reproduce it they can only guess at its cause and their corrections are just potential fixes that can’t be validated.

If someone writes a report for a bug they can’t reproduce, how can they validate that the developer assigned to the task has satisfactorily fixed the problem? They can’t. So why report and track something that can’t be closed except by a leap of faith?

Maybe because the developer will find the reproduction steps? They might, but then again who stands a better chance of doing this?

The best person to reproduce a bug is the person who first encountered it. To give an analogy, it’s like if you found a secret glade in a forest and you can’t find it again. So you ask someone who’s never been there to find it for you and you don’t give them any directions. Even if they find one on their own, is it the same one you found?

Another situation is the QA or reporter can reproduce it, but the developer can’t. This might be because the reproduction steps aren’t clear or just missing. It could also be that there is a difference between the environments or actions of the reporter and the developer.

Sometimes the bug reporter has produced the issue but hasn’t tried to reproduce it consistently. They write down with they did but haven’t tested their steps further to see if they reproduce the original results every time.

All of these situations can be greatly mitigated by writing good reproduction steps and testing them before submitting the bug report.

How to write reproduction steps:

  • Most bugs report should include reproduction steps. If reproduction can’t be achieved, in most cases, testing should continue.
  • Steps should be clear and concise.
  • Each step should be absolutely necessary to reproduce the bug. This removes clutter but also forces the reporter to distill the bug to it’s most minimal conditions, which helps to define it correctly.
  • No assumptions should be made while writing the steps. This can be very hard as we all have implicit assumptions that are not apparent at first glance. For example, if a user was created for testing purposes, how was it created? What sort of user is it? Is the username included in the report?
  • Before submitting the report, the reproduction steps themselves should be tested. Did they reproduce the issue? Were some required operations omitted because the seemed evident? For example if navigation to a screen occurred and there are several ways to navigate to this screen, which one was used?
  • During the testing of the reproduction steps the scenario should be altered a little. Does the scenario still reproduce the bug with the alterations. If so the altered steps are either unnecessary or could be generalized further. Examples of this could be testing with a different user or changing the sequence of some operations.

In addition to the reproduction steps bug reports for mobile applications should include the OS and device name.

Bug reports for web applications should include the browser and it’s version in the case of Internet Explorer or non evergreen browsers.

All bug reports should include the version number of the application that was under test as well as the environment the bug was found on: dev, qa, prod.

Taking these steps will minimize the back and forth between QA and development and will reduce the time required to close bugs.

 

Choosing a BDD framework for .Net

tl;dr

Specs aren’t written by developers: choose between SpecFlow and NBehave.

Specs are written by developers: my personal recommendation NSpec. Runner up StoryQ.

Introduction

BDD or behavior driven development is the practice of writing an executable testable specification that describes the application’s behavior. This specification is often written in a fluent interface, a DSL or in plain English (or rather close to plain English).

While in TDD the focus is on writing tests that single out units of the application, BDD is focused on writing tests on the behavior of the application. This can also be tought as testing features of the application.

Before presenting the frameworks I will make a distinction between xBehave and xSpec style frameworks.

xBehave

xBehave frameworks are about writing user level stories in a form comprehensible by anyone. These stories can be written by the users themselves or by a group consisting of developers, users and testers.

These framework typically use a story defined in a DSL close to English and then map this story to a test written in code by the developers. Cucumber is a well known example of such a framework.

xSpec

xSpec frameworks are usually about developers writing tests in code using an approach that favors testing behavior and functionality. These tests are usually closer to unit tests in both appearance and granularity but feature some differences.

Both approaches could be combined on a single project. The user stories could be converted to xBehave tests, while the developers could adopt an xSpec framework to replace or complement xUnit tests.

Available frameworks

Here is a compilation of xSpec and xBehave BDD frameworks for .Net. This list is current as of the writing of this post and while I tried my best to find most frameworks, I realized while compiling this list that there are just too many, I am sure I must have missed a few.

By the way, it’s not an error in the table, I have found two frameworks named NSpec which appear to be totally independent. The first one listed is the most active and most popular. The second one is an older project that hasn’t seen any recent activity.

Name Project development still active Type License NuGet package
SpecFlow Yes xBehave BSD style Yes
specunit-net No xSpec Apache License 2.0 No
Machine.Specifications / MSpec Yes xSpec xUnit and MS-PL Yes
NSpec Yes xSpec MIT License Yes
NSpec (older project) No xSpec zlib/libpng license No
NBehave Yes xBehave BSD 3 Yes
StoryQ No *** MIT Yes
NSpecify No, seems never to have taken off. xSpec ? No
.NetSpec No xSpec None No
xbehave.net Yes xSpec MIT Yes
SubSpec No xSpec MS-PL Yes
NJasmine Yes xSpec MIT Yes
SpecsFor Yes xSpec None Yes
Behavioral No xSpec BSD Yes
NaturalSpec Yes xSpec MIT / MS-PL Yes
BDDish No xSpec None Yes

*** This framework is harder to classify. Based on my interpretation of xSpec/xBehave and my comprehension of the framework, I personally place it in a gray zone. They suggest to start with a plain text story but it isn’t directly used by the code. The specifications are directly in C# test code contrary to other more “pure” xBehave frameworks. It is a sort of hybrid and does tend to lean much more on the xSpec side than on the xBehave side.

Narrowing the selection

First of all, I would eliminate the following frameworks from investigation: specunit-net, NSpec (older project), NSpecify and .NetSpec as they are abandoned inactive projects with most never having really made it to “production status”.

NaturalSpec is in F# which is great for those doing F#. There is a link to a post on it’s GitHub page about how to use it with C# objects, but I would personally prefer a library that targets C# for a C# project.

I wasn’t able to see much activity for BDDish (last commit to GitHub more than 2 years ago, only 3 questions on StackOverflow, not much documentation or examples). I would also skip this one.

As an unrelated side-note, out of the remaining more serious contenders xbehave.net and SubSpec are strikingly similar. At first glance, both projects seem to be closely related. If you are considering one of these two, you might as well look into the other as well.

Making a selection for an xBehave framework

The first question you need to ask yourself is if you want to do xBehave style tests in a language closer to plain text, with specs that can be written by non developers.

If that is so, your choice is pretty slim. Your choice is between SpecFlow and NBehave.

StoryQ could also be considered if SpecFlow or NBehave are not satisfactory.

Both SpecFlow and NBehave use Gherkin (the DSL used by Cucumber) to write their specs. Both store specs in *.feature files. Both also use attributes on methods when writing tests.

[Given("I am not logged in")]
public void LogOut()
{

Both have documentation available on GitHub.

SpecFlow seems to have more documentation and it is also recommended in this StackOverflow question in the top two rated answers.

On the other hand the SpecFlow doc is pretty dry on code examples. I personally preferred NBehave’s documentation over SpecFlow.

Making a selection for an xSpec framework

If you do not need for your specs to be written in a plain text like format and want to save time and work use an xSpec framework. You won’t need to write the binding code and writing tests will be easier for those already familiar with an xUnit framework.

On the xSpec side, you have a lot of options.

I narrowed my search to the following frameworks: MSpec, NSpec, SpecsFor, StoryQ and xBehave.net.

The following table compares those frameworks by considering their total number of questions on StackOverflow. I also rated the official documentation on a scale of 1 to 3, with 1 being the poorest and 3 the best score. This rating is largely subjective and represents my appreciation of the available documentation. It should be noted I prefer a style which includes code samples and which gets you rapidly on your feet.

Name Number of questions on StackOverflow Documentation
MSpec 519* 1 / 3
NSpec 62 3 / 3
SpecsFor 5 3 / 3
StoryQ 36 3 / 3
xBehave.net 5 2 / 3

*: I added the number of questions for both MSpec and Machine.Specifications. Even if just looking at the number of questions for either MSpec or Machine.Specifications, it is the clear winner here.

My personal recommendation for an xSpec framework is NSpec.

After trying each of these frameworks for a simple two tests scenario here is my personal rating:

  1. NSpec
  2. StoryQ
  3. SpecsFor

Here is a code sample of two simple tests using NSpec:

namespace NSpecTests
{
    class TicTacToe_specifications : nspec
    {
        void given_a_new_board()
        {
            it["A new tic tac toe board is empty"] = () => ticTacToeBoard.IsEmpty.should_be_true();

            context["When placing a first X"] = () => 
            {
                before = () => ticTacToeBoard.PlaceXat(1);
                it["Board should not be empty"] = () => ticTacToeBoard.IsEmpty.should_be_false();
            };
        }
        TicTacToeBoard ticTacToeBoard = new TicTacToeBoard();
    }
}

And here is the output that will be generated using NSpecRunner, NSpec default test runner.

Output :
TicTacToe specifications
  given a new board
    A new tic tac toe board is empty
    When placing a first X
      Board should not be empty

2 Examples, 0 Failed, 0 Pending

Here are the two same tests and their output in StoryQ:

namespace StoryQTests
{   
    [TestClass]
    public class TicTacToe_specifications
    {
        TicTacToeBoard ticTacToeBoard;
        bool result;

        [TestMethod]
        public void given_a_new_board()
        {
            new Story("A new tic tac toe board is empty")
                .InOrderTo("to start a new game")
                .AsA("player")
                .IWant("to have an empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeTrue)
                .Execute();

            new Story("After placing a first X")
                .InOrderTo("to make my first move")
                .AsA("player")
                .IWant("to have a non empty board")
                .WithScenario("new board")
                .Given(ANewBoard)
                .And(PlacingAFirstX)
                .When(CheckingIfBoardIsEmpty)
                .Then(ItShouldBeFalse)
                .Execute();
        }
        
        public void ANewBoard() { ticTacToeBoard = new TicTacToeBoard(); }
        public void PlacingAFirstX() { ticTacToeBoard.PlaceXat(1); }
        public void CheckingIfBoardIsEmpty() { result = ticTacToeBoard.IsEmpty; }
        public void ItShouldBeTrue() { Assert.IsTrue(result); }
        public void ItShouldBeFalse() { Assert.IsFalse(result); }
    }
}

And the output:

Story is A new tic tac toe board is empty
  In order to to start a new game
  As a player
  I want to have an empty board

      With scenario new board
        Given a new board                 => Passed
        When checking if board is empty   => Passed
        Then it should be true            => Passed
Story is After placing a first X
  In order to to make my first move
  As a player
  I want to have a non empty board

      With scenario new board
        Given a new board               => Passed
          And placing a first x         => Passed
        When checking if board is empty => Passed
        Then it should be false         => Passed

I tried to emulate the style in the StoryQ examples but I’m not sure if I am using StoryQ as efficiently as possible.

NSpec

Pros:

  • concise
  • easy to get started with
  • makes me think of RSpec
  • it kicks ass

Cons:

  • can’t use MS test runner
  • must manually rebuild test library before each NSpecRunner run
  • tests will fail if some conventions aren’t followed (ie: underscores in method names)

StoryQ

Pros:

  • uses MSTest or NUnit test runners and standard test attributes
  • great tools
  • focus on user stories

Cons:

  • not sure if I am using it correctly
  • very verbose

Conclusion

For my conclusion read the tl;dr section at the start of this post.