Making your own simple test framework in F#

I’m currently programming in F# under Linux. This means no Visual Studio and no MSTest GUI test runner. My first reaction was to start looking at using NUnit or xUnit for my project.

I then had a thought. Why not write my own simple test “framework”. It sounded like something easy to do and pretty fun. In retrospect, it was.

Of course this is nothing like a full featured test framework but it’s enough for my purposes.

Here is the test framework:

module TestFramework

// assert functions
let assertAreEqual expected actual =
    if expected <> actual then
        sprintf "Test failed, expected %A, actual %A" expected actual
    else
        "Test passed"

let assertIsGreaterThan target actual =
     if target >= actual then
        sprintf "Test failed, expected %A to be greater than %A" target actual
     else
        "Test passed" 

// test runner
let runSingleTest (testName, testFunction) =
    sprintf "%s... %s" testName (testFunction())       

let runTests testList =
    testList |> List.map (runSingleTest)

let consoleTestRunner testList =
    runTests testList |> List.iter (printfn "%s")
    printfn "%s" "Ran all tests."

F#’s automatic generalization cuts down on explicit generics or function overloads in the assert functions.

This makes the code more succinct and development less tedious.

Right now there is only a consoleTestRunner but more types of test runners could be implemented to accommodate other interfaces like a GUI.

And here are some sample tests:

module Tests

open TestFramework

// modules under test
open HeightMap
open MidpointDisplacement

// tests included in run
let testsToRun =
    [
        "newHeightMap will return a 0 initialized height map",
        fun() ->
            let hm = newHeightMap 5
            let result = hm.Map |> Array.sum
            assertAreEqual 0.0 result;

        "set will update height map",
        fun() ->
            let hm = newHeightMap 5
            hm.Set 1 1 0.5
            assertAreEqual 0.5 (hm.Get 1 1);

        "convertFloatToRgb will convert 0.0 to r:0, g:0, b:0",
        fun() ->
            let red, green, blue = convertFloatToRgb 0.0
            assertAreEqual (0, 0, 0) (red, green, blue);

        "convertFloatToRgb will convert 1.0 to r:255, g:255, b:255",
        fun() ->
            let red, green, blue = convertFloatToRgb 1.0
            assertAreEqual (255, 255, 255) (red, green, blue);                                               

        "convertFloatToRgb will convert 0.5 to r:127, g:127, b:127",
        fun() ->
            let red, green, blue = convertFloatToRgb 0.5
            assertAreEqual (127, 127, 127) (red, green, blue);

        "middle will set the midpoint value between each corner to the average of the corners plus the result of a function",
        fun() ->
            let variationFunction x = x + 0.1
            let hm = newHeightMap 2
            hm.Set 0 0 0.5
            hm.Set 0 4 0.5
            hm.Set 4 0 1.0
            hm.Set 4 4 0.25
            middle hm (0, 0) (4, 0) (0, 4) (4, 4) variationFunction
            let middleValues = [hm.Get 0 2; hm.Get 2 0; hm.Get 2 4; hm.Get 4 2]
            assertAreEqual [0.6; 0.85; 0.475; 0.725] middleValues;
    ]

The tests themselves are represented as data, which is a common concept in functional programming. Each test is a tuple with the first argument being the name of the test as it will appear in the console during a test run.

Here is how to call the framework along with a sample test run output :

open TestFramework
open Tests

consoleTestRunner testsToRun

newHeightMap will return a 0 initialized height map… Test passed
set will update height map… Test passed
init corners will assign values to the height map corners… Test passed
convertFloatToRgb will convert 0.0 to r:0, g:0, b:0… Test passed
convertFloatToRgb will convert 1.0 to r:255, g:255, b:255… Test passed
convertFloatToRgb will convert 0.5 to r:127, g:127, b:127… Test passed
set will correctly change the values of an heightmap… Test passed
middle will set the midpoint value between each corner to the average of the corners plus the result of a function… Test passed
Ran all tests.

Since I’m just beginning with F#, I had my code reviewed on codereview.stackexchange.com and had some great answers. I can’t recommend this site enough for when you are starting out with a new language.

Using CS figures names for test data

Here’s something that falls more along the lines of an interesting thing to do rather than a good practice.

When writing unit tests we often need to create test data to populate our test objects. Our Person or Client objects needs to have a name, a birth date and so on.

I’ve stumbled on many practices for doing this: writing meaningless data, the developer’s own name, names from a popular fiction series, etc.

One thing I’ve taken a fancy to is using the names of historical computer science figures. I find that a lot of developers tend not to know the great pioneers in our field and this can be a fun way to introduce some of these people. While this knowledge isn’t necessary, I think we should still have a basic grasp of the history of our field.

This is pretty common in other fields. You would be hard pressed to find a physicist who doesn’t know who Niels Bohr or Isaac Newton are.

Here are some of the names I have been using:

Ada Lovelace, she wrote the very first algorithm intended to be carried out by a machine!

Alan Turing, known for the Turing machine and the Turing test. During the second World War his work was instrumental in breaking German ciphers.

Dennis Ritchie, author of the C programming language and co-author of the Unix operating system.

Edsger Dijkstra, known for his classic graph traversal algorithm, the dining philosophers problem and his article against the GOTO statement.

Grace Hopper, while many know her as the first person to come up with the word “bug”, much more importantly she wrote the first compiler.

John von Neumann, an early pioneer who was involved with some of the first computers. He is know for the Von Neumann computer architecture.

Tony Hoare, amongst other thing, he came up with the Quicksort algorithm.

Testing that an exception isn’t thrown in C#

In my previous post, Testing for exceptions in C#, I mentioned how to create an Assert Extension class to check that an exception is thrown, much like in NUnit.

You can also create a method to test that an exception isn’t thrown, be it a general or specific exception. NUnit includes such a method and in the interest of completion I will give an example.

Most of the time though, you should only be testing for the correct behavior or testing that a specific exception is being raised.

Almost every case where people are looking to test that a specific exception isn’t thrown can be changed and the changed tests will end up delivering better value.

With that out of the way, here is how you would implement it:

public static void DoesNotThrows<T>(Action expressionUnderTest, string exceptionMessage = "Expected exception was thrown by target of invocation.") where T : Exception
{
    try
    {
        expressionUnderTest();
    }
    catch (T)
    {
        Assert.Fail(exceptionMessage);
    }
    catch (Exception)
    {
        Assert.IsTrue(true);
    }

    Assert.IsTrue(true);
}

Here is how you would use it:

// Testing that no exception is being thrown
[TestMethod]
public void ExampleDoesNotThrows()
{
    int someInteger = 10;
    AssertExtension.DoesNotThrows<Exception>(() => someInteger.ToString());
}

// Testing that correct behavior will not throw an exception
[TestMethod]
public void ExampleDoesNotThrowsNoException()
{
    // Arrange
    var classUnderTest = new StringlyType();
    classUnderTest.Setup();
    int toStringlyfy = 1;

    // Act && Assert
    AssertExtension.DoesNotThrows<NullReferenceException>(() => classUnderTest.Stringlyfy(toStringlyfy));
}

// Testing that another exception than the one we expected isn't thrown
[TestMethod]
public void ExampleDoesNotThrowsWithSpecificType()
{
    // Arrange
    var classUnderTest = new StringlyType();
    int toStringlyfy = 1;

    // Act && Assert
    AssertExtension.DoesNotThrows<IndexOutOfRangeException>(() => classUnderTest.Stringlyfy(toStringlyfy));
}

You can refer to this previous post for more context on the test methods.

A DoesNotThrows method could be used to replace a test that does not have any asserts and whose only way of failing is through raising an exception.

In any case, when you are about to use a DoesNotThrows, stop and really think it through.

Testing for exceptions in C#

The traditional method to test for exceptions with the Microsoft unit testing framework is to use the ExpectedException attribute.

This method has shortcomings. The biggest is that it checks the whole method for the exception which can lead to false positives. Here is a contrived example to show what I am talking about.

The StringlyType class is by the way, not serious.

    // class under test
    public class StringlyType
    {
        private StringBuilder stringlyTypeBuilder = null;

        // contrived setup method
        public void Setup()
        {
            stringlyTypeBuilder = new StringBuilder();
        }

        public string Stringlyfy(object toStringlyFy)
        {
            stringlyTypeBuilder.Append(toStringlyFy.ToString());
            return stringlyTypeBuilder.ToString();
        }
    }

    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        [ExpectedException(typeof(NullReferenceException))]
        public void TraditionalExpectedException()
        {
            // Arrange
            var classUnderTest = new StringlyType();
            int toStringlyfy = 1;

            // Act
            classUnderTest.Stringlyfy(toStringlyfy);
        }

        [TestMethod]
        [ExpectedException(typeof(NullReferenceException))]
        public void TraditionalExpectedExceptionFalsePositive()
        {
            // Arrange
            var classUnderTest = new StringlyType();

            StringlyType randomClass = null;
            randomClass.Setup();

            int toStringlyfy = 1;

            // Act
            classUnderTest.Stringlyfy(toStringlyfy);
        }

The first test method will pass. We haven’t called the Setup method on StringlyType so the call to Append will throw our NullReferenceException.

On the second test method, we will get a false positive. The call to randomClass.Setup() will throw an exception and our test will be none the wiser. If we change the behaviour of StringlyType so that the StringBuilder is initialized in the constructor, the false positive will keep on telling us the code is throwing an exception.

A better way

NUnit, an open-source alternative to Microsoft’s framework, has been using a better approach for years. Their approach is having an Assert.Throws that works like an ordinary assert.

When I am in an environment that isn’t using NUnit I like to create something equivalent along the lines of:

public static class AssertExtension
    {
        public static T Throws<T>(Action expressionUnderTest,
                                  string exceptionMessage = "Expected exception has not been thrown by target of invocation."
                                 ) where T : Exception
        {
            try
            {
                expressionUnderTest();
            }
            catch (T exception)
            {
                return exception;
            }

            Assert.Fail(exceptionMessage);
            return null;
        }
    }

Since the Assert class is static we can’t create extension methods for it, hence I’m creating another AssertExtension class.

This AssertExtension class will allow to write the previous tests in the following way.

        [TestMethod]
        public void ExampleWithAssertThrows()
        {
            // Arrange
            var underTest = new StringlyType();
            int t = 1;

            // Act && Assert
            AssertExtension.Throws<NullReferenceException>(() => underTest.Stringlyfy(t));
        }

        [TestMethod]
        public void WillNotGetAFalsePositive()
        {
            // Arrange
            var underTest = new StringlyType();

            StringlyType randomClass = null;
            randomClass.Setup();

            int t= 1;

            // Act && Assert
            AssertExtension.Throws<NullReferenceException>(() => underTest.Stringlyfy(t));
        }

Notice that the second method we will not be a false positive.

This method has several advantages:

• prevents false positives
• easier to see which statement should throw an exception
• more succinct and idiomatic than using a try catch
• allows to return the thrown exception

Here is an example of the last point:

var thrownException = AssertExtension.Throws<InvalidOperationException>(
                      () => classUnderTest.Stringlyfy(toStringlyfy));

Assert.AreEqual("some message", thrownException.Message);

I made a GitHub Gist if you want to get/modify the AssertExtension class.