Saturday, May 30, 2009

Your First Concordion.Net Project (Part 5)

Running Specs with Gallio

Part 1Part 2Part 3Part 4

We have our specifications developed so now we want to actually run this beasty! After trying to develop my own test runner for Concordion I decided to settle on writing a plugin for the Gallio framework instead.

Gallio is an open testing framework that runs all sorts of .Net test frameworks on a number of different platforms. You can find out more about its adaptability and rich reporting capabilities here.

So how do we get Gallio to run our specifications? We need to use a plugin. In the Concordion distributable file you will notice a directory called Gallio.ConcordionAdapter. This is the plugin specifically written to allow Concordion specifications to be run on Gallio.

What I like to do when running specifications is to start off by running them with Gallio.Echo. This is the command-line executable and I find this easiest to debug problems with. There are a number of other runners for Gallio (like Icarus (the GUI); or runners for Visual Studio, Test-Driven.Net, etc.).

I like to create the specification file in the root directory of the specification project so let’s do that right now. Here’s the file in the Calculator.Spec project folder


The file is called run-spec-with-echo.cmd … as you can tell I really like long and descriptive names!

Now let’s fill this up with a command-line. We will need to tell Gallio where to find the Gallio.ConcordionAdapter plugin and where to find our specification assembly. This folder has contents like this:


To tell Gallio to find this directory we add the command-line option


This tells Gallio to search in this directory for a .plugin file that it can use to work with the Concordion plugin. Now we need to tell Concordion where our specification assembly is.


Since the batch file is in our project directory but our specification assembly is in the project output folder we need to add the relative path to the specification assembly. Unless otherwise specified, Gallio.Echo’s working directory is the current folder. Our command file should now resemble this

set GALLIO_PATH=C:\Dev\concordion-net\tools\Gallio-trunk\bin
%GALLIO_PATH%\Gallio.Echo.exe /pd:c:\Concordion\Gallio.ConcordionAdapter bin\debug\Calculator.Spec.dll

So let’s try and run the file! If you do you should see the following results on screen


Uh oh … Houston we have a problem! Notice that there is a failure in our specification. If you remember back to the last post we intentionally made our ArithmeticTest fixture a dumb one so that we could see it fail.

Fixing the Fixture

Red-Green-Refactor is the name of the game here! Ok, let’s fix up our test by filling out the Calculator class in our Calculator project and calling that from the specification to perform the arithmetic operations. Our Calculator class should resemble this:

public class Calculator
public long Add(long first, long second)
return first + second;

public long Subtract(long first, long second)
return first - second;

public long Multiply(long first, long second)
return first * second;

public long Divide(long first, long second)
return first / second;

And we will want to modify our fixture like so

public class ArithmeticTest
public long firstOperand;
public long secondOperand;

public long Addition(long first, long second)
return new Calculator().Add(first, second);

public long Subtraction(long first, long second)
return new Calculator().Subtract(first, second);

public long Multiplication(long first, long second)
return new Calculator().Multiply(first, second);

public long Division(long first, long second)
return new Calculator().Divide(first, second);

Ooops … we forgot something! Did you notice? For the above code to work we need to add a reference to the Calculator project in the Calculator.Spec project


Good, now we have our reference so it should compile and run successfully now. Let’s see how it goes.


Woot! All of our tests now pass! Let’s take a peek at the output now.


Notice the green surrounding the results now? That means our Calculator class successfully performs the arithmetic we want!


If you want to see the source code for this series of posts you can download it here.

This sample was pretty basic but it should be enough to get you up and running!

Thanks for trying out this tool, I hope you enjoy it. Stay tuned, there will be more to come on advanced features of Concordion!


  1. Great stuff! I'm definitely excited to see where this all can go. Thanks for the series of posts and all the hard work behind them!

  2. Thanks jeremy ... positive feedback much appreciated. I hope it helps you out some.

  3. Great work Jeffrey, this sounds very cool. I've checked out your sample (the zip file) and the console runner works but it does not found any test (0 run, 0 passed, 0 failed). This happens with Gallio version 3.0.5 and 3.0.6. What version do you use?
    Thanks :-)

  4. Hi Carlos,

    The concordion adapter (Gallio plugin) is written for a specific version of Gallio.

    Check in the download section for the specific Gallio version.

    You can download Gallio using this link:

    Hope this help

  5. Thanks Erlis, Gallio works fine with Concordion as stated in the downloads page.
    Wonderful :-)

  6. Test runner is now running the tests but the html output does not contain red or green, just the same input html. It is not rendering the html. In addition, when a test fail console does not show the reason, it just says "test failed". Is it possible to see the failure reason?

  7. @Carlos: Would you go tot he Google Group and post your question there please? It's tough to respond appropriately to a bug on a blog posting! Thanks!

  8. This was a great series of posts, it helped to clarify Concordion for me. I've been trying it out at home and it's looking good.

  9. @mikedm: Thanks! I love positive feedback! If you have build 41 of Concordion you may want to download build I just put it up this morning as there was a bug preventing breadcrumbs from being created properly. Any other questions feel free to post to the discussion group.

  10. Am I missing something?? You create these .html files but don't ever publish them as a website?? I want to use this like FitNesse and it falls short if you can't publish the specification pages?
    I completed the tutorial and am left wondering how to use this...

  11. If all my product owners have to install VS or Gallio then this tool is not useful in an Agile environment.

  12. Hi Doug,

    Thank you for your comments.

    This tutorial was only meant to show beginners how to use Concordion, not all of its possible applications/usages.

    First off, only the developers (and possibly a continuous integration machine) need to have VS and/or Gallio installed. Your product owners likely only care for the results, not the process.

    Secondly, If you are working in an agile environment then you probably have a continuous integration process. Simply run the specifications as part of your build process and deploy/publish the static html files to a web site or file share or as part of your distributable for your product owners to view. No need to a web server, no need for special software to run it other than a web browser to view the static html. Concordion even puts breadcrumbs and other navigation features in the result so that the pages behave like a web site. If you look at the Concordion specifications you will see great care is taken to organize the specs in a logical manner.

    Regarding Fitnesse, the trouble with Fitnesse is that it is tough to setup and use in an agile environment, especially one that uses continuous integration.

    I am a Fitnesse user myself and I can say through experience that using Fitnesse in an agile environment can be done but is very ugly for several reasons. There is poor support for source control, the built in web server makes running as part of a CI process awkward and refactoring scripts is immensely difficult.

    If you would like to see a better description of why Concordion was created and why people who use Concordion do so as a preference over Fitnesse you can read on this web page:

  13. I had to add the command line option /runner:local in the file run-spec-with-echo. Otherwise I got the result "0 run, 0 passed, 0 failed".

  14. Thanks for the great explanations for setting all this up. I have gotten it nearly working, but the final step, getting the reports to display correctly, is causing me some trouble. In my project, Firefox displays the Gallio output files as raw XML (no style information). I can either change the extension to html (insteal of xhtml, which is what gallio appends), or edit the report and manually add xmlns="" to the html tag, and that fixes the problem, but of course that means I'm noodling with the generated report. If I put xmlns="" in the html tag of the .html file in my project, Gallio overrides it by putting xmlns="" in my head tag. ARGH!