The Mingle team is absolutely crazy about testing and automation. Outside of testing installer integrity and production data migrations, we've totally replaced the manual regression test phase of the release cycle with a massive suite of automated acceptance tests. I'm mostly going to write about how the Mingle team has overcome some of the difficulties teams encounter when working with large acceptance suites, but, before I do that, I want to talk about our attitude towards testing. Why? Because without the right approach to testing, automation is pretty much useless.

Mingle testers start working with new features even before they are built. They participate in the requirement and scope discussions. When development starts, they sit right there with the developers, analysts, and product manager. They do not go up to the 2nd floor to work on 327 Excel worksheets full of test matrices based upon some pre-written specification. Rather, they start testing the actual feature as soon as development begins. This allows our testers to be extremely informed in their testing of features. They're actually talking to the team and they're using the development builds. They have tremendous insight into what's most important and what's most likely to break. They explore the right areas of the application in all the right ways. This allows them to write the correct automated tests. It's one thing to say you automate tests, it's quite another to build a robust, well-organized, well-targeted automated suite that actually gives the entire team confidence. We've done that.

So, back to Mingle's acceptance suite... We use Selenium RC, with tests written in Ruby. At last check, the suite was over 31,000 LOC and required over 12 hours of execution time for a given platform. Multiply that by the number of platforms we support (database, operating system, Java version, ...) and you see a real problem. How do we get feedback to the team in a timely fashion. If it takes a day to get feedback on a commit, the CI (Continuous Integration) cycle is completely broken. The team will stop authoring and maintaining tests. This is always the rub with a large end-to-end acceptance suite. It is slow. The team stops caring. CI practices break down.

The first step for us to get faster feedback was to break the tests up into smaller suites that could be run in parallel. This presented two immediate problems. First, how would we maintain the suites? Second, what CI tool would be able to run the suites in parallel and then roll them up into a single result?

The suite maintenance problem we knew how to tackle. One of our testers had already implemented a means of tagging tests such that we could associate our tests with specific parts of the domain. This made it possible for a developer or tester to run small, targeted sets of tests that correlated to a specific feature. So long as we appropriately tag tests we will always have nice options for grouping tests. To extend this concept to our CI build, we just wrote a little bit of rake code that would take advantage of those same tags to build sensible suites at runtime.

Now, how to run these suites in a CI build? Initially, for each test platform, we setup 8 CC (open source CruiseControl) boxes, each running a different acceptance target. Each of those boxes was writing results to a shared CC Dashboard. 12 hours of tests split across 8 boxes was getting us feedback in approximately 90 minutes for the entire acceptance suite.

This was certainly a huge improvement, but, as you might imagine, this was far from ideal. The different CC projects were building independently and never really rolled up into a single result. There was also no guarantee that the different projects ever ran against the same source. There was quite a bit of detective work when determining a green revision or exactly when the application broke. And we were maintaining a separate CC project (along with its very own server) for each individual acceptance target. And all of this multiplied by the number of platforms we were targeting. Clearly, our CI infrastructure was not good enough.

Around 2 months ago, the Mingle team moved its CI build to new internal Studios build grid that was running early versions of our newest product, Cruise. Given the challenges of the existing infrastructure, you can imagine that the team was quite excited to move to the new product. Today, I can confidently state that Cruise changes the game in a big way for teams running large acceptance suites. Cruise's concepts of pipelines, stages, and jobs allowed us to implement our CI build exactly as we see it. There are no kludges.

To start, we defined a pipeline where our 'Acceptance' stage only runs after the 'Units and Functionals' stage has passed. Cruise makes this part quite easy. We do not need to hack around with special build targets and tasks outside of the tool to implement the pipeline concept. Cruise supports the pipeline as a first-class concept. Better yet, in our 'Acceptance' stage, we can specify any number of the smaller, split-out acceptance suites as the jobs from which that stage is composed. Cruise manages executing those jobs in parallel and rolls up the results to a single result at build plan level. For a given pipeline run, everything builds against the same source. We're no longer wasting time manually rolling up a bunch of loosely-tied results. Cruise is managing and reporting on our build exactly as we need.

Pipelines in Cruise

We've also placed a manually triggered 'Installers' stage after the 'Acceptance' stage in our pipeline. Any team member can look for a label where the 'Acceptance' stage has passed and kick-off the 'Installers' stage. This will make those installers available to the entire team.

The Cruise Grid is a huge advantage as well. Agent resource tagging allows us to more efficiently share the agent hardware across target platforms, e.g., if an agent is tagged as providing 'Ubuntu' and 'Firefox 3' it can pick up any scheduled job that requires those resources. And when the suites grow larger, or we want to test more platforms, it's fairly trivial for us to add agents to the grid and increase our capacity.

The Mingle team is extremely excited about the new Cruise product. This post only begins to dive into what Cruise offers. As you move forward with the pipeline concept and manual gates it's quite easy to see how Cruise can extend typical CI activities all the way into the deployment phase. And the dynamic build properties and RESTful API will allow emergent usages we've yet to even envision.