A New Kind of Mocking

There are many interpretations of what unit testing means. For the scope of this post I will define unit testing as testing a class in isolation by mocking any dependencies. On the other hand an integration test involves several classes and sometimes external systems like databases or legacy systems. A healthy project would typically have a combination of both.

Unit testing is typically fast and can show precisely where the errors are, but it also requires adapting the design, often leading to very granular classes with poor cohesion. Unit tests are expensive to maintain and hinder any refactoring which involves changing class contracts. Additionally, while unit tests can verify that a single class fulfills its contract, they say nothing about the correctness of the system as a whole.

Integration tests don’t impose contracts on single classes, make refactoring much easier and leave more design freedom. On the other side, they don’t pinpoint errors and are typically slow. Slow tests lead to sporadic test runs, which lead to frequent build breaks, which lead to bugs and poor productivity.

On a recent project, due to some inexperience and poor oversight from my side, we ended up having a massive test suite with only integration tests involving the back-end, a slow legacy system. Test coverage was good, but the test suite took 40 minutes to execute. We started dwindling between breaking the tests too often or reading blogs while the test-suite was running. We even tried writing stubs for emulating the legacy systems, but the stubs became overly complicated and we had no guarantee that they faithfully reproduced the legacy system.

I wanted to try the following:

1. run the integration tests and automatically record all interaction with the slow legacy systems.
2. on any following test run, just replay the interaction

I couldn’t find anything similar, so I quickly wrote a record/replay framework that would do this for me. It required a few hacks with dynamic proxies, byte code instrumentation and serializing non-serializable classes. Soon I realized it would be more complex because the interaction traces were specific to test methods, but there were interactions outside of the test methods, static fields, fixtures shared by test classes etc. After some careful analysis and design with pen and paper I realized what were the four types of context and how to handle each one of them and hide complexity from the user. The result looks something like:

@RunWith(RecMocks.class)
public class MyIntegrationTest {
  // ...
  private SlowSystem legacy = RecMocks.recmock(new SlowSystem());

  public void testSomeStuff() {
    // ...
    legacy.doHeavyStuff();
    // ...
  }
}

We only needed to add two lines of code to each test class. First we needed to set RecMocks as the JUnit runner, then we needed to wrap the underlying system with a spy/mock. The default mode is record, so if we run the tests normally, all interaction with the underlying system will be recorded and saved to the file system. If we set the environment variable REPLAY to TRUE, all interaction with the underlying system will be hijacked and the recorded responses will be replayed instead.

We can run individual test methods in either record/replay mode. Test classes that used to take over a minute to execute now run in a fraction of a second. In addition to that all the fancy caching and lazy initialization we were doing to improve test performance is not required anymore.

The framework is still in an early stage of development and every day we find a new problem we need to fix, but so far it seems to be paying its price. If you want to check it out it is available on maven central, but I suggest you to take the source code from github instead, since it is still unlikely it will fit your project out of the box and you might need to tweak it.