Skip to main content
Blog

Unit test isolation

By 31 oktober 2013januari 30th, 2017No Comments

Once upon a time we delivered our first version of our application. All units were tested thoroughly. Some had more than 20 unit tests! We were very content with our the unit test coverage. The test architect most of the times looked like this 🙂

A minor change request came in to change some of the units. We did the changes in code and ran the unit tests. All tests failed.
We were not so content now with our test suite. We had a hard time changing all the tests because of unclear unit test objectives and hard to read tests, but most of all because it were a lot of tests. General feeling: 🙁

A large testset with high coverage sounds nice, but can result in maintainability issues. Ideally if a single feature of a unit has changed at most a single unit test should fail. This is called unit test isolation, which will be the topic of this blog post.

Simple example


First a simple example to make the aspects of unit test isolation more concrete.
Suppose you have a method named doSomething that takes two arguments x and y. The allowed value for x and y are 0 <= x <= 2 and 0 <= y <= 2. The following two (junit) tests have been defined.

@Test
public void testSmallX() {
    int x = 0;
    int y = 2;

    result = doSomething(x, y);

    assertEquals(0.5, result.getX(), 1e-8);
    assertEquals(2.5, result.getY(), 1e-8);
}

@Test
public void testSmallY() {
    int x = 2;
    int y = 0;

    result = doSomething(x, y);

    assertEquals(2.5, result.getX(), 1e-8);
    assertEquals(0.5, result.getY(), 1e-8);
}

When the allowed range for x is now changed to 0.5 <= x <= 1.5, both tests will fail. These unit tests are therefore not isolated from each other. Furthermore from the set of failing tests alone it is hard to see what the problem is; both fail so it could be because of x or could be y. To fix the unit tests such that they adhere to the new requirement (i.e. 0.5 <= x <= 1.5) both tests have to be adjusted.

Unit tests are meant to test a certain feature of a unit in isolation. If you need to change a lot of unit tests if a single feature changed, the unit tests become a burden.

Thus, the level of isolation of your unit tests play a big part in the usefulness and maintainability (or burden) of your unit test set.

If you would have defined the above example like below the test set would be more isolated.

@Test
public void testSmallXb() {
    int x = 0;
    int y = 1;

    result = doSomething(x, y);

    assertEquals(0.5, result.getX(), 1e-8);
}

@Test
public void testSmallYb() {
    int x = 1;
    int y = 0;

    result = doSomething(x, y);

    assertEquals(0.5, result.getY(), 1e-8);
}

Notice that in the simple example both the input preparation part of the unit test and the output verification assertions were not limited to the test objective.

Test objective

The test objective is very crucial part of isolation. If you cannot break up the tests needed for a unit into a mutually exclusive set of test objectives it will limit the degree of isolation possible for a unit.

Generally a junit test is build up in three parts:

  1. Input preparation (Given)
  2. Unit call (When)
  3. Output verification (Then)

All parts are closely related to the test objective.

In the next sections I will discuss the parts of a unit test more thoroughly.

Given


The test objective in the example of the testSmallX unit test is small x. However, the values assigned to y is the upper boundary of y. In the improved testSmallXb() I have assigned y a different value that is not close to the upper boundary of y.

In general you should only give inputs boundary values if its important for the test objective. These might be simple values like in the example, but could also be a large data structure where all values combined result in certain behavior. Try to give all inputs that are not involved in the test objective common or are average values. In fact these inputs should not influence the outcome.

Keep in mind that unit tests also have to be changed in the future. Therefore make as clear as possible that those values are not used for the test objective, for example for string values use “something”.

When


You might wonder, why address the unit code when you want to improve your unit test isolation. Whether you like it or not, the unit code plays also a role in test isolation. When a method does not have clear responsibilities or has side effects, it might become very hard to create isolated tests (probably also test objectives will not be mutually exclusive).
For example when the doSomething method from the example changes x and y depends on an external factor (or maybe using a static util method),
you have no guarantee that your test will always pass.

Then


When looking at testSmallX unit test, it is clear that y should not be asserted because it is not part of the test objective.
In more complicated situations it is probably not that simple. Combinations of outputs will cover the test objective and it might be harder to realize isolation. However, when it is very hard to isolate your tests with respect to assertions, your test objective might be too large. You can split the test objective in several parts.
Do not try to assert every output. Instead assert the outputs that are involved in the objective.

Unit test vs. integration test


A unit test covers a single unit while an integration test test multiple units.
In a unit test you would probably use some mocking framework to be able to isolate you test while in an integration test you would simply take the actual code.
Generally when involving more units, the probability that the test will fail because of a change becomes higher and also adjusting the test will need more effort.
Furthermore when you have integration and unit tests in your test set you probably will have some overlap.
Also the test preparation part of your test set generally becomes quite big for integration tests.
I am not opposed to integration tests, but you should take extra care that you don’t end up with tests that are a burden instead of helpful.

Conclusion


Coming back to isolation of tests, in fact the same holds for unit tests as well as for integration test.

  1. Define a clear test objective (for integration tests the objective should be about integration)
  2. Give only specific values to inputs that are directly involved in the objective.
  3. Give most common or average values to the inputs that are not directly involved in the objective.
  4. Assert only outputs that are relevant for your test objective

The key aspect of isolation is clear mutually exclusive test objectives.