Asserting exceptions in unit tests is often forgotten or thought of as not useful. However a (very) crucial part of the flow of an application is contained in the bad weather flow and I think this should be tested in the unit tests. I have seen multiple solutions in asserting your exceptions. Some have big downfalls and some are in my opinion plain wrong.
Unit test mindset
The same as with all tests, a unit test should be pessimistic. Meaning, it is better if the test fails and the behavior is as expected than that the test succeeds and the behavior is not as expected. The reason behind this is that people tend to run the unit tests and only look at the failed unit tests (which is of course a good thing, because nobody in their right mind would waste his time looking at arbitrary unit tests).
Therefore, a false positive test is the biggest risk when writing unit tests. Keep this in mind when looking at the below JUnit examples.
For ease of reading, the org.junit.Assert class has been statically imported.
Assert exceptions in catch statement
@Test
public void example1() {
try {
PersonWithoutFeet.walk();
} catch(WalkException exc) {
assertEquals("no feet", exc.getMessage());
}
}
The code from example1 looks in first instance OK. The person without feet throws the WalkException and the exception message is asserted.
However, if the PersonWithoutFeet does walk somehow and therefore does not throw an exception, the unit test will pass while it should fail (i.e. a false positive).
In example2 code solves the shortcoming from example1.
@Test(expected = Exception.class)
public void example2() {
try {
PersonWithoutFeet.walk();
} catch(WalkException exc) {
assertEquals("no feet", exc.getMessage());
}
}
However, Example2 introduces two new problems. The communication between the PersonWithoutFeet and the writer of the unit test has not been very good. Instead of throwing the WalkException, the UnableToWalkException is thrown (not related in class hierarchy). While the test fails when no exception is thrown, it succeeds when another exception is thrown (uh-oh false positive) and the exception message has not been asserted!
Furthermore if the PersonWithoutFeet does throw the WalkException, the test will fail. Therefore, this unit test actually does not test the required behavior.
The code in Example 3 fixes these problems
@Test(expected = WalkException.class)
public void example3() {
try {
PersonWithoutFeet.walk();
} catch(WalkException exc) {
assertEquals("no feet", exc.getMessage());
throw exc;
}
}
Another approach you could take is adding a call to the fail method in the try block like in Example 4.
@Test
public void example4() {
try {
PersonWithoutFeet.walk();
fail();
} catch(WalkException exc) {
assertEquals("no feet", exc.getMessage());
}
}
Both example 3 and 4 are good examples of unit tests that assert the exception. They safeguard false positives when the implementation of PersonWithoutFeet changes.
JUnit ExpectedException rule
Since JUnit 1.4.8 the feature to use rules has been added. This makes it possible to add some standard behaviour before and-or after each junit test in your test class. This makes it possible to define some generic logic to verify a certain behavior. You can use junit rules actually for many situations, for example initializing an in-memory database, preparing connections, mocking standard classes, etc.
JUnit comes out of the box with the ExpectedException rule. The ExpectedException rule makes it possible to verify an exception and/or message that is thrown out of a unit test.
In this post I will mainly focus on the ExpectedException rule (and more general asserting exceptions). The biggest advantage in using the ExpectedException rule is that you make the unit test more readable and it saves you unit test code (and also time to implement probably).
Some people think this is the solution to avert the issues in the above mentioned examples. However, You can still make similar mistakes. In the next couple of examples I will show some of the mistakes.
Example 2b is the original example 2 unit test implemented with the expected exception.
@Rule
public ExpectedException thrown = ExpectedException.none();
@Test
public void example2b() {
thrown.expect(Exception.class);
try {
PersonWithoutFeet.walk();
} catch (WalkException exc) {
assertEquals("no feet", exc.getMessage());
}
}
It is not exactly the same, but still when another exception is thrown, the test will not fail (assuming this is the objective of the unit test). Of course it contains the same problem as in the original.
However, there’s less code and it looks more readable.
It is not possible to implement example 4 with the ExpectedException rule. However, example 3 can be similarly implemented like this:
@Test
public void example3b() {
thrown.expect(WalkException.class);
thrown.expectMessage("no feet");
PersonWithoutFeet.walk();
}
Uh oh, we have introduced another problem.
The expectMessage expresses a contains relation. If the exception message changes, but still contains “no feet”, the test passes while we wanted to assert the complete error message. Thus another false positive.
So if you do the following, any exception results in a passing test!
@Test
public void example5() {
thrown.expectMessage("");
PersonWithoutFeet.walk();
}
This should be prevented. In the current implementation of ExpectedException the “contains” relationship is chosen. It is however very easy to create a custom Rule to assert the complete message.
If also error codes or something else that has been added to an exception/exceptions in your application you could also easily create a Rule for that.
Conclusion
You might wonder whether asserting exception messages is useful in your unit tests, because it requires quite some maintenance when messages change. In my opinion it is a crucial part of your applications business logic and you should therefore test it. Whether you test it in unit, integration or application tests is debatable.
When you use JUnit tests to test exception messages (or other information contained in the exception) take care that you actually test what you want to test and prevent possible future false positives.
