Junit 5 timeouts – Fail if test not executed within time

In this article we will see what are the Junit 5 timemouts, how to use @Timeout in writing test cases, how to use global timeouts and how to disable timeouts globally with several examples.

1. Junit 5 timeouts

Junit 5 timeouts allows to declare that a test, test factory, test template, or lifecycle method should fail if its execution time exceeds a given duration and will get java.util.concurrent.TimeoutException as results. The time unit for the duration defaults to seconds but is configurable.

Following are the several ways Junit 5 supports for the timeouts.

1. Using @Timeout annotation.
2. Using assertTimeout() or assertTimeoutPreemptively() .
3. Using configuration parameters to specify global timeouts .

1.1. Using @Timeout annotation

The following example shows how @Timeout is applied test methods and test class and test lifecycle.

Example 1 :

If @Timeout declared at class level, same timeouts applied for test methods, nested tests and test factories, but not applied timeout for @BeforeEach, @BeforeAll, @AfterEach, @AfterAll life cycle callback methods.

@Timeout(7) // each test fails if execution time exceeds 7 seconds
public class Junit5_Timeout_Annotation_ClassLevel_Test {
	
	@BeforeEach
    void setUp() throws Exception {
		TimeUnit.SECONDS.sleep(10); // waits 10 seconds
		System.out.println("@BeforeEach");
		System.out.println("-------------------------");
    }
	
	@BeforeAll
    static void setUpBeforeAllTests() throws Exception {
		TimeUnit.SECONDS.sleep(10);
		System.out.println("@BeforeAll");
    }
	
	@Test
	void test_Add() throws Exception {
		
		TimeUnit.SECONDS.sleep(10);
		assertEquals(5, MathUtil.add(3, 2));
		System.out.println("test_Add()");
	}
	
	@Test
	void test_Multiply() throws Exception {
		
		TimeUnit.SECONDS.sleep(5); // waits 5 seconds
		System.out.println("test_Multiply()");
	}
}

Output :

@BeforeAll
@BeforeEach
-------------------------
@BeforeEach
-------------------------
test_Multiply()

Results in eclipse :

Example 2 :

If @Timeout declared at @BeforeAll and exceeds timeout, execution will be aborted, skips the rest of test cases.

public class Junit5_Timeout_Annotation_Test {
	
	@BeforeAll
	@Timeout(4) // skip all tests if this method execution time exceeds 4 seconds
    static void setUpBeforeAllTests() throws Exception {
		TimeUnit.SECONDS.sleep(3);
		System.out.println("@BeforeAll");
    }
	
	@BeforeEach
	@Timeout(4) // fail all tests if this method execution time exceeds 4 seconds
    void setUp() throws Exception {
		TimeUnit.SECONDS.sleep(3);
		System.out.println("@BeforeEach");
    }
	
	
	@Test
	@Timeout(9) // this test fails if execution time exceeds 9 seconds
	void test_Add() throws Exception {
		
		TimeUnit.SECONDS.sleep(10); // waits 10 seconds
		System.out.println("test_Add()");
	}
	
	@Test
	@Timeout(9) // this test fails if execution time exceeds 9 seconds
	void test_Multiply() throws Exception {
		
		TimeUnit.SECONDS.sleep(8); // waits 8 seconds
		System.out.println("test_Multiply()");
	}
	
	@Test
	@Timeout(value = 900, unit = TimeUnit.MILLISECONDS)
	void test_isPrime() throws Exception {
		
		// fails if execution time exceeds 900 milli seconds
		TimeUnit.MILLISECONDS.sleep(1000); // waits 1000 milli seconds
		assertEquals(10, MathUtil.multiple(5, 2));
		System.out.println("test_isPrime()");
	}
}

Output :

@BeforeAll
@BeforeEach
@BeforeEach
test_Multiply()
@BeforeEach

1.2. assertTimeout() or assertTimeoutPreemptively()

1.2.1. assertTimeout() – Test method will be failed if not executed within time, but assertTimeout() completes its execution process.

1.2.2. assertTimeoutPreemptively() – Test method will be failed if not executed within time, assertTimeoutPreemptively() will be aborted execution process. This is useful to test transaction management tests, to ensure that abort the transaction operation if it exceeds the time.

@Test
	void test_Add() throws Exception {
		
        assertTimeout(Duration.ofSeconds(5), () -> { 
        	           TimeUnit.SECONDS.sleep(4);
                       assertEquals(5, MathUtil.add(3, 2));
                       System.out.println("test_Add()");
                    });
	}
	
	@Test
	void test_Multiply() throws Exception {

		// fails but completes the process, see output and run duration
		assertTimeout(Duration.ofSeconds(5), () -> { 
		        TimeUnit.SECONDS.sleep(6);
	            assertEquals(5, MathUtil.add(3, 2));
	            System.out.println("test_Multiply()");
	         });
	}
	
	@Test
	void test_assertTimeoutPreemptively() throws Exception {
		
        // fails but abort the process, see output and run duration
        assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { 
	        TimeUnit.SECONDS.sleep(6);
            assertEquals(5, MathUtil.add(3, 2));
            System.out.println("test_assertTimeoutPreemptively()");
        });
	}

Output :

test_Add()
test_Multiply()

Execution Time and status of each Test :

1. test_Add() – Passed – 4.035 s
2. test_Multiply() – Failed – 10.011 s
3. test_assertTimeoutPreemptively() – Failed – 5.016 s

1.2.3. Declaring @Timeout on a @TestFactory method checks that the factory method returns within the specified duration but does not verify the execution time of each individual DynamicTest generated by the factory.  We can use assertTimeout()  or assertTimeoutPreemptively() for that purpose.

Junit 5 Timeout for Dynamic tests example :

@TestFactory
	@Timeout(5)
	Collection<DynamicTest> test_dynamicTestsTimeouts() {
		return Arrays.asList(
				
				dynamicTest("1st dynamic test", () -> {
					TimeUnit.SECONDS.sleep(3);
					assertEquals(5, MathUtil.add(3, 2));
					System.out.println("Dynamic Test 1");
				}),
				
				dynamicTest("2nd dynamic test", () -> {
					TimeUnit.SECONDS.sleep(2);
					assertEquals(6, MathUtil.multiple(3, 2));
					System.out.println("Dynamic Test 2");
				}));
	}
	
	@TestFactory
	Collection<DynamicTest> test_dynamicTests_AssertTimeouts() {
		return Arrays.asList(
				
				dynamicTest("3rd dynamic test", () -> {
					assertTimeout(Duration.ofSeconds(5), () -> { 
	        	           TimeUnit.SECONDS.sleep(10);
	                       assertEquals(5, MathUtil.add(3, 2));
	                       System.out.println("Dynamic Test 3");
	                    });
				}),
				
				dynamicTest("4th dynamic test", () -> {
					assertTimeoutPreemptively(Duration.ofSeconds(5), () -> { 
	        	           TimeUnit.SECONDS.sleep(10);
	                       assertEquals(5, MathUtil.add(3, 2));
	                       System.out.println("Dynamic Test 4");
	                    });
				}));
	}

Output :

Dynamic Test 1
Dynamic Test 2
Dynamic Test 3

Execution Time and status of each Test :

1. 1st dynamic test – Passed – 3.006 s
2. 2nd dynamic test – Passed – 2.004 s
3. 3rd dynamic test – Failed – 10.004 s
4. 4th dynamic test – Failed – 5.003 s

1.3. Junit 5 global timeouts

Configuration parameters can be used to specify global tiemouts. For example to specify timeouts for all the @BeforeEach methods in entire test application set the value for configuration property junit.jupiter.execution.timeout.beforeeach.method.default

Following can be used to specify global timeouts. Values can be set in junit-platform.properties file under test/resources folder or can be set in maven sure-fire plugin configurationParameters  properties or can set as System property value can be provide from maven command.

1. junit.jupiter.execution.timeout.default – Default timeout for all testable and lifecycle methods.
2. junit.jupiter.execution.timeout.testable.method.default – Default timeout for all testable methods.
3. junit.jupiter.execution.timeout.test.method.default – Default timeout for @Test methods.
4. junit.jupiter.execution.timeout.testtemplate.method.default – Default timeout for @TestTemplate methods.
5. junit.jupiter.execution.timeout.testfactory.method.default – Default timeout for @TestFactory methods.
6. junit.jupiter.execution.timeout.lifecycle.method.default – Default timeout for all lifecycle methods.
7. junit.jupiter.execution.timeout.beforeall.method.default – Default timeout for @BeforeAll methods.
8. junit.jupiter.execution.timeout.beforeeach.method.default – Default timeout for @BeforeEach methods.
9. junit.jupiter.execution.timeout.afterall.method.default – Default timeout for @AfterAll methods.
10. junit.jupiter.execution.timeout.aftereach.method.default – Default timeout for @AfterEach methods.

Example junit-platform.properties :

 junit.jupiter.execution.timeout.beforeeach.method.default=200 ms
 junit.jupiter.execution.timeout.test.method.default=10 s

2. Disable @Timeout globally

When we are debugging test cases timeout may influence the results of test execution. For example assume that, when writing tests we set the timeout value 10 seconds to complete transactions related tests, but later it increased to 20 seconds. All the tests may impact and mark the test as failed although all assertions were met.

JUnit Jupiter supports the junit.jupiter.execution.timeout.mode configuration parameter to configure when timeouts are applied.  There are three modes.

1. enabled
2. disabled
3. disabled_on_debug

Example junit-platform.properties :

junit.jupiter.execution.timeout.mode=disabled

Conclusion

In this article we have seen what are the Junit 5 timemouts, how to use @Timeout in writing test cases, how to use global timeouts and how to disable timeouts globally with several examples. Declarative timeouts are an experimental feature, but this is very useful feature to handle timeouts globally.

References

1. Junit 5 document