Skip to content
javahacks.dev
Testing

How to Find and Fix Flaky Tests in Java Projects Fast

Find and fix flaky Java tests: detect them with reruns and random order, learn the causes with examples and quarantine safely.

9 min read
Summarize in AI
Flat illustration of a test report with one failing test, with the OpenJDK, JUnit5, Docker and Jenkins logos.

The build is red. A developer reruns it and it goes green. Nobody investigates, because it has done this before. That moment, repeated a few hundred times, is what a flaky test costs: not the minutes of the rerun, but the habit it builds. Once a team learns that red sometimes means nothing, real failures are dismissed as flakes too, and the test suite stops being evidence. Flaky tests are not mysterious. They have a short list of causes, and each has a standard fix. This post shows how to detect flaky tests rather than rely on memory, then walks through the causes with a failing and a fixed version of each, and finishes with the process that keeps a suite clean. It applies to unit and integration tests alike; for browser tests specifically, the causes are extended in the Playwright for Java guide.

Detect them on purpose

A test you have not noticed being flaky cannot be fixed. Four techniques find them.

Rerun failures and look at the report. Maven Surefire can rerun failing tests, and distinguishes a test that eventually passed from one that always failed:

pom.xmlXML
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <rerunFailingTestsCount>2</rerunFailingTestsCount>
    </configuration>
</plugin>
Text
[WARNING] Flakes:
[WARNING] OrderServiceTest.sendsConfirmationEmail:74 » Wanted but not invoked
[ERROR]   Run 1: OrderServiceTest.sendsConfirmationEmail:74 Wanted but not invoked: mailSender.send(...)
[INFO]    Run 2: PASS

The build passes, and the Flakes section lists the test. The danger is exactly that: reruns make the build green and the problem invisible. Use them to *report* flakiness, publish the list of flaky tests from every build, and treat each entry as a defect. Gradle has a test retry plugin with similar output.

Run in random order. Tests that depend on the order of execution fail when the order changes. JUnit 5 supports random method order with a configuration property, and Surefire can randomise class order:

src/test/resources/junit-platform.propertiesproperties
junit.jupiter.testmethod.order.default=org.junit.jupiter.api.MethodOrderer$Random
Shell
mvn test -Dsurefire.runOrder=random

The test output prints the seed, so a failure can be replayed in the same order: -Djunit.jupiter.execution.order.random.seed=1234567.

Repeat suspects. A test that fails one time in fifty needs fifty runs to show it. @RepeatedTest(100) on a suspect, or a loop in CI, gives a rate:

java
@RepeatedTest(200)
void sendsConfirmationEmail() { /* ... */ }

Stress the machine. Many flakes only appear under load, when threads are scheduled differently. Run the suite on a busy machine, with parallel execution on, or in a container limited to one CPU, and a latent race shows up.

Keep the evidence: record the failure message, the seed and the environment, as for any bug report.

Cause 1: shared state between tests

The most common cause. A static field, a singleton, a database row or a file written by one test and read by another makes the result depend on order.

java
class CounterTest {
    private static final List<String> events = new ArrayList<>();      // shared across tests

    @Test
    void recordsOne() {
        events.add("one");
        assertThat(events).hasSize(1);          // fails if another test ran first
    }

    @Test
    void recordsTwo() {
        events.add("two");
        assertThat(events).hasSize(1);
    }
}

The fix is to make state per test: an instance field (JUnit creates a new instance per test by default, see the test instance lifecycle), a fresh object in @BeforeEach or a local variable. For databases, give each test its own data with unique keys, roll back a transaction after each, or clean the tables the test touched. In Spring tests, a mutable singleton bean in the cached application context carries state between test classes; reset it in @BeforeEach, or mark the context dirty only when you must, since @DirtiesContext makes the suite much slower.

Cause 2: time

Tests that depend on the current time or time zone fail near midnight, at month end, during a daylight saving change or on a CI machine in another zone.

java
@Test
void invoiceIsDueThirtyDaysLater() {
    Invoice invoice = service.issue(order);               // uses LocalDate.now() inside
    assertThat(invoice.dueOn()).isEqualTo(LocalDate.now().plusDays(30));    // flaky at midnight
}

If the clock ticks over to the next day between the two now() calls, the test fails once in a blue moon. Worse are tests that assume a time zone. The cure is to take time as a dependency:

java
@Test
void invoiceIsDueThirtyDaysLater() {
    Clock fixed = Clock.fixed(Instant.parse("2026-12-31T23:59:59Z"), ZoneOffset.UTC);
    var service = new InvoiceService(fixed);

    assertThat(service.issue(order).dueOn()).isEqualTo(LocalDate.of(2027, 1, 30));
}

The pattern, with injected Clock, is described in replacing java.util.Date with java.time and in deterministic tests for time and randomness. Also fix the default Locale and TimeZone for the test JVM (-Duser.timezone=UTC -Duser.language=en) so formatting does not vary by machine.

Cause 3: asynchronous code and sleeps

A test that triggers asynchronous work and then checks the result immediately races with the work. The fix people reach for first is a sleep:

java
@Test
void publishesAnEventAfterPayment() throws Exception {
    service.pay(order);
    Thread.sleep(500);                                       // passes on a fast machine, fails on a slow one
    verify(publisher).publish(any(OrderPaid.class));
}

A sleep is a guess about timing. Too short and the test is flaky; long enough to be safe and the suite is slow. Replace it with a condition that is polled until it holds, with a timeout. Awaitility is the standard library for that:

java
@Test
void publishesAnEventAfterPayment() {
    service.pay(order);

    await().atMost(Duration.ofSeconds(5)).untilAsserted(() ->
            verify(publisher).publish(any(OrderPaid.class)));
}

untilAsserted retries the assertion every 100 ms by default until it passes or five seconds elapse. On a fast machine the test finishes in milliseconds; on a slow one it waits as long as it needs, up to the bound. See Awaitility for asynchronous tests. Where possible, avoid waiting at all: make the asynchronous component injectable with a synchronous executor in tests (Runnable::run), or return a CompletableFuture the test can join. For code with schedulers, inject the scheduler, so the test advances time instead of waiting, as in testing async code with schedulers.

Cause 4: concurrency bugs in the code

Sometimes the test is right and the code has a race. A test that passes nine times in ten is reporting a real defect: an unsynchronised counter, a check then act on a map, a missing happens-before edge.

java
@Test
void countsAllRequests() throws Exception {
    var counter = new RequestCounter();
    runConcurrently(8, () -> { for (int i = 0; i < 10_000; i++) counter.increment(); });

    assertThat(counter.value()).isEqualTo(80_000);       // fails sometimes: lost updates
}

Do not "fix" such a test by retrying it or loosening the assertion. Fix the code, as covered in Java race conditions and how to spot them, and keep the stress test as a regression guard. Writing such tests reliably is a skill of its own, with latches to start threads together and enough repetition to expose the race, described in testing concurrent code to find race conditions.

Cause 5: resource clashes

Fixed ports, shared temp files and external services are shared resources. Two test classes running in parallel, or two builds on the same CI agent, both bind port 8080 or write /tmp/export.csv:

java
Path out = Path.of("/tmp/export.csv");           // another run may be using it

Use random ports (server.port=0, RANDOM_PORT), per test temporary directories (JUnit's @TempDir) and unique names. For containers, let Testcontainers choose the host port and read it from the container object. Tests that call a real external service fail when the service is slow, down or rate limiting; stub it, as in mocking external APIs with WireMock. A related cause is a container that reports ready before the service inside is: wait for a real readiness condition, not a port being open.

Cause 6: randomness and ordering assumptions

Random data that occasionally hits an edge case, a UUID that happens to start with a digit where the code assumed a letter, or a name with an apostrophe that breaks a query, fails rarely and unreproducibly. Seed the random generator in the test and log the seed, or use a property based tool that reports the seed on failure, as in property based testing with jqwik. Another quiet source: iteration order. HashMap and HashSet have no defined order, and a test asserting on map.toString() or on the first element of a set can pass on one JVM and fail on another. Assert with order insensitive matchers (containsExactlyInAnyOrder) or sort first, and use LinkedHashMap where order is a requirement.

Triage: is it the test or the code

When a flake is found, decide where the defect is before changing anything.

ObservationLikely locationAction
Fails only after another test, passes aloneThe test (shared state)Isolate state
Fails under load or on a slow machineThe test (timing) or the code (a race)Replace sleeps; stress the code
Fails near midnight or in another time zoneThe test (uses the real clock)Inject a clock
Fails on a specific port or pathThe test or the environmentRandom ports, temp directories
Fails with a stack trace in production codeThe codeTreat as a real bug
Fails once a month, no patternUnknownAdd logging, capture seeds and the environment; repeat in CI

Add information when a failure is rare: log the seed, the thread, the time, the data. A flake you cannot reproduce can still be understood from enough evidence captured at the moment it occurs.

Quarantine without losing the test

A flaky test you cannot fix today still hurts every build. The pragmatic response is quarantine: move it out of the gating suite into a separate run, with an owner and a deadline, instead of deleting it or leaving it in. JUnit tags make this easy:

java
@Tag("quarantine")
@Test
void sendsConfirmationEmail() { /* known flaky: TICKET-482 */ }
pom.xmlXML
<configuration>
    <excludedGroups>quarantine</excludedGroups>
</configuration>

The main build skips quarantined tests and a separate job runs them nightly and reports the pass rate. The rules that make it work: every quarantined test has a ticket and an owner, the list is visible on a dashboard and a test that stays in quarantine past its deadline is deleted or fixed. Quarantine without a deadline is a graveyard, a suite that silently shrinks. Equally, auto retry in the gating build, without reporting, is a way of living with the problem; if you use retries, publish the flake list and track its size over time as a team metric.

Prevent them

Most flaky tests could be avoided by the habits that make tests good to start with. Isolate state per test. Inject time, randomness and threads. Prefer deterministic seams to waits. Run tests in random order and in parallel in CI from the start, because flakiness found in week one is cheap and flakiness found in year three is a project. Review tests for sleeps and shared statics as you would review production code for SQL injection: it is a pattern with a known failure. And watch the numbers: the percentage of builds that needed a rerun, and the age of the oldest quarantined test.

When this does not apply

If every failure reproduces, the suite is deterministic and this does not apply. Tests against genuinely unreliable infrastructure, such as a shared staging environment, flake because of the environment, and the fix is to test against something you control. For the tools, see the Surefire documentation on rerunning failing tests and the Awaitility documentation.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.