Skip to content
javahacks.dev
JVM & Performance

How to Write a Correct Microbenchmark With JMH in Java

Write a correct Java microbenchmark with JMH: project setup, state classes, Blackhole, warmup, forks, avoiding dead code elimination and reading the results.

6 min read
Summarize in AI
Flat illustration of a flame graph with one hot frame, with the OpenJDK, OpenTelemetry and Prometheus logos.

Timing a loop with System.nanoTime() produces numbers that say little. The JIT compiler compiles the code during the run, removes results that nobody uses, hoists work out of loops and specialises on the types it has seen so far, so a naive timing measures the compiler more than the code. JMH, the Java Microbenchmark Harness, exists to handle those effects: it warms the code up, runs it in fresh JVMs and gives you a statistically meaningful number with an error bar. This guide sets JMH up, writes a benchmark, explains the four rules that decide whether a result is valid, shows how to read the output and ends with what a microbenchmark cannot tell you. It uses JMH 1.37 on JDK 25.

Set up

Use the Maven archetype that OpenJDK publishes, or add the dependencies to a dedicated module:

XML
<dependency>
  <groupId>org.openjdk.jmh</groupId>
  <artifactId>jmh-core</artifactId>
  <version>1.37</version>
</dependency>
<dependency>
  <groupId>org.openjdk.jmh</groupId>
  <artifactId>jmh-generator-annprocess</artifactId>
  <version>1.37</version>
  <scope>provided</scope>
</dependency>

The annotation processor generates the harness code that runs your methods, which is why it is on the compile classpath only. Benchmarks run from a built jar, not from the IDE test runner: the IDE adds its own agents and class loading, and either changes the result. Keep benchmarks in a separate module or source set so they do not run with the unit tests and do not slow the regular build.

A benchmark

java
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(2)
@State(Scope.Thread)
public class JoinBenchmark {

    @Param({"10", "1000"})
    int size;

    List<String> parts;

    @Setup
    public void setup() {
        parts = IntStream.range(0, size).mapToObj(Integer::toString).toList();
    }

    @Benchmark
    public String stringJoin() {
        return String.join(",", parts);
    }

    @Benchmark
    public String streamCollect() {
        return parts.stream().collect(Collectors.joining(","));
    }
}

The class compares two ways of joining strings at two input sizes. @State(Scope.Thread) gives each benchmark thread its own instance, @Param runs every benchmark once per listed value, and @Setup builds the input outside the timed region. Run it with:

Shell
mvn -q package
java -jar target/benchmarks.jar JoinBenchmark

A full run with the settings above takes a few minutes, because every method, size and fork runs warm-up and measurement iterations. For a quick check while writing a benchmark, pass -f 1 -wi 2 -i 2, and use the full settings only for numbers you will quote.

The rules that matter

Consume results. If a benchmark computes a value and drops it, the JIT may decide the code has no effect and delete it, and you then measure an empty method. Return the value, or hand it to a Blackhole:

java
@Benchmark
public void many(Blackhole bh) {
    for (String p : parts) {
        bh.consume(p.length());
    }
}

Avoid constant folding. A call such as String.join(",", List.of("a", "b")) with literal arguments can be computed at compile time or by the JIT, and the result is a measurement of nothing. Take inputs from @State fields, which the JIT cannot assume are constant between iterations.

Do not loop inside a benchmark. JMH repeats the method itself, in a way it controls. A hand-written loop invites unrolling and elimination, and it changes what is measured from one operation to a mix of one and a loop. If a loop is the thing under test, return a value that depends on every iteration.

Warm up and fork. Warm-up lets the JIT reach steady state, so the first iterations, which include interpretation and compilation, do not pollute the average. Forking runs each trial in a fresh JVM, so profile pollution left by an earlier benchmark does not affect the next, and two or more forks reveal run-to-run variance that a single JVM hides.

Reading the output

Text
Benchmark                   (size)  Mode  Cnt     Score    Error  Units
JoinBenchmark.stringJoin        10  avgt   10    62.1 ±   1.8   ns/op
JoinBenchmark.streamCollect     10  avgt   10   118.4 ±   3.2   ns/op

The score is the mean and the error is a 99.9 percent confidence interval. If the intervals of two results overlap, the difference may be noise, and you cannot claim one is faster. These numbers are illustrative of the format; your own will differ with hardware and JDK. When you publish a result, state the JDK version, the CPU, the operating system and the JMH settings next to the table, so a reader can judge whether it applies to them. A single number without its error bar and context is the commonest way benchmarks mislead.

What JMH cannot tell you

A microbenchmark tells you how a small piece of code performs in isolation. It cannot tell you whether that piece matters in your application, where the real cost may be I/O, a database call or lock contention. Profile the application first, find the hot method, and only then benchmark alternatives for that method. Common mistakes that invalidate results are listed in benchmark mistakes. Also beware that microbenchmark inputs tend to be friendlier than production inputs: the same three strings joined a million times stay in the CPU cache, while real data does not.

Profilers

JMH can attach profilers. -prof gc reports allocation per operation and garbage collection time, and -prof stack samples stack traces. Allocation per operation often explains a difference that timing alone cannot: the stream version above is slower partly because it allocates more. If one variant allocates less, it will look even better under memory pressure in a real service, a topic covered in reducing allocation.

Choosing the benchmark mode

Mode.AverageTime reports time per operation and is the right default for comparing implementations. Mode.Throughput reports operations per second and suits code that runs in a tight loop, such as a parser. Mode.SampleTime records a distribution, so you see percentiles and not only the mean, which matters when a method is usually fast and sometimes slow because of allocation or a rare branch. Mode.SingleShotTime measures one cold call, which is how you measure start-up cost that warm-up would hide. Pick the mode by the question: how fast is it steady, or how fast is the first call.

Keeping results comparable over time

Store the JSON output (-rf json -rff result.json) of a benchmark run in your build artifacts and compare against the previous release. Run on the same machine type every time, because a shared CI agent has noisy neighbours that add variance. Treat a change smaller than the error bar as no change, and investigate only the differences that clearly exceed it. A benchmark that is run once and never again documents a moment, while one run per release documents a trend.

When JMH is the wrong tool

Load tests of a whole service use different tools, and performance regressions in a pipeline are better caught by a benchmark with a stable baseline than by timing assertions in unit tests. For the testing side, see the Java testing strategy. References: the JMH samples, which are the best documentation, and the JMH project page.

Also inTesting
Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.