Skip to content
javahacks.dev
JVM & Performance

Java Benchmarking Mistakes That Invalidate Your Results

Common Java benchmarking mistakes: no warmup, dead code elimination, constant folding, single runs, GC noise, wrong machine and misread averages.

8 min read
Summarize in AI
Flat illustration of a flame graph with one hot frame, with the OpenJDK, OpenTelemetry and Prometheus logos.

A benchmark is an experiment, and an experiment on the JVM has more ways to lie than most. The code you wrote is not the code that runs: the JIT compiler rewrites it, removes parts it can prove are useless, folds constants and recompiles it three times while your timer is running. A benchmark written as a loop around System.nanoTime() can report that an operation takes zero time, or twice as long as it does, or twenty times faster than it will in production, and each of those numbers will look plausible. This post lists ten mistakes that invalidate Java benchmark results, shows several with real measurements from running the code, and ends with a checklist. The right tool for the job is JMH, covered in writing a correct JMH microbenchmark; the point here is to recognize what it protects you from. Timings below were measured on a 10-core machine with JDK 17 and vary from machine to machine.

1. Timing a cold JVM

The JVM starts by interpreting bytecode. Hot methods are compiled by the C1 compiler, then re-compiled by C2 after more profiling, and the timing of the first thousands of calls is nothing like the steady state. A benchmark that runs once measures class loading and interpretation. Run the same measured section repeatedly and look at each round:

src/main/java/dev/javahacks/bench/Bench.javajava
public class Bench {

    static int work(int n) {
        int s = 0;
        for (int i = 0; i < n; i++) {
            s += i % 3;
        }
        return s;
    }

    public static void main(String[] args) {
        for (int r = 0; r < 6; r++) {
            long t = System.nanoTime();
            int x = 0;
            for (int i = 0; i < 2000; i++) {
                x += work(10_000);
            }
            long us = (System.nanoTime() - t) / 1000;
            System.out.println("run " + r + ": " + us + " us (result " + x + ")");
        }
    }
}
Text
run 0: 32706 us (result 19998000)
run 1: 25997 us (result 19998000)
run 2: 57499 us (result 19998000)
run 3: 44623 us (result 19998000)
run 4: 30559 us (result 19998000)
run 5: 30295 us (result 19998000)

The same work took between 26 and 57 milliseconds. A single timing of run 2 would have declared this code twice as slow as run 1. The first rounds also include compilation. Warm up with enough iterations that compilation settles (JMH defaults to several warm-up iterations of a second each), then measure.

2. Dead code elimination

If the result of a computation is never used, the JIT may delete the computation. A benchmark that discards its result can end up timing an empty loop:

java
long t = System.nanoTime();
for (int i = 0; i < 2000; i++) {
    work(10_000);                       // result thrown away
}
System.out.println("discarded: " + (System.nanoTime() - t) / 1000 + " us");

On the machine above this still took about 28,000 us, so work was not eliminated: its loop has no side effect but the compiler did not prove it removable in this shape. That is the unsettling part. Whether the elimination happens depends on the compiler version, the surrounding code and the method's size, so the same benchmark can be right today and measure nothing after a JDK upgrade or a refactor. The defense is to consume every result: return it from the benchmark method, or pass it to a Blackhole in JMH, which the compiler cannot see through.

3. Constant folding

If the inputs are constants, the JIT can compute the answer at compile time. Math.sqrt(2.0) in a loop costs nothing after folding, and a benchmark that measures it reports an operation faster than a memory access. In the example above, work(10_000) with a constant argument invites the compiler to specialize. Make inputs come from fields that the compiler cannot prove constant: in JMH, a non-final field annotated @State that is read inside the benchmark. A static final is the opposite of what you want.

4. A single run and no variance

One number is an anecdote. The table above shows a two-fold spread in one process in one minute. Run multiple forks (separate JVMs), because each JVM compiles in its own order and can settle into a different profile, and report the mean with its error or the distribution, not the best run. If two variants differ by less than the run-to-run noise, you have measured nothing. Say "A and B are not distinguishable at this precision" and stop. Quote the percentile that matters to users: for latency, the p99, not the average.

5. Measuring the wrong thing

A microbenchmark tells you how long a snippet takes in isolation. The question is usually how fast the system is. Three common gaps:

  • The snippet is 2 percent of the request. A 50 percent improvement of it is a 1 percent improvement of the product. Find where time goes first with tracing or a profiler, as in finding slow endpoints with Micrometer and tracing.
  • The benchmark runs hot in cache with one input, while production touches many inputs and misses the cache constantly.
  • The benchmark excludes the work around the operation: serialization, locking, allocation. Measure the whole path you care about.

Benchmark the thing at the level where a decision depends on it: a library choice at the call site, a system change with a load test.

6. Garbage collection noise

Allocation costs time later, in garbage collection, and the cost may land in a different measurement than the allocation that caused it. A variant that allocates 10 times more can look equal in a short run, and then pay in pauses. Look at allocation, not only time. JMH's GC profiler prints bytes allocated per operation:

Shell
java -jar benchmarks.jar -prof gc
Text
Benchmark                       Mode  Cnt   Score    Error   Units
Parse.fast                      avgt   10   <ns>     <ns>    ns/op
Parse.fast:gc.alloc.rate.norm   avgt   10   <bytes>  <bytes>  B/op

The B/op row is the number to compare, and the placeholders stand for the values your run prints. Use the same heap size and collector across variants, because a tiny heap makes collections frequent and drowns small differences. Techniques for reducing allocation on hot paths are in reducing allocation rates on hot paths, and reading the collector's own log in reading GC logs.

7. Noisy environment

Other processes, thermal throttling, power-saving CPU frequency changes and a browser with forty tabs all move your numbers. Laptops on battery or with a throttled CPU produce slow, erratic runs. A virtual machine or a shared CI runner has neighbors you cannot see. Reduce the noise: close other work, plug in, disable frequency scaling or pin the governor to performance on Linux, and use a dedicated machine for results you will quote. Always run the comparison variants interleaved in the same session, not one in the morning and one at night. Record the machine, JDK and flags alongside the numbers; a result without them cannot be compared.

8. Profile pollution

The JIT specializes code based on what it has seen. If a benchmark exercises a call site with one type, the compiler inlines that type's method aggressively. Run another benchmark in the same JVM that calls the same site with a second type and the site becomes bimorphic, and a third makes it megamorphic and slower. The first benchmark's result depended on the absence of the others. Run each benchmark in its own fork so one cannot pollute another's profile; JMH does this by default, and a hand-rolled harness running several cases in one main does not. A consequence is that results measured on a clean benchmark can be better than in an application whose call sites see many types, which is one more reason to confirm a micro result at the system level.

9. Unrealistic data

Sorting an already sorted array, parsing the same JSON string a million times, hashing a short constant and hitting one database row all produce flattering numbers. Branch predictors learn a repeated pattern, caches hold the one value, and the JIT specializes on it. Use data with the distribution of production: varied sizes, random keys, a realistic proportion of misses. Generate it with a fixed seed so runs are comparable, and make it large enough that it does not fit in the CPU cache if production data does not. For anything that touches storage or the network, a micro benchmark with local mocks measures none of the real cost.

10. Misreading the result

Results can be right and still misread. A faster operation in nanoseconds does not matter at 10 calls per second. Throughput and latency are different questions: average time per operation hides a tail of slow calls. Comparing ns/op between two machines or JDKs is comparing the machines. And a result that looks too good, such as an operation taking 0.3 ns, is a signal that the work was eliminated, not that you found a miracle: a single cycle at 3 GHz is 0.33 ns, so anything below about 1 ns per iteration means the loop was removed or folded. Treat implausible numbers as bugs in the benchmark until proven otherwise.

A checklist

  1. Use JMH, not a hand-written timing loop.
  2. Warm up until compilation settles, and measure several iterations.
  3. Run several forks and report variance.
  4. Consume every result with a return value or Blackhole.
  5. Take inputs from @State fields, not constants.
  6. Compare allocation (-prof gc) as well as time.
  7. Quiet the machine and record the environment.
  8. Use realistic data with a fixed seed.
  9. Confirm a micro result with a system-level test before acting on it.
  10. Be suspicious of numbers below a nanosecond and differences below the noise.

The JMH samples show a worked example of each hazard, and Aleksey Shipilev's JVM Anatomy Quarks explain what the compiler is doing underneath.

When this does not apply

A benchmark that compares two algorithms by orders of magnitude, a hundred times faster, survives sloppy methodology; the mistakes above matter when the gap is small. For end-to-end performance of a service, you want load testing and production metrics, not a microbenchmark at all. And for code that runs once at startup, cold performance is the real number, and measuring a cold JVM is correct.

Also inTesting
Taggedexplainer
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.