Debug Memory Leak Java Services Running in Production
Debug a Java memory leak in production: confirm the growth, capture histograms and dumps safely, read the dominator tree and fix the holder.

The memory chart is a sawtooth that climbs a little higher every day. Each garbage collection frees less than the one before, until a pod is restarted at 4 a.m. or the service dies with OutOfMemoryError: Java heap space. A memory leak in Java is not memory the program forgot to free, since the collector frees everything unreachable. It is objects that are still reachable and no longer useful, held by something that never lets go. Finding the holder in a production service is a procedure, and the first rule of it is to be cheap and careful before you do anything expensive: a heap dump of a large heap pauses the application for seconds. This post walks through confirming that it is a leak, taking cheap histograms, capturing a dump safely, reading it, recognizing the usual holders, fixing and proving the fix, and what to do when you cannot take a dump. The output below is from a real run of a small leaking program on JDK 17.
Is it a leak
Growth is not proof. A cache that fills toward its limit, a warming JIT and a heap that has been allowed to grow all look like a rising line. The distinguishing measurement is memory after full garbage collections, the retained floor, plotted over days. If that floor rises, something retains. Read it from the GC log or from Micrometer's jvm.memory.used for the old generation after collection (the jvm.gc.live.data.size metric reports exactly this):
[2026-11-06T04:11:02.114+0000][gc] GC(881) Pause Full (Allocation Failure) 3911M->3688M(4096M) 412.311ms
[2026-11-06T09:02:41.903+0000][gc] GC(1203) Pause Full (Allocation Failure) 3934M->3797M(4096M) 455.870msMemory after the full collection (the second number) rose from 3,688 MiB to 3,797 MiB over five hours, and each collection frees less. That is a leak signature: the live data set is growing. Reading these lines is covered in reading GC logs to find the real problem. A flat floor with a high peak means the heap is undersized or the allocation rate is high, which is a sizing problem and not a leak, see Java heap size in a container. Also confirm it is the heap. A container killed with OOMKilled while the heap chart is flat is native memory, direct buffers or thread stacks, which needs a different tool: native memory tracking.
Step 1: cheap histograms
A class histogram counts live objects per class without writing a file, and it is the safest first look. Reproduce the idea with a program that leaks on purpose:
import java.util.*;
public class Leak {
static final Map<String, byte[]> CACHE = new HashMap<>(); // never evicted
public static void main(String[] args) throws Exception {
for (int i = 0; ; i++) {
CACHE.put("session-" + i, new byte[10_000]);
if (i % 5000 == 0) {
long used = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
System.out.println("entries=" + CACHE.size() + " usedMB=" + used / (1 << 20));
}
}
}
}java -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof Leak &
jcmd $! GC.class_histogram | head -6 num #instances #bytes class name (module)
-------------------------------------------------------
1: 47815 201325008 [B ([email protected])
2: 20997 671904 java.util.HashMap$Node ([email protected])
3: 27726 665424 java.lang.String ([email protected])One class owns almost everything: [B, the byte[], 201 MB of the heap. The runner's own output confirms the climb, 49 MB at 5,000 entries and 98 MB at 10,000, linear in entries, until:
entries=15001 usedMB=147
entries=20001 usedMB=200
entries=25001 usedMB=245
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/leak.hprof ...
Heap dump file created [268380961 bytes in 0.261 secs]Take two histograms some minutes apart and compare. The leaking class is the one whose instance count grows steadily. jcmd <pid> GC.class_histogram forces a full collection first (so it shows live objects only) and pauses the application briefly, which on a large heap can be seconds; run it on one instance, outside peak traffic, and know the pause cost beforehand. The tools and their safety are covered in using jcmd, jstack and jmap safely in production.
Common top entries tell a story. byte[] and String growth means cached or accumulated data; an application class with millions of instances means a collection of your own objects; ClassLoader or Class growth means a metaspace-adjacent leak, see metaspace and class loader leaks.
Step 2: a heap dump, carefully
A histogram says what; a dump says why, because it keeps the reference graph and lets you ask who holds an object. The cost is real. Dumping a 4 GiB heap writes a 4 GiB file, pauses the JVM for the duration (seconds to a minute) and the file contains every secret in memory: tokens, passwords, personal data. Precautions:
- Take it from one instance, after removing it from the load balancer if you can, so users do not wait on the pause.
- Have the disk space (about the size of the used heap) on a volume that survives the pod, not the container's temporary layer.
- Treat the file as sensitive, restrict access and delete it after analysis.
- **Prefer the automatic dump on
OutOfMemoryError** (-XX:+HeapDumpOnOutOfMemoryError) for services that die: it captures the moment of failure with no extra step. Set the path to a mounted volume.
To take one on demand:
jcmd <pid> GC.heap_dump -all=false /dumps/shop-$(date +%s).hprofThe default (live objects only) forces a full collection before dumping, so the file is smaller and the leak is not drowned by garbage. Copy the file off the machine and analyze it there.
Step 3: analyze
Open the file in Eclipse Memory Analyzer (MAT), documented at eclipse.dev/mat. Its leak suspects report is a good first screen, and its dominator tree is the most useful view: it lists objects by the memory that would be freed if they were removed. The question it answers is who is keeping this alive.
For the example above, the dominator tree shows one java.util.HashMap retaining 99 percent of the heap, held by a static field Leak.CACHE. The path to GC roots makes it explicit: Leak.class (the class object, a GC root) to the static field CACHE to the map to its nodes to the arrays. In a real service the chain is longer, but the method is the same: find the biggest retained object, then follow "path to GC roots" excluding weak references to see which reference keeps it alive. The step-by-step version with screenshots of the views is in capturing and analyzing a heap dump with Eclipse MAT.
Usual holders
Most leaks are one of a short list.
- Unbounded caches and maps. A
HashMapused as a cache with no size limit or expiry, the example above. Use a real cache with eviction (Caffeine) or a bounded structure. - Listeners and callbacks never unregistered. A long-lived publisher holds references to short-lived subscribers.
- ThreadLocals. Values set on pooled threads and never removed survive for the life of the thread; see ThreadLocal leaks and how to avoid them.
- Static collections that only grow: registries, "seen" sets, metrics keyed by an unbounded id (a tag with a user id creates a new time series per user).
- Unclosed resources. Connections, statements, streams; a pool whose borrowed connections are never returned holds them and everything they reference. The diagnostic for that is in monitoring and debugging JDBC connection leaks.
- Class loader leaks in application servers on redeploy, where something in the shared loader references a class from the old application.
- Queues that grow because the consumer is slower than the producer. Not a leak in code, but the same heap shape.
Step 4: fix and prove
Fix the holder: bound the cache, unregister the listener, remove the thread local in a finally, close the resource.
static final Cache<String, byte[]> CACHE = Caffeine.newBuilder()
.maximumSize(10_000)
.expireAfterAccess(Duration.ofMinutes(30))
.build();Then prove it, because a leak fix that is not verified is a hypothesis. Run the fixed build under realistic load for long enough that the old one would have grown, and compare the retained floor: it should be flat. Take a histogram at the start and after an hour; the suspect class's instance count should plateau. Add a regression guard where you can: a test that runs the operation many times and asserts the live set after a GC stays bounded, or an alert on the floor rising over days.
Without a dump
Sometimes a dump is not possible: the heap is too big, the data too sensitive, the platform forbids it. You still have options. Use histograms over time and diff them. Enable allocation profiling with Java Flight Recorder (jcmd <pid> JFR.start settings=profile), whose old-object sampling event (jdk.OldObjectSample) records objects that survived many collections and the path to the root, at very low overhead. Add metrics for the structures you suspect (the size of each cache and queue) and chart them. Bisect by traffic: replay one endpoint at a time against a staging instance and see which one moves the floor. Read the commonly missed place, the configuration: an unbounded pool or a missing eviction setting is cheaper to check than a dump.
Where this does not apply
If the process is killed with exit code 137 and the heap is small, it is not a heap leak; check native memory and the container limit. If memory is stable and the service is merely slow, you want a CPU profile or a look at pauses, not a dump. And if the growth stops at a plateau well below the maximum, you are looking at caching and the JIT, which is normal. The Java troubleshooting guide has a chapter on diagnosing memory leaks that complements this procedure.


