Skip to content
javahacks.dev
Debugging

How to Use jcmd jstack and jmap Safely in Production

Use jcmd, jstack and jmap on a live Java process without causing an outage: which commands pause the JVM, how to attach in containers and what to capture.

7 min read
Summarize in AI
Flat illustration of a source file compiled to bytecode and run by the JVM, with the OpenJDK, Gradle, IntelliJ IDEA and Apache Maven logos.

When a Java service misbehaves, the JDK ships the tools to look inside, and some of them can make things worse. A heap dump of a 16 GB heap on a struggling service can turn a slowdown into an outage. This post lists what each tool does, which are cheap, which are costly and how to run them in practice. It complements debugging with only logs and uses the vocabulary of JVM memory areas.

Find the process

Shell
jcmd -l

lists Java processes with their pid and main class. The tools attach to a running JVM through an attach mechanism, which requires the same user and the same view of the file system (a /tmp/.java_pid<pid> socket). If the process runs as app and you are root, use sudo -u app jcmd .... A "Unable to open socket file" error means the user or the temp directory differs.

The cheap ones

Thread dump.

Shell
jcmd <pid> Thread.print
jcmd <pid> Thread.print -l       # include lock information

or jstack <pid>. The JVM reaches a safepoint, collects every thread's stack and resumes. For a typical service this takes milliseconds to a few hundred milliseconds. Take three dumps five seconds apart: threads in the same place in all three are stuck, as in fixing deadlocks. Thread dumps include the deadlock report.

Heap summary and flags.

Shell
jcmd <pid> GC.heap_info
jcmd <pid> VM.flags
jcmd <pid> VM.native_memory summary      # needs -XX:NativeMemoryTracking
jcmd <pid> VM.system_properties
jcmd <pid> VM.uptime

These read counters and settings and do not pause the application in a meaningful way.

Class histogram without a collection.

Shell
jcmd <pid> GC.class_histogram -all

The -all form includes unreachable objects and skips the full GC, so it shows what exists now; it is cheap. A histogram of the biggest classes is often enough to spot a leak.

The costly ones

Class histogram with live objects. jmap -histo:live <pid> and jcmd <pid> GC.class_histogram (without -all) run a full GC first so only live objects are counted. Full GC pauses the application for the duration, from tens of milliseconds to seconds on a large heap. Avoid it on a service that is already struggling.

Heap dump.

Shell
jcmd <pid> GC.heap_dump /var/tmp/heap.hprof

This forces a full collection and writes the entire heap to disk. The file is about as large as the used heap, writing it takes time proportional to that, and the application is paused while it runs. Preconditions: free disk space of at least the heap size, one instance (take it from a replica taken out of the load balancer if you can), off-peak if possible. The file contains everything in memory: passwords, tokens and personal data. Handle it like a secret and delete it after use.

To capture one automatically at failure, use -XX:+HeapDumpOnOutOfMemoryError, covered in GC overhead limit exceeded.

JFR: continuous low-overhead profiling

Shell
jcmd <pid> JFR.start name=incident duration=120s filename=/var/tmp/incident.jfr settings=profile
jcmd <pid> JFR.dump name=incident filename=/var/tmp/incident.jfr

JDK Flight Recorder records CPU samples, allocation, locks, I/O and GC events with low overhead (the default settings aim for roughly one percent or less). It is the best tool for "what is the CPU doing" and "what allocates". Open the file in JDK Mission Control. Many teams keep a rolling recording on all the time.

Containers

The tools attach through the process's file system and process namespace. In Docker or Kubernetes:

Shell
kubectl exec -it pod/shop-7d9 -- jcmd 1 Thread.print

The process is usually pid 1 in the container. The image needs the jcmd binary, which means a JDK, not a JRE. Many slim images lack it. Options: use a JDK base image, an ephemeral debug container that shares the process namespace (kubectl debug --target), or jlink a runtime with jdk.jcmd. Write dumps to a volume that outlives the pod.

What to avoid

  • jmap -dump on a huge heap during peak traffic.
  • kill -3 on a service whose standard output is not captured (the dump goes where the stdout goes).
  • Running tools as a different user and then concluding the process is unreachable.
  • Leaving heap dumps on disk or attaching them to tickets.
  • Attaching a debugger (JDWP) to a production process; it can suspend threads.

Practice on a deadlock you made yourself

The first time you read a thread dump should not be during an incident. This program deadlocks two threads on purpose:

java
public class DeadlockDemo {

    static final Object A = new Object();
    static final Object B = new Object();

    public static void main(String[] args) throws Exception {
        Thread t1 = new Thread(() -> lockInOrder(A, B), "worker-1");
        Thread t2 = new Thread(() -> lockInOrder(B, A), "worker-2");
        t1.start();
        t2.start();
        t1.join();
    }

    static void lockInOrder(Object first, Object second) {
        synchronized (first) {
            sleep(200);
            synchronized (second) {
                System.out.println(Thread.currentThread().getName() + " got both");
            }
        }
    }

    static void sleep(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }
}
Shell
java DeadlockDemo.java &
sleep 2
jcmd $(jcmd -l | grep DeadlockDemo | cut -d' ' -f1) Thread.print | grep -A12 "Found one Java-level deadlock"

The dump ends with a section that names both threads, the monitor each one holds and the monitor each one waits for. That cycle, worker-1 waiting for what worker-2 holds and the reverse, is how a deadlock looks in production too, with waiting to lock and locked lines and the frames that took each lock. Practicing on a ten-line program makes the same pattern recognizable in a 400-thread dump. The follow-up on fixing the ordering is in deadlocks with thread dumps.

A collection script that is safe to run under pressure

During an incident, people type commands from memory and forget the ones that matter. A script that does the cheap collection in a fixed order, and refuses to do the costly step without being asked, removes the guesswork:

Shell
#!/bin/sh
# usage: collect.sh <pid> [--heap]
set -eu
PID="$1"
OUT="/var/tmp/diag-$(date +%Y%m%dT%H%M%S)-$PID"
mkdir -p "$OUT"

jcmd "$PID" VM.version      > "$OUT/version.txt"
jcmd "$PID" VM.flags        > "$OUT/flags.txt"
jcmd "$PID" VM.uptime       > "$OUT/uptime.txt"
jcmd "$PID" Thread.print -l > "$OUT/threads-1.txt"
sleep 5
jcmd "$PID" Thread.print -l > "$OUT/threads-2.txt"
jcmd "$PID" GC.heap_info    > "$OUT/heap-info.txt"
jcmd "$PID" GC.class_histogram -all | head -40 > "$OUT/histogram.txt"
jcmd "$PID" JFR.start name=collect duration=60s settings=profile filename="$OUT/profile.jfr" > /dev/null

if [ "${2:-}" = "--heap" ]; then
  FREE_KB=$(df -Pk /var/tmp | awk 'NR==2 {print $4}')
  HEAP_KB=$(ps -o rss= -p "$PID")
  if [ "$FREE_KB" -lt "$HEAP_KB" ]; then
    echo "not enough free disk for a heap dump; skipping" >&2
  else
    jcmd "$PID" GC.heap_dump "$OUT/heap.hprof"
  fi
fi
echo "collected in $OUT"

The heap dump is behind a flag and behind a disk check that uses the resident size as an upper bound on the file. The recording starts last and runs for a minute in the background, so the script returns while data is still being collected. Keep the script in the repository next to the runbook, so it is reviewed and so nobody has to remember the commands at 3 a.m.

Kubernetes without a JDK in the image

Slim runtime images have no jcmd. An ephemeral debug container that shares the target's process namespace gives you the tools without changing the pod:

Shell
kubectl debug -it pod/orders-7d9f6b-x2k4q \
  --image=eclipse-temurin:25-jdk \
  --target=orders \
  --share-processes -- sh

# inside the debug container
jcmd -l
jcmd 1 Thread.print -l

Attach needs the same user ID as the target process and access to its temporary directory, which is a common reason jcmd answers with "Could not attach". If the application runs as a non-root user, start the debug container as that user (the --custom flag in recent kubectl versions accepts a profile that sets runAsUser), and read the [reference for kubectl debug](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/) for what your cluster version supports.

Reading a recording without a GUI

JDK Mission Control is the friendliest viewer, and the jfr command in the JDK answers common questions from a terminal, which helps on a bastion host:

Shell
jfr summary incident.jfr
jfr print --events jdk.ExecutionSample incident.jfr | grep "at " | sort | uniq -c | sort -rn | head -15
jfr print --events jdk.JavaMonitorEnter,jdk.ThreadPark incident.jfr | grep -E "duration|monitorClass" | head

The second command counts frames across the CPU samples, so the methods at the top are where the application spends its time. The third surfaces lock contention and parking. Neither replaces an analysis in Mission Control, and either one turns "the CPU is high" into "this method, in this package, accounts for most of the samples", which is what you need to decide the next step.

Where this does not apply

On a managed platform that does not give a shell, rely on the platform's profiler and on Actuator endpoints. The [jcmd reference](https://docs.oracle.com/en/java/javase/25/docs/specs/man/jcmd.html) lists every command, and the JDK Flight Recorder documentation covers JFR.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.