Skip to content
javahacks.dev
Debugging

Arthas Java Diagnostics for Live Production Troubleshooting

Use Arthas to diagnose a running JVM without restarting it: attach, find hot threads, trace slow methods, watch arguments and decompile loaded classes safely.

7 min read
Summarize in AI
Flat illustration of a stack trace of frames with a bug beside it, with the OpenJDK, Spring Boot, Sentry and IntelliJ IDEA logos.

Production is slow, one endpoint, and only on one pod. You cannot reproduce it locally, the logs do not say why, and attaching a debugger to a live service is out of the question. The usual tools each cover part of it: a thread dump shows what threads are doing now, a profiler needs a restart or a special flag, and adding a log line means a deploy that may clear the very condition you are chasing. Arthas, an open source diagnostic tool from Alibaba, attaches to a running JVM without a restart and lets you ask questions of it interactively: which method is slow, what arguments it received, what it returned, which jar a class came from. This post walks through attaching, orienting yourself, finding a slow call, looking at live data, inspecting loaded classes, the other commands worth knowing, and the safety rules that apply when the target is production. Examples use Arthas 4 and JDK 25.

Attach

Arthas is a single jar. Download it on the host (or in the container, if its filesystem allows), run it as the same user as the target process and pick the JVM from the list:

Shell
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar
Text
[INFO] arthas-boot version: 4.0.5
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 23871 dev.javahacks.shop.ShopApplication
  [2]: 23990 org.apache.catalina.startup.Bootstrap
1
[INFO] arthas home: /home/app/.arthas/lib/4.0.5/arthas
[INFO] Try to attach process 23871
[INFO] Attach process 23871 success.
[arthas@23871]$

It attaches through the JVM's attach mechanism, loads an agent into the target and gives you a prompt. The attach works only if you are the same OS user that owns the process, and in a container if the JVM's temp directory is shared, which is why a non-root user with a read-only filesystem needs a writable /tmp. If you run in Kubernetes, kubectl exec -it <pod> -- sh and run the jar from there, or use an ephemeral debug container that shares the process namespace. Detach cleanly with stop when you finish; that unloads the instrumentation.

Get oriented

The dashboard command is a live overview, refreshed every few seconds:

Text
[arthas@23871]$ dashboard -n 1
ID  NAME                      GROUP  PRIORITY  STATE      %CPU   TIME   INTERRUPTED  DAEMON
31  http-nio-8080-exec-3      main   5         RUNNABLE   63.2   0:14   false        true
28  http-nio-8080-exec-1      main   5         RUNNABLE   21.7   0:05   false        true
 9  G1 Young RemSet Sampling  -      9         RUNNABLE    1.1   0:00   false        true
Memory                       used   total  max     usage  GC
heap                         612M   1024M  1024M   59.8%  gc.g1_young_generation.count  41

Threads sorted by CPU, memory areas and GC counts in one screen. Here two request threads are doing the work, so the problem is in request handling, not in the collector. Two commands drill into threads. thread -n 3 prints the stack of the three busiest, and thread -b finds the thread that blocks others:

Text
[arthas@23871]$ thread -n 1
"http-nio-8080-exec-3" Id=31 cpuUsage=63.2% RUNNABLE
    at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:4234)
    at java.util.regex.Matcher.match(Matcher.java:1270)
    at dev.javahacks.shop.search.QueryParser.normalize(QueryParser.java:52)
    at dev.javahacks.shop.search.SearchController.search(SearchController.java:31)

A hot thread inside a regex match called from QueryParser.normalize is already a lead, obtained without stopping the service. The thread-dump approach on its own, with jstack, is compared in using jcmd, jstack and jmap safely in production, and deadlocks in particular in finding deadlocks in a thread dump.

Find the slow call

trace instruments a method and prints the time spent in each call it makes, as a tree, for matching invocations. It answers "where inside this method does the time go" without code changes:

Text
[arthas@23871]$ trace dev.javahacks.shop.search.SearchController search '#cost > 200'
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 143 ms, listenerId: 1
`---ts=2026-11-06 14:03:11;thread_name=http-nio-8080-exec-3;id=31;is_daemon=true
    `---[1204.5ms] dev.javahacks.shop.search.SearchController:search()
        +---[0.8ms] dev.javahacks.shop.search.QueryParser:parse() #29
        +---[1190.7ms] dev.javahacks.shop.search.QueryParser:normalize() #31
        `---[12.1ms] dev.javahacks.shop.search.SearchRepository:find() #34

The condition #cost > 200 prints only calls slower than 200 ms, which keeps the output to the interesting ones. The tree shows normalize took 1,190 ms of a 1,204 ms request, so the database is innocent. Here the cause is a regular expression with catastrophic backtracking on some input. Because the method came from your own code, you now know exactly where to look and what input to look for, which the next command supplies.

stack does the inverse: it prints the call path that leads to a method, useful for finding who calls an unexpected method. Both commands add overhead to the instrumented method for as long as they run, so stop them with Q as soon as you have what you need.

See the data

watch observes the arguments, return value or exception of a method call, with an OGNL expression choosing what to print:

Text
[arthas@23871]$ watch dev.javahacks.shop.search.QueryParser normalize '{params, returnObj, throwExp}' -x 2 -n 3 '#cost > 200'
method=dev.javahacks.shop.search.QueryParser.normalize location=AtExit
ts=2026-11-06 14:03:11; [cost=1190.7ms] result=@ArrayList[
    @Object[][
        @String[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!],
    ],
    @String[aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!],
    null,
]

The argument is a long run of a characters followed by !, the classic input that sends a backtracking regex into exponential time. The flags: -x 2 expands objects two levels deep, -n 3 stops after three matches, and the condition limits it to the slow calls. You found the bug's trigger from production traffic in about two minutes. Reproduce it in a unit test with that input, fix the pattern, deploy.

A caution that comes with this power: watch prints real data. Arguments may contain tokens, personal data and passwords. Treat the terminal and any recorded session as sensitive, and limit what you print with narrow expressions. The same goes for tt, which records invocations for later replay.

Check what is loaded

Arthas answers classloading questions that are otherwise guesswork. Which jar did a class come from, and is it the version you think?

Text
[arthas@23871]$ sc -d dev.javahacks.shop.search.QueryParser
 class-info        dev.javahacks.shop.search.QueryParser
 code-source       /app/shop.jar!/BOOT-INF/classes!/
 class-loader      +-org.springframework.boot.loader.launch.LaunchedClassLoader@7b1d7fff

sc (search class) with -d shows the code source and the class loader chain. sm lists a class's methods. jad decompiles a loaded class back to source, which answers "is the running code what was deployed" in one command:

Text
[arthas@23871]$ jad --source-only dev.javahacks.shop.search.QueryParser normalize

classloader -t prints the loader hierarchy as a tree. These are the same questions as in debugging classpath and class loading problems, asked of a live process rather than a restarted one. Comparing the jad output with the repository is also how you spot a stale jar deployed to one pod of a fleet, a surprisingly common explanation for "only one pod is wrong".

Other useful commands

CommandUse
vmoptionRead and change some JVM flags live, such as HeapDumpOnOutOfMemoryError
logger --name ROOT --level debugChange a log level without a restart
heapdump /tmp/heap.hprofWrite a heap dump for later analysis
profiler start / profiler stopSample CPU with async-profiler and write a flame graph
ognl '@System@getenv("APP_ENV")'Evaluate an expression against the live process
monitor -c 5 Class methodAggregate call count, failure count and average cost every 5 seconds
redefineReplace a class with a recompiled version (see the safety section)

profiler is often the quickest route from "it is slow" to a flame graph, and monitor turns a hunch into numbers: calls per interval, failures and mean latency for one method.

Safety

An agent that can read any argument and change classes is powerful, so use it with care in production.

  • Overhead. trace, watch and monitor instrument bytecode and add cost to the target methods, and watch with deep expansion on a hot method can slow a service noticeably. Use conditions, -n limits and short runs, and always end with stop or reset.
  • Access. Anyone who can attach can read memory and call methods. Restrict who may run it, keep it off the image by default (download it only during an incident) and never expose the telnet or HTTP ports it can open beyond localhost; the built-in port is unauthenticated unless you configure a password. The security basics are in Java security basics for backend developers.
  • Data. Output and session logs contain live data. Do not paste them into a ticket unredacted.
  • Mutation. redefine, ognl calls with side effects and logger changes alter the running process. A hot-patched class is not in your repository and disappears at restart. Use it to confirm a hypothesis, then fix the code properly.
  • Change control. Treat the session like a production change: someone else knows you are attached, and what you ran is written down.

Where this does not apply

If you can reproduce the problem locally, a debugger and a profiler on your machine are safer and richer. If your platform forbids attaching to processes (a read-only image with no /tmp, a distroless container with no shell) you will need an ephemeral debug container or the JVM's built-in tools instead. For continuous visibility rather than incident work, metrics and tracing are the right layer. Arthas shines as the incident tool: one running JVM, an unknown cause, and no time for a redeploy. The commands are documented at arthas.aliyun.com and the source is on GitHub.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.