Skip to content
javahacks.dev
Debugging

How to Debug a Production Issue With Only Logs and Metrics

Debug a production Java problem without a debugger: build a timeline from logs, follow one request with a correlation id and raise log levels safely.

7 min read
Summarize in AI
Flat illustration of a stack trace of frames with a bug beside it, with the Docker, IntelliJ IDEA, Sentry, OpenJDK and Spring Boot logos.

You cannot attach a debugger to production, and the bug exists only there. What you have is logs, metrics and a shell. Used with discipline they are enough for most problems. This post is a procedure for that situation, with the logging setup that makes it work. For the commands that inspect a running JVM safely see jcmd, jstack and jmap, and to read errors in the logs how to read a stack trace.

Step 1: define the symptom and the time

Write down what is wrong, for whom and since when: "about 2 percent of checkout requests return 500 since 14:20 UTC". Then ask what changed near that time: a deployment, a configuration change, a traffic spike, a dependency's incident, a certificate expiry, a database failover, a scheduled job. Most production incidents correlate with a change. Metrics (error rate, latency percentiles, pool usage, GC, CPU) show when and how the curve moved. If the change was a deployment, the diff is a short list of suspects.

Step 2: narrow with the logs

Search for the error at the time of the first occurrence. Count by type:

Shell
grep "ERROR" app.log | sed -E 's/[0-9a-f-]{36}/<id>/g' | cut -c1-160 | sort | uniq -c | sort -rn | head

This groups repeated errors after replacing identifiers. The most frequent distinct message is usually the symptom; the earliest is often the cause. In a log platform, do the same with a query on the error class and group by time.

Step 3: follow one request

Logs from many threads interleave, so give each request an id and put it in every line. With SLF4J's MDC:

java
@Component
class CorrelationFilter extends OncePerRequestFilter {

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        String id = Optional.ofNullable(request.getHeader("X-Request-Id")).orElse(UUID.randomUUID().toString());
        MDC.put("requestId", id);
        response.setHeader("X-Request-Id", id);
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.remove("requestId");
        }
    }
}

Configure the log pattern to print %X{requestId}, and pass the id to downstream calls in a header so one search finds the whole path across services. Distributed tracing (Micrometer Tracing with OpenTelemetry) does this with structure. With the id of one failing request, read its lines in order: what it asked, what it got, where it stopped.

Step 4: raise the level, only where needed

A bug may sit in code that logs at DEBUG. Changing the level for one logger at runtime avoids a redeploy. With Spring Boot Actuator's loggers endpoint exposed to administrators:

Shell
curl -X POST localhost:8080/actuator/loggers/dev.javahacks.shop.checkout \
  -H 'Content-Type: application/json' -d '{"configuredLevel":"DEBUG"}'

Set it back to INFO when done, since debug logging can multiply log volume and cost. Never enable debug for the whole application on a busy service, and never log request bodies, tokens or personal data while investigating; logs outlive incidents.

Step 5: structured logs

Plain text lines are hard to filter. Spring Boot 3.4 and later can write structured JSON logs through properties (logging.structured.format.console=ecs), and Logback encoders do the same on earlier versions. Fields like requestId, userId, orderId and durationMs become filterable and aggregatable. Log events with the identifiers needed to find the entity, not entire payloads.

Step 6: inspect the running process, carefully

If logs are not enough, a thread dump shows where threads are stuck, and a heap histogram or a JFR recording shows memory and CPU. These tools attach without stopping the service when used correctly. Keep in mind the cost of a heap dump.

Step 7: form a hypothesis, test it, write the test

A hypothesis names a cause and a prediction: "the pool is exhausted, so a thread dump will show threads waiting in getConnection". Check the prediction. When you find the input that fails, reproduce it in a test locally, as in a practical testing strategy, so the fix is proven and cannot regress. Add the log line or metric that would have shown the problem sooner.

What to avoid

  • Changing several things at once, so you cannot tell what worked.
  • Restarting before capturing a thread dump or logs; a restart erases evidence.
  • Fixing the symptom (raise the timeout) without finding the cause.
  • Printing secrets to logs.

Metrics first, then logs

Logs tell you what happened to one request. Metrics tell you which requests to read. Start with the rate, error and duration numbers per endpoint, because they point at the right log lines. If you export Micrometer timers, a query for the failing endpoint answers three questions at once: did the error rate really rise, when exactly, and did latency move with it:

Text
sum(rate(http_server_requests_seconds_count{uri="/checkout",status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count{uri="/checkout"}[5m]))

histogram_quantile(0.99,
  sum by (le) (rate(http_server_requests_seconds_bucket{uri="/checkout"}[5m])))

If errors rose but latency did not, the failure is quick, which points at validation, authorization or a dependency that refuses immediately. If latency rose first and errors followed, something is slow and callers are timing out, which points at the pool, a downstream service or a lock. That single distinction removes half the hypotheses before you open a log file.

Add the metric you wish you had

When the existing metrics cannot answer the question, add one, deploy it and wait for the next occurrence. A timer around the suspicious call costs a few lines:

java
@Service
class PaymentGatewayClient {

    private final Timer chargeTimer;
    private final Counter chargeFailures;

    PaymentGatewayClient(MeterRegistry registry) {
        this.chargeTimer = Timer.builder("payment.charge")
            .publishPercentileHistogram()
            .register(registry);
        this.chargeFailures = Counter.builder("payment.charge.failures")
            .tag("reason", "timeout")
            .register(registry);
    }

    Receipt charge(ChargeRequest request) {
        return chargeTimer.record(() -> {
            try {
                return gateway.charge(request);
            } catch (SocketTimeoutException e) {
                chargeFailures.increment();
                throw new PaymentUnavailable(e);
            }
        });
    }
}

Keep tag values bounded: a user id or an order id as a tag creates a new time series per value and can overload the metrics backend. Put identifiers in logs, and categories in metrics.

Querying structured logs

With JSON logs, a few jq commands replace the grep and sed pipeline from step two. Pull the failing requests for the time window, group them by the exception class and look at one full example:

Shell
jq -r 'select(.level=="ERROR") | .["error.type"] // .message' app.json.log \
  | cut -c1-120 | sort | uniq -c | sort -rn | head

jq -c 'select(.requestId=="9f1c2a74") | {ts: .["@timestamp"], level, logger: .logger_name, message}' app.json.log

The second command reconstructs the life of one request, in order. Read it with the question "what did this request expect, and what did it get instead". Frequently the answer is visible two or three lines before the error: a retry, a timeout of a dependency, a fallback that returned an empty list.

Capturing evidence before you restart

A restart is the most common cure and the most common destroyer of evidence. Make a habit of collecting four things from the instance first, which together take less than a minute:

Shell
POD=orders-7d9f6b-x2k4q
kubectl logs $POD --since=30m > $POD.log
kubectl exec $POD -- jcmd 1 Thread.print > $POD.threads.txt
kubectl exec $POD -- jcmd 1 GC.heap_info > $POD.heap.txt
kubectl exec $POD -- jcmd 1 JFR.dump name=1 filename=/tmp/incident.jfr

Take two thread dumps ten seconds apart, because one dump shows where threads are, and two show whether they are moving. Threads parked in the same getConnection frame in both are evidence of pool exhaustion. Threads in different places are evidence of normal work, and the problem is elsewhere. JFR.dump only works if a recording was already started, so enabling continuous recording at deployment time is the preparation that pays off now.

A runbook for the next time

The first time an incident costs hours, the second should cost minutes. Write the investigation into a runbook in the repository, next to the code, with the commands that worked:

Markdown
## Checkout 5xx rising

1. Error rate and p99 for /checkout, 5-minute window (dashboard: Checkout / Overview)
2. Recent deploys and config changes (link to the deploy log)
3. Top error types: `jq` query above, last 30 minutes
4. Pool and dependency health: `hikaricp_connections_pending`, `payment_charge_seconds`
5. Capture threads twice, 10 seconds apart, before any restart
6. Mitigations: roll back the last deploy; raise payment timeout to 5 s via config; disable promo pricing flag

Update it after each incident with what was missing. The runbook is also the place to record the decisions that were hard to make under pressure, such as who can authorize a rollback, so that nobody has to invent the rule during an outage.

Where this does not apply

If you can reproduce the problem in a staging environment with a debugger, do that instead. Incidents with data loss or security impact follow an incident process first. The Spring Boot logging reference and the Actuator loggers endpoint documentation are the primary sources.

Also inDevOps
Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.