Skip to content
javahacks.dev
JVM & Performance

How to Find Slow Endpoints With Micrometer and Tracing

Find slow endpoints in a Java service with Micrometer timers, percentile histograms, distributed tracing and slow query logs.

5 min read
Summarize in AI
Flat illustration of a browser and a server trading a request and a response, with the OpenJDK, Docker and Spring Boot logos.

"The site is slow" is not a diagnosis. The questions that lead to a fix are: which endpoint, how slow, for whom, and where does the time go inside it. Metrics answer the first three, and tracing answers the fourth. Spring Boot gives both through Micrometer, with little setup. This post shows how to rank endpoints by latency, why you must look at percentiles and not averages, how to add tracing so a slow request breaks into spans, and how to confirm what you find in the database and the JVM. It targets Spring Boot 3 and 4 with Micrometer.

Metrics: which endpoints are slow

Spring Boot records every HTTP request in a timer named http.server.requests, tagged with the URI template, method and status. Expose it and ask for percentile histograms:

properties
management.endpoints.web.exposure.include=health,metrics,prometheus
management.metrics.distribution.percentiles-histogram.http.server.requests=true

With Prometheus scraping the endpoint, a query ranks routes by their slow tail:

Text
histogram_quantile(0.95,
  sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m])))

That lists the 95th percentile latency for each route over five minutes. Sort descending and the top few lines are where to look. Use the URI template, such as /orders/{id}, and never the raw path, or the metric would have a different series for every id. Boot already does this, which is why tagging by route is safe.

Look at the tail, not the average

An average of 120 ms can hide that one request in twenty takes two seconds. Users remember the slow ones, and so do timeouts and retries upstream. Look at p95 and p99, and compare them with the median. A route where the median is 40 ms and p99 is 3 seconds has an intermittent cause: a lock, a cold cache, a garbage collection pause, a query plan that sometimes flips. A route where everything is uniformly 800 ms has a constant cost, usually a slow query or a remote call on every request. The two shapes need different investigations, and the percentile view tells you which one you have.

Check the rate and the errors

Latency means little without volume. A route with p99 of five seconds that receives two requests a day is a curiosity, and one that serves a thousand a minute is an incident. Plot the request rate next to latency, and the error rate too: slow failures and fast failures look different. A route that returns 500 after 30 seconds is waiting on a timeout, and the fix is the timeout configuration and the dependency behind it. The same metrics, with a rate and an error ratio, are the start of an alerting scheme.

Tracing: where the time goes

A metric says that /orders/{id} is slow. A trace says what it was doing. Distributed tracing records a span for each operation within a request: the controller, each database call, each outgoing HTTP call. Add the bridge and an exporter:

XML
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
properties
management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://collector:4318/v1/traces

Sample a fraction in production, since tracing every request has a cost. The trace viewer in Jaeger, Tempo or any OpenTelemetry-compatible backend shows a waterfall for one request. A single long bar is the dominant span; a staircase of identical short spans is a loop calling the same thing repeatedly, the N+1 pattern. Find a trace for a slow request by searching on duration, and read the longest span first. The documentation for the library is in the Micrometer Tracing reference.

Add your own timers

Auto-instrumentation covers HTTP and often JDBC and clients. For your own expensive operation, add a span or a timer:

java
@Service
class PricingService {

    private final ObservationRegistry registry;

    PricingService(ObservationRegistry registry) {
        this.registry = registry;
    }

    Price price(Cart cart) {
        return Observation.createNotStarted("pricing.calculate", registry)
                .lowCardinalityKeyValue("currency", cart.currency())
                .observe(() -> calculate(cart));
    }
}

One Observation produces both a timer and a trace span, which is the point of the API. Keep tag values low in cardinality: a currency or a plan name is fine, a customer id is not, since each distinct value creates a new time series.

Confirm with the database

If the dominant span is a database call, take that statement to the database. Logging slow statements, as described in Hibernate statistics and slow query logging, identifies it, and a repeated cheap statement is fixed as in fixing N+1. Then run the plan. A trace tells you which query is slow; the plan tells you why.

When a trace shows time spent in no particular span, between two operations, the delay is in the JVM or the thread pool: a request waiting for a worker, a garbage collection pause, a lock. Check the thread pool metrics, which Boot exposes for Tomcat as tomcat.threads.busy, and take thread dumps while the slowness is present, as in reading thread dumps. If threads are saturated, the tuning in Tomcat threads and throughput applies. Add the GC pause metric from jvm.gc.pause to the same dashboard, since a spike in p99 that lines up with a pause points at memory settings.

Compare to a threshold

Decide what slow means for each route, and write it down: a search must answer in 300 ms at p95, a report may take five seconds. Alert on the objective and not on a vague feeling, so that the endpoints that matter get attention first. Review the ranked list weekly and fix the top item before it becomes an incident.

A short routine to keep

Once a week, open the ranked p95 list and the error list, pick the worst route that carries real traffic, and open three traces from its slowest requests. Write one sentence naming the dominant span. If it is a query, open the plan; if it is a remote call, check the timeout and whether the call can run in parallel or be cached; if it is a gap between spans, look at thread pool saturation. Fix it, deploy, and watch the same query for a week to confirm the tail moved. Repeat. The routine costs an hour and, applied steadily, removes the incidents that otherwise arrive as complaints. Keep a note of what each fix changed in the p95, so the team can see that the work pays.

When this is not the tool

Measuring the behaviour of the whole system from the user's side, including the network and the browser, needs real-user monitoring, which sits outside the server. For a quick local look at one slow method, a profiler is faster than setting up tracing. The Spring Boot documentation lists the available metrics.

Also inDevOps
Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.