Skip to content
javahacks.dev
DevOps

How to Set Up Prometheus and Grafana for Java Metrics

Expose Micrometer metrics from a Spring Boot app, scrape them with Prometheus, build Grafana panels for JVM and HTTP health and add custom business metrics.

6 min read
Summarize in AI
Flat illustration of settings switches, a slider and a gear, with the Prometheus, Grafana, OpenJDK and Spring logos.

Metrics answer questions that logs cannot easily: how many requests per second, how long do they take, how full is the heap, how often does garbage collection pause. Micrometer collects them in the application, Prometheus stores them and Grafana shows them. This uses Spring Boot 4 with JDK 25.

Expose the metrics

XML
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
properties
management.endpoints.web.exposure.include=health,prometheus
management.endpoint.health.probes.enabled=true
management.metrics.tags.application=${spring.application.name}

GET /actuator/prometheus returns plain text:

Text
jvm_memory_used_bytes{area="heap",id="G1 Eden Space"} 4.5E7
http_server_requests_seconds_count{method="GET",status="200",uri="/orders/{id}"} 1832
http_server_requests_seconds_sum{method="GET",status="200",uri="/orders/{id}"} 41.2

Boot registers JVM, system, Tomcat, HikariCP and HTTP server metrics automatically. The uri tag uses the route template, not the raw path, which keeps cardinality low. Protect the endpoint, or serve it on a separate management port, since it reveals internals.

Prometheus

A minimal scrape configuration:

YAML
scrape_configs:
  - job_name: shop
    metrics_path: /actuator/prometheus
    scrape_interval: 15s
    static_configs:
      - targets: ['shop:8080']

On Kubernetes with the Prometheus Operator, describe the target with a ServiceMonitor:

YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: shop
spec:
  selector:
    matchLabels: { app: shop }
  endpoints:
    - port: http
      path: /actuator/prometheus
      interval: 15s

Prometheus finds the pods through the Service and adds pod and namespace labels. The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager and Grafana together.

Grafana

Add Prometheus as a data source. For a start, import a community JVM (Micrometer) dashboard by its id from grafana.com, then build your own panels with PromQL:

PanelQuery
Requests per secondsum(rate(http_server_requests_seconds_count[5m]))
Error ratiosum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m]))
p99 latencyhistogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket[5m])))
Heap usedjvm_memory_used_bytes{area="heap"}
GC pause timerate(jvm_gc_pause_seconds_sum[5m])
Pool waithikaricp_connections_pending

Histogram buckets are not enabled by default for http.server.requests. Turn them on for percentile queries:

properties
management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.slo.http.server.requests=100ms,500ms,1s

Each bucket multiplies series, so enable them for the metrics you chart.

Reading JVM panels

A sawtooth of heap with rising lows suggests a leak, see finding memory leaks. Frequent long GC pauses call for the log analysis in reading GC logs. Many pending Hikari connections point to pool exhaustion.

Custom metrics

java
@Component
class OrderMetrics {
    private final Counter placed;
    private final Timer checkout;

    OrderMetrics(MeterRegistry registry) {
        this.placed = Counter.builder("orders.placed").tag("channel", "web").register(registry);
        this.checkout = Timer.builder("checkout.duration").publishPercentileHistogram().register(registry);
    }

    void orderPlaced() { placed.increment(); }
    <T> T timeCheckout(Supplier<T> work) { return checkout.record(work); }
}

Name by what is measured, use bounded tags (channel, payment method), never ids. A tag with a user id creates a series per user and can bring Prometheus down.

Dashboards that help

Build one dashboard per service with the RED signals at the top: rate, errors, duration, then resource panels below. Link it from alerts, see alerts that matter.

Queries worth knowing

A dashboard is a set of queries, and a few cover most needs. The request rate per route is sum by (uri) (rate(http_server_requests_seconds_count[5m])). The error ratio is the rate of 5xx responses divided by the total. The 95th percentile latency, if you enabled histograms, is histogram_quantile(0.95, sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m]))). For the JVM, watch jvm_memory_used_bytes{area="heap"} against the maximum, the rate of jvm_gc_pause_seconds_count, and jvm_threads_live_threads. A heap that rises between collections and never returns to a baseline is a leak, and a GC pause rate that climbs over weeks is a heap that is too small.

Cardinality is the cost

Every distinct combination of label values is a separate time series that Prometheus stores and indexes. A label such as uri is safe because Micrometer records the route template, /orders/{id}, and not the actual path. A label that holds a user id, an order id or an email would create a series per user, and a few hundred thousand of them can bring Prometheus down. Review each custom tag for the number of values it can take, and keep it to tens, not thousands. If you need to find a single request, use tracing or logs instead.

Alerting on symptoms

Alert on what a user would notice, such as a high error ratio or slow responses for several minutes, and not on causes such as CPU at 80 percent for one minute. Cause-based alerts wake people for nothing and teach them to ignore the pager. A useful first set is three alerts: the service is down, the error ratio is above a threshold for ten minutes, and the 95th percentile latency is above its objective for ten minutes. Add saturation signals, such as a connection pool near its limit, as warnings that go to a channel and not to a phone.

Securing the endpoint

The metrics endpoint exposes internal names and counts. Serve it on a separate management port with management.server.port, keep that port off the public network, and let only Prometheus reach it. If it must share the main port, require authentication for it and scrape with credentials stored in a secret.

The first dashboard

Build one dashboard per service with the same layout for all of them, so that anyone on call can read any of them. Put the golden signals at the top: request rate, error ratio, latency percentiles and saturation, where saturation is the thing that fills up first in your service, such as the connection pool or the thread pool. Below them put the JVM panels for heap, garbage collection and threads, and below those the dependencies, such as database latency and downstream call errors. A dashboard with a fixed order trains the eye, and the person who sees the pool panel red knows where to look next.

Give every panel a title that states the question it answers, "Are requests failing?" instead of "http_server_requests", and set the thresholds on the axis so a glance shows whether a value is fine. Link each panel to the alert that watches it.

Running Prometheus in the real world

The default retention is fifteen days, which is plenty for dashboards and short for capacity planning. Decide how long you need, size the disk for it, and put the data on a persistent volume, or you lose history with every pod restart. Prometheus is not highly available by itself, so for production run two identical instances that scrape the same targets, or use a managed service. Back up the configuration and the alert rules in version control, and test rule changes with promtool check rules in CI before they reach the server.

Labels from the platform

In Kubernetes, discovery adds labels such as namespace, pod and container to every series, which is how you can split a metric by pod. Drop the labels you will never query with relabeling rules, since every label adds storage. Add an application tag in the Micrometer configuration with management.metrics.tags.application, so a series can be traced back to a service even when several services scrape into one Prometheus.

Where this does not apply

Short lived batch jobs may disappear before a scrape; push through a gateway or use Kubernetes Job metrics. Docs: Micrometer Prometheus and Prometheus getting started.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.