Skip to content
javahacks.dev
Debugging

Fix Java Container OOMKilled Errors for Spring Services

Fix a Java container killed with OOMKilled and exit code 137 by budgeting heap, metaspace, threads and native memory inside the container limit.

6 min read
Summarize in AI
Flat illustration of a log with a failing line and a wrench, with the OpenJDK, Docker and Spring logos.

A service restarts under load with no log line to explain it. The orchestrator says:

Text
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

(Kubernetes, kubectl describe pod). In Docker, docker inspect --format '{{.State.OOMKilled}}' <container> prints true. Exit code 137 is 128 plus signal 9: the kernel's OOM killer sent SIGKILL because the container's cgroup went over its memory limit. The JVM had no chance to write an OutOfMemoryError, so the logs end abruptly. This is the container version of the heap-versus-process confusion described in JVM memory areas.

Why the heap alone is the wrong number

A common setup: container limit 1 GiB, -Xmx1g. The heap can grow to the entire limit, and the JVM needs more on top: Metaspace, thread stacks, the JIT code cache, direct buffers, GC bookkeeping. The first time the heap fills, the process exceeds the limit and is killed. The error is predictable from the arithmetic.

AreaTypical size in a Spring Boot service
HeapThe share you configure
Metaspace100 to 250 MB, depending on libraries
Thread stacksThreads times about 1 MB reserved; 200 threads is 200 MB of address space
Code cacheUp to 240 MB reserved, much less used
Direct buffers (Netty, NIO)Tens of MB, can grow
GC structuresA few percent of the heap

These are ranges to measure, not guarantees.

Fix 1: size the heap as a share of the limit

Shell
java -XX:MaxRAMPercentage=65 -jar app.jar

The JVM reads the container limit and sets the maximum heap to 65 percent of it. On a 2 GiB limit that is about 1.3 GiB, leaving 700 MiB for the rest. A value of 70 to 75 can work for services with few threads and small metaspace, and 50 for ones with many threads or heavy native use. Do not leave -Xmx in the image from another environment: an explicit -Xmx overrides the percentage.

Kubernetes users should set matching requests and limits for memory, because unequal values change scheduling and eviction behavior.

Fix 2: measure the rest

Enable Native Memory Tracking in a test run:

Shell
java -XX:NativeMemoryTracking=summary -XX:MaxRAMPercentage=65 -jar app.jar
jcmd <pid> VM.native_memory summary

Add up committed memory for each category, and compare with the container's resident memory (kubectl top pod, or docker stats). If resident memory is higher than the NMT total, something outside the JVM's accounting uses memory: a native library, memory-mapped files, or glibc malloc fragmentation. For the last, setting MALLOC_ARENA_MAX=2 in the environment reduces arena overhead in multi-threaded Java processes.

Fix 3: reduce the consumers

  • Thread stacks: reduce the maximum number of Tomcat threads (server.tomcat.threads.max) to what the service needs, or use virtual threads.
  • Metaspace: remove unused libraries, and check for class loader leaks (Metaspace errors).
  • Direct memory: set -XX:MaxDirectMemorySize to cap NIO buffers.
  • Heap: a real leak grows until the kill; take a heap dump from a test environment (heap analysis).

Fail clearly inside the JVM

If the heap does fill, you want a Java OutOfMemoryError and a restart, not a silent kill:

Shell
java -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps -jar app.jar

The JVM exits cleanly on the first OutOfMemoryError instead of limping on, and the dump gives you evidence. Mount a volume for /dumps, or the dump is lost with the container.

Confirm it is a kill, not a crash

dmesg | grep -i "killed process" on the node, or the events in kubectl describe pod, show the OOM kill with the memory usage at the time. A JVM crash writes hs_err_pid*.log; a kill does not.

Reproduce it on a laptop

A kill you can trigger on demand is one you can prove fixed. This program allocates direct buffers, which sit outside the heap, so a generous -Xmx does not protect the container:

java
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;

public class OffHeapHog {

    public static void main(String[] args) throws Exception {
        List<ByteBuffer> buffers = new ArrayList<>();
        for (int i = 1; ; i++) {
            ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024 * 1024);
            for (int p = 0; p < buffer.capacity(); p += 4096) {
                buffer.put(p, (byte) 1);                 // touch each page so it becomes resident
            }
            buffers.add(buffer);
            System.out.println("direct total: " + (i * 16) + " MB");
            Thread.sleep(100);
        }
    }
}
Shell
docker run --memory 300m --name hog -v "$PWD:/src" eclipse-temurin:25-jdk \
  java -Xmx100m -XX:MaxDirectMemorySize=1g /src/OffHeapHog.java
echo "exit code: $?"
docker inspect hog --format '{{.State.OOMKilled}}'
docker rm hog

The heap is capped at 100 MB, and the process should still die when the container reaches 300 MB, with exit code 137, OOMKilled true and no Java stack trace. Run it again with -XX:MaxDirectMemorySize=64m and the result changes: the JVM throws OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory once the cap is reached, which is a Java exception you can log, catch and alert on. That difference is the whole argument for setting caps: a limit inside the JVM turns a silent kill into a diagnosable error.

A Kubernetes manifest that leaves room

Here is the shape of a deployment that applies the fixes above. The limit and the request match, the heap is a percentage, and the JVM exits on its own out-of-memory error so the pod restarts with a clear reason:

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders }
  template:
    metadata:
      labels: { app: orders }
    spec:
      containers:
        - name: orders
          image: registry.example.com/orders:1.8
          resources:
            requests: { memory: "1536Mi", cpu: "500m" }
            limits:   { memory: "1536Mi" }
          env:
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -XX:MaxRAMPercentage=60
                -XX:MaxDirectMemorySize=128m
                -XX:+ExitOnOutOfMemoryError
                -XX:+HeapDumpOnOutOfMemoryError
                -XX:HeapDumpPath=/dumps
          volumeMounts:
            - { name: dumps, mountPath: /dumps }
      volumes:
        - name: dumps
          emptyDir: { sizeLimit: 2Gi }

With a 1,536 MiB limit, 60 percent gives a heap of about 920 MiB and leaves roughly 600 MiB for Metaspace, stacks, the code cache, the 128 MiB direct cap and the rest. There is deliberately no CPU limit in this sketch, because a CPU limit throttles the JIT and GC threads and shows up as latency, but that is a team policy and worth deciding on its own. An emptyDir survives a container restart within the pod, so the heap dump is there when you kubectl cp it out.

When a pod is killed, the first question is what the JVM thought its limits were. Log them once at startup so the answer is in the log of the previous instance:

java
@Component
class RuntimeReport implements ApplicationRunner {

    private static final Logger log = LoggerFactory.getLogger(RuntimeReport.class);

    @Override
    public void run(ApplicationArguments args) {
        Runtime rt = Runtime.getRuntime();
        log.info("jvm cpus={} maxHeap={}MB inputArguments={}",
            rt.availableProcessors(),
            rt.maxMemory() >> 20,
            ManagementFactory.getRuntimeMXBean().getInputArguments());
    }
}

If maxHeap is close to the container limit, the percentage is not applied (an old -Xmx or an old JDK that ignores the cgroup). If cpus is 1 on a node with 16 cores, the quota is shaping the garbage collector and thread pools, which is a separate cause of slowness that shows up beside memory trouble.

Which kill was it

A short table separates the cases that all end in a restart:

EvidenceMeaningNext step
OOMKilled, exit 137, no Java error in the logThe cgroup limit was crossedNMT, then lower the heap percentage or raise the limit
java.lang.OutOfMemoryError: Java heap space in the logThe heap is too small or leakingHeap dump, then fix the retention
OutOfMemoryError: Metaspace or Direct buffer memoryA non-heap cap was hitRaise the cap or find the leak
hs_err_pid*.log existsA native crash, not memory pressureRead the crash report
Exit 137 with a liveness probe failure in eventsThe kubelet killed a hung podFix the probe or the hang, see reading a stack trace

Work down the table in order. Most "memory problems" are decided by the first two rows, and teams lose days by tuning the heap for a pod that the kubelet killed for a failing probe.

Where this does not apply

An exit code 137 can come from a manual kill -9 or from a liveness probe failure that restarts the container, so check the reason line. Exit code 143 is a normal SIGTERM shutdown. The Docker documentation on resource constraints and the Kubernetes documentation on pod resources explain the limits. For the Spring side of the image, see a Spring Boot Docker image that will not start.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.