Fix Java Container OOMKilled Errors for Spring Services
Fix a Java container killed with OOMKilled and exit code 137 by budgeting heap, metaspace, threads and native memory inside the container limit.

A service restarts under load with no log line to explain it. The orchestrator says:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137(Kubernetes, kubectl describe pod). In Docker, docker inspect --format '{{.State.OOMKilled}}' <container> prints true. Exit code 137 is 128 plus signal 9: the kernel's OOM killer sent SIGKILL because the container's cgroup went over its memory limit. The JVM had no chance to write an OutOfMemoryError, so the logs end abruptly. This is the container version of the heap-versus-process confusion described in JVM memory areas.
Why the heap alone is the wrong number
A common setup: container limit 1 GiB, -Xmx1g. The heap can grow to the entire limit, and the JVM needs more on top: Metaspace, thread stacks, the JIT code cache, direct buffers, GC bookkeeping. The first time the heap fills, the process exceeds the limit and is killed. The error is predictable from the arithmetic.
| Area | Typical size in a Spring Boot service |
|---|---|
| Heap | The share you configure |
| Metaspace | 100 to 250 MB, depending on libraries |
| Thread stacks | Threads times about 1 MB reserved; 200 threads is 200 MB of address space |
| Code cache | Up to 240 MB reserved, much less used |
| Direct buffers (Netty, NIO) | Tens of MB, can grow |
| GC structures | A few percent of the heap |
These are ranges to measure, not guarantees.
Fix 1: size the heap as a share of the limit
java -XX:MaxRAMPercentage=65 -jar app.jarThe JVM reads the container limit and sets the maximum heap to 65 percent of it. On a 2 GiB limit that is about 1.3 GiB, leaving 700 MiB for the rest. A value of 70 to 75 can work for services with few threads and small metaspace, and 50 for ones with many threads or heavy native use. Do not leave -Xmx in the image from another environment: an explicit -Xmx overrides the percentage.
Kubernetes users should set matching requests and limits for memory, because unequal values change scheduling and eviction behavior.
Fix 2: measure the rest
Enable Native Memory Tracking in a test run:
java -XX:NativeMemoryTracking=summary -XX:MaxRAMPercentage=65 -jar app.jar
jcmd <pid> VM.native_memory summaryAdd up committed memory for each category, and compare with the container's resident memory (kubectl top pod, or docker stats). If resident memory is higher than the NMT total, something outside the JVM's accounting uses memory: a native library, memory-mapped files, or glibc malloc fragmentation. For the last, setting MALLOC_ARENA_MAX=2 in the environment reduces arena overhead in multi-threaded Java processes.
Fix 3: reduce the consumers
- Thread stacks: reduce the maximum number of Tomcat threads (
server.tomcat.threads.max) to what the service needs, or use virtual threads. - Metaspace: remove unused libraries, and check for class loader leaks (Metaspace errors).
- Direct memory: set
-XX:MaxDirectMemorySizeto cap NIO buffers. - Heap: a real leak grows until the kill; take a heap dump from a test environment (heap analysis).
Fail clearly inside the JVM
If the heap does fill, you want a Java OutOfMemoryError and a restart, not a silent kill:
java -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps -jar app.jarThe JVM exits cleanly on the first OutOfMemoryError instead of limping on, and the dump gives you evidence. Mount a volume for /dumps, or the dump is lost with the container.
Confirm it is a kill, not a crash
dmesg | grep -i "killed process" on the node, or the events in kubectl describe pod, show the OOM kill with the memory usage at the time. A JVM crash writes hs_err_pid*.log; a kill does not.
Reproduce it on a laptop
A kill you can trigger on demand is one you can prove fixed. This program allocates direct buffers, which sit outside the heap, so a generous -Xmx does not protect the container:
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
public class OffHeapHog {
public static void main(String[] args) throws Exception {
List<ByteBuffer> buffers = new ArrayList<>();
for (int i = 1; ; i++) {
ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024 * 1024);
for (int p = 0; p < buffer.capacity(); p += 4096) {
buffer.put(p, (byte) 1); // touch each page so it becomes resident
}
buffers.add(buffer);
System.out.println("direct total: " + (i * 16) + " MB");
Thread.sleep(100);
}
}
}docker run --memory 300m --name hog -v "$PWD:/src" eclipse-temurin:25-jdk \
java -Xmx100m -XX:MaxDirectMemorySize=1g /src/OffHeapHog.java
echo "exit code: $?"
docker inspect hog --format '{{.State.OOMKilled}}'
docker rm hogThe heap is capped at 100 MB, and the process should still die when the container reaches 300 MB, with exit code 137, OOMKilled true and no Java stack trace. Run it again with -XX:MaxDirectMemorySize=64m and the result changes: the JVM throws OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory once the cap is reached, which is a Java exception you can log, catch and alert on. That difference is the whole argument for setting caps: a limit inside the JVM turns a silent kill into a diagnosable error.
A Kubernetes manifest that leaves room
Here is the shape of a deployment that applies the fixes above. The limit and the request match, the heap is a percentage, and the JVM exits on its own out-of-memory error so the pod restarts with a clear reason:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
replicas: 3
selector:
matchLabels: { app: orders }
template:
metadata:
labels: { app: orders }
spec:
containers:
- name: orders
image: registry.example.com/orders:1.8
resources:
requests: { memory: "1536Mi", cpu: "500m" }
limits: { memory: "1536Mi" }
env:
- name: JAVA_TOOL_OPTIONS
value: >-
-XX:MaxRAMPercentage=60
-XX:MaxDirectMemorySize=128m
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumps
volumeMounts:
- { name: dumps, mountPath: /dumps }
volumes:
- name: dumps
emptyDir: { sizeLimit: 2Gi }With a 1,536 MiB limit, 60 percent gives a heap of about 920 MiB and leaves roughly 600 MiB for Metaspace, stacks, the code cache, the 128 MiB direct cap and the rest. There is deliberately no CPU limit in this sketch, because a CPU limit throttles the JIT and GC threads and shows up as latency, but that is a team policy and worth deciding on its own. An emptyDir survives a container restart within the pod, so the heap dump is there when you kubectl cp it out.
Print what the JVM decided at startup
When a pod is killed, the first question is what the JVM thought its limits were. Log them once at startup so the answer is in the log of the previous instance:
@Component
class RuntimeReport implements ApplicationRunner {
private static final Logger log = LoggerFactory.getLogger(RuntimeReport.class);
@Override
public void run(ApplicationArguments args) {
Runtime rt = Runtime.getRuntime();
log.info("jvm cpus={} maxHeap={}MB inputArguments={}",
rt.availableProcessors(),
rt.maxMemory() >> 20,
ManagementFactory.getRuntimeMXBean().getInputArguments());
}
}If maxHeap is close to the container limit, the percentage is not applied (an old -Xmx or an old JDK that ignores the cgroup). If cpus is 1 on a node with 16 cores, the quota is shaping the garbage collector and thread pools, which is a separate cause of slowness that shows up beside memory trouble.
Which kill was it
A short table separates the cases that all end in a restart:
| Evidence | Meaning | Next step |
|---|---|---|
OOMKilled, exit 137, no Java error in the log | The cgroup limit was crossed | NMT, then lower the heap percentage or raise the limit |
java.lang.OutOfMemoryError: Java heap space in the log | The heap is too small or leaking | Heap dump, then fix the retention |
OutOfMemoryError: Metaspace or Direct buffer memory | A non-heap cap was hit | Raise the cap or find the leak |
hs_err_pid*.log exists | A native crash, not memory pressure | Read the crash report |
| Exit 137 with a liveness probe failure in events | The kubelet killed a hung pod | Fix the probe or the hang, see reading a stack trace |
Work down the table in order. Most "memory problems" are decided by the first two rows, and teams lose days by tuning the heap for a pod that the kubelet killed for a failing probe.
Where this does not apply
An exit code 137 can come from a manual kill -9 or from a liveness probe failure that restarts the container, so check the reason line. Exit code 143 is a normal SIGTERM shutdown. The Docker documentation on resource constraints and the Kubernetes documentation on pod resources explain the limits. For the Spring side of the image, see a Spring Boot Docker image that will not start.


