OOMKilled means the kernel killed your process because it asked for more memory than it was allowed to use. It is not a Kubernetes error nor an application failure: it is the system enforcing a limit somebody wrote.
`kubectl describe pod <name>` shows `Last State: Terminated` with `Reason: OOMKilled` and `Exit Code: 137`. That 137 is 128 plus 9, the SIGKILL signal: nobody asked the process to shut down cleanly, it was killed.
There are two different situations behind the same name and they are worth separating. The first is that **your container exceeded its own limit**: the cgroup kills it and only that container dies. The pod still exists and restarts.
The second is that **the whole node ran out of memory**. There the kubelet starts evicting pods to save the node, and you will see `Evicted` with a memory-pressure message, not `OOMKilled`. Evicted pods get rescheduled elsewhere. If you are chasing an OOMKilled and you actually have evictions, you are looking at the wrong container: the problem is cluster capacity.
The `request` is what the scheduler reserves to decide which node the pod fits on. The `limit` is the ceiling the kernel enforces. Only the second one causes an OOMKilled.
If you set a `limit` and no `request`, Kubernetes assumes the request equals the limit. If you set a `request` and no `limit`, the container can grow until it consumes the node — and then you end up with evictions rather than a tidy OOMKilled.
When `request` and `limit` are equal, the pod gets the Guaranteed service class and is the last candidate for eviction. When they differ it is Burstable, and with neither it is BestEffort, the first to go when the node gets tight. For a service that matters, making them equal is what buys stability.
This is the most common cause in Java shops and the least explained. By default the JVM sizes its maximum heap as a fraction of the memory it sees available. On a normal machine that works. Inside a container, older versions saw the whole **node** memory rather than the cgroup limit.
The result is a JVM that believes it has 64 GB to hand out while the container has a 2 GB limit. It grows comfortably until it goes over, and the kernel kills it. The application logs show nothing odd, because from its point of view everything was fine.
Modern JVMs respect the cgroup with `-XX:+UseContainerSupport`, which is on by default, and the heap is tuned with `-XX:MaxRAMPercentage`. Even so, leave headroom: a Java process uses memory outside the heap — metaspace, direct buffers, thread stacks — and that consumption is not in the heap percentage.
The practical rule is not to give the heap more than 70% to 80% of the container limit. Give it 100% and the heap simply reaches the ceiling before the JVM gets a chance to collect.
Raising the limit unblocks the service, and sometimes it is correct: the limit was miscalculated from the start and real consumption is stable at a higher value.
But if there is a memory leak, raising the limit only changes when it falls over. At 512 MB it died every two hours; at 2 GB it dies every eight. Nobody notices the difference until a long weekend.
The way to tell them apart is the shape of consumption over time. Consumption that rises, settles and stays flat is a miscalculated limit. Consumption that climbs in steps and never comes down, even when traffic drops overnight, is a leak.
And there is a cost that does not show: memory over-reserved across every pod in a cluster is pure overspend. Requests inflated "just in case" make the scheduler believe a node is full when it is half empty, and the autoscaler adds nodes nobody uses. It is one of the most frequent sources of excess cost we find in FinOps work.
When a stable service starts dying on memory, the causes are usually three: someone lowered the limit in a commit, a new application version consumes more, or a new dependency brought its own consumption.
All three are changes with a date and time, and all three are answered by the revision history: which image and commit were active before the first OOMKilled, and what changed between that revision and the next — including the limit value itself, which lives in the manifest.
Rebuilding that by hand is what stretches time to recovery. Diagnosing an OOMKilled is among the simplest there is; what takes time is finding out since when and why.
Moonin keeps a revision history per service with its image, its commit and what changed against the previous one, including resource limits. Seeing that the memory limit dropped from 1 Gi to 512 Mi in the revision from two days ago is the whole answer on one screen.
And it records restarts against the active revision, so a service that always restarted is distinguishable from one that started with the latest deployment.
What it does not do: it does not adjust limits, does not restart pods and takes no action on its own. The only possible mutation in the entire product is HPA and replica adjustment by the Scaling Rules Agent, which is optional and ships disabled.
The demo is open and needs no form: you can look at how the revision history appears with its resource limits and the associated restarts, before installing anything.
About usPricingSecurity and permissionsPartner programmevs Datadogvs Prometheus & Grafanavs New Relicvs Dynatracevs ELK and ElasticsearchCrashLoopBackOffOOMKilledImagePullBackOffPod stuck in PendingDocumentationDemoSign upContact usArguz, the consultancy
It is 128 plus 9: the SIGKILL signal. The process did not get a request to shut down cleanly, it was killed. In Kubernetes it is almost always the kernel enforcing the container memory limit, and `kubectl describe pod` confirms it with `Reason: OOMKilled`.
OOMKilled is your container exceeding its own limit: the cgroup kills it and the pod restarts on the same node. Evicted is the whole node running out of memory: the kubelet evicts pods to save itself and those pods get rescheduled elsewhere. The first is a limits problem; the second is cluster capacity.
Because the JVM maximum heap is computed as a fraction of available memory, and if it does not respect the cgroup it sees the whole node memory instead of the container limit. `-XX:+UseContainerSupport` makes it respect the limit, and `-XX:MaxRAMPercentage` tunes the heap. Do not go past 70% to 80% of the limit, because a JVM also uses memory outside the heap: metaspace, direct buffers and thread stacks.
Only if consumption is stable at a value higher than the limit you set. If there is a leak, raising the limit changes when it falls over, not whether. Tell them apart by the shape: rises then flattens is a miscalculated limit; climbs in steps and never comes down even when traffic drops overnight is a leak.
For a service that matters, yes. When they are equal the pod gets the Guaranteed class and is the last eviction candidate when the node gets tight. When they differ it is Burstable, and with neither it is BestEffort, the first to go. The cost is that you reserve the full memory even when you are not using it all the time.