Memory-related incidents are among the most common and confusing problems in Kubernetes production environments.

A Pod restarts unexpectedly, an application becomes unavailable, or a Kubernetes node suddenly becomes unstable. One of the first things an SRE may notice is:

OOMKilled

But does OOMKilled always mean that the Kubernetes node ran out of memory?

No.

Understanding the difference between Container OOM, Node Memory Pressure, Pod Eviction, and Node-level OOM is extremely important for Kubernetes SREs.

1. Container OOM vs Node OOM

The simplest way to understand the difference is:

Container OOM is a container-level memory problem. Node OOM is a node-level memory problem.

Area	Container OOM	Node OOM
Scope	One container/process	Entire Kubernetes node
Main cause	Container exceeds its memory limit	Node runs critically low on memory
Typical evidence	`OOMKilled`	`MemoryPressure`, kernel OOM messages
Who gets killed?	Usually the offending container/process	Linux kernel may kill processes/containers
Pod behavior	Container may restart	Pods may be evicted, killed, or become unstable
Other workloads affected	Usually limited	Potentially many workloads
Memory limit involved?	Usually	Not necessarily
Severity	Application/container-level	Node/cluster-level

The distinction becomes clearer with a production example.

2. What is Container OOM?

Consider a Kubernetes node with 64 GB of memory.

A Pod is running an application with the following configuration:

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "2Gi"

The container's application gradually consumes more memory:

500 MB
   ↓
1 GB
   ↓
1.5 GB
   ↓
2 GB
   ↓
2.2 GB

The container has a memory limit of 2 GiB.

If the container attempts to consume memory beyond its allowed limit, the process can be killed due to an out-of-memory condition.

Kubernetes may then report:

Reason: OOMKilled

You can investigate the Pod with:

kubectl describe pod myapp-xxx -n production

You may see:

Last State:
  Terminated:
    Reason: OOMKilled
    Exit Code: 137

This is an important clue that the container experienced a memory-related termination.

3. What does Exit Code 137 mean?

You will frequently encounter:

Exit Code: 137

The number comes from:

128 + 9 = 137

Signal 9 is:

SIGKILL

Therefore, exit code 137 generally indicates that the process was killed with SIGKILL.

However, an important SRE rule is:

Do not assume that every exit code 137 automatically means OOM.

Always verify the termination reason and supporting system evidence.

For example:

kubectl describe pod <pod> -n <namespace>

Look for:

Reason: OOMKilled

Then check the node and kernel logs if necessary.

4. What is Node OOM?

Now consider a different situation.

Suppose a Kubernetes node has 64 GB of memory:

Kubernetes Node
Memory = 64 GB

Several workloads are consuming memory:

Pod A       → 15 GB
Pod B       → 12 GB
Pod C       → 18 GB
Pod D       → 10 GB
System      →  8 GB
---------------------
Total       → 63 GB

The node is now under severe memory pressure.

If memory consumption continues and the Linux kernel cannot satisfy memory allocations, the kernel may invoke the OOM killer.

This is fundamentally different from a single container exceeding its configured memory limit.

The problem is now:

The node itself is running out of available memory.

5. How to identify Node Memory Pressure

From the Kubernetes control plane, check the node:

kubectl describe node <node-name>

Look at the Conditions section.

You may find:

Conditions:
  MemoryPressure   True

This means the kubelet has detected memory pressure on the node.

You should then investigate the actual node.

Run:

free -h

Example:

               total        used        free
Mem:             64Gi         62Gi       500Mi
Swap:               0           0           0

Other useful commands include:

top

vmstat 1

ps aux --sort=-%mem

These help identify which processes are consuming memory.

6. Check Linux Kernel Logs

For a node-level OOM investigation, kernel logs are extremely important.

Run:

dmesg -T | grep -i -E "oom|out of memory|killed process"

Or:

journalctl -k | grep -i -E "oom|out of memory|killed process"

You may find something similar to:

Out of memory: Killed process 12345 (java)

This is strong evidence that the Linux kernel invoked the OOM killer.

At this point, your investigation has moved beyond Kubernetes objects and into the Linux operating system layer.

That is an important skill for a Kubernetes SRE.

7. The SRE Mental Model

Think about a Kubernetes node like this:

                 KUBERNETES NODE
        ┌─────────────────────────────┐
        │                             │
        │  Pod A                      │
        │  ┌───────────────────────┐  │
        │  │ Container             │  │
        │  │ Memory limit: 2 GiB   │  │
        │  └───────────────────────┘  │
        │                             │
        │  Pod B                      │
        │                             │
        │  Pod C                      │
        │                             │
        │  kubelet                    │
        │  system processes           │
        │                             │
        └─────────────────────────────┘

Container OOM

Container memory usage
        ↓
Exceeds container limit
        ↓
Container/process killed
        ↓
Kubernetes reports OOMKilled
        ↓
Container may restart

Node-level OOM

Total node memory consumption
        ↓
Available memory becomes critically low
        ↓
Linux kernel cannot satisfy allocation
        ↓
Kernel OOM killer
        ↓
Process/container killed
        ↓
Potential impact to multiple workloads

8. Don't confuse OOM with Pod Eviction

This is where Kubernetes troubleshooting becomes more interesting.

There are at least three different situations an SRE should distinguish.

Situation 1 — Container OOM

Container exceeds memory limit
        ↓
Container/process killed
        ↓
OOMKilled

Situation 2 — Node Memory Pressure

Node available memory becomes low
        ↓
Kubelet detects memory pressure
        ↓
MemoryPressure = True
        ↓
Kubernetes may evict Pods

Situation 3 — Node-level Linux OOM

Node cannot satisfy memory allocation
        ↓
Linux kernel OOM killer
        ↓
Process/container killed

These situations are related, but they are not the same event.

9. Production Troubleshooting Methodology

When investigating a memory-related incident, don't immediately jump to:

"The application has an OOM."

Follow an evidence-based approach.

Step 1 — Check Pod status

kubectl get pod <pod> -n <namespace>

Example:

NAME        READY   STATUS             RESTARTS
myapp-01    0/1     CrashLoopBackOff   8

Step 2 — Describe the Pod

kubectl describe pod <pod> -n <namespace>

Look for:

Last State:
  Terminated:
    Reason: OOMKilled
    Exit Code: 137

Also check Events.

Step 3 — Check container logs

kubectl logs <pod> -n <namespace>

If the container has restarted:

kubectl logs <pod> -n <namespace> --previous

The --previous option is particularly useful because the current container may have already restarted.

10. Check Resource Requests and Limits

Inspect the Pod configuration:

kubectl get pod <pod> -n <namespace> -o yaml

Look for:

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "2Gi"

Ask:

Is the memory limit too low?
Is the application memory usage increasing?
Is there a memory leak?
Are requests and limits configured correctly?
Has application behavior changed recently?

11. Check the Node

Find which node is running the Pod:

kubectl get pod <pod> -n <namespace> -o wide

Then:

kubectl describe node <node-name>

Check:

MemoryPressure
Allocatable memory
Allocated resources
Conditions
Events

12. Check Node Memory from Linux

On the affected node:

free -h

top

vmstat 1

ps aux --sort=-%mem

Also check:

df -h

Although df -h primarily checks filesystem capacity rather than RAM, it is useful during broader node-health investigations because memory incidents can occur alongside other resource exhaustion problems.

13. Check Kernel Evidence

Finally:

dmesg -T | grep -i -E "oom|out of memory|killed process"

or:

journalctl -k | grep -i -E "oom|out of memory|killed process"

Now correlate:

Kubernetes evidence
        +
Container evidence
        +
Node evidence
        +
Linux kernel evidence
        ↓
Root Cause

This is the approach an experienced SRE should follow.

14. Example Production RCA

Suppose an application Pod restarted several times.

You run:

kubectl describe pod payment-01 -n production

and find:

Last State:
  Terminated:
    Reason: OOMKilled
    Exit Code: 137

The Pod configuration shows:

resources:
  requests:
    memory: "1Gi"
  limits:
    memory: "2Gi"

Monitoring shows the application memory usage increased steadily:

10:00 → 1.1 GiB
10:10 → 1.4 GiB
10:20 → 1.7 GiB
10:30 → 1.9 GiB
10:35 → 2.0 GiB
10:36 → OOMKilled

The evidence indicates:

Application memory consumption increased
             ↓
Container reached 2 GiB limit
             ↓
Container was killed
             ↓
Kubernetes reported OOMKilled
             ↓
Container restarted

This is primarily a container-level OOM, not proof of a node-level OOM.

15. Example Node-Level Incident

Now consider:

Node memory = 64 GiB

Pod A = 15 GiB
Pod B = 12 GiB
Pod C = 18 GiB
Pod D = 10 GiB
System = 8 GiB

The node becomes heavily memory constrained.

You observe:

kubectl describe node worker-01

MemoryPressure: True

Then:

journalctl -k | grep -i oom

returns:

Out of memory: Killed process 12345 (java)

Now you have evidence of a node-level memory exhaustion event.

The impact can potentially extend beyond one application.

16. Interview Question

A common senior Kubernetes SRE interview question is:

A Pod shows OOMKilled. Does that mean the Kubernetes node experienced an OOM?

Correct answer

No.

OOMKilled indicates that the container/process was killed because of an out-of-memory condition, commonly because the container exceeded its configured memory limit.

It does not by itself prove that the Kubernetes node experienced a system-wide OOM.

To determine whether the node itself experienced memory exhaustion, check:

kubectl describe node <node>

Look for:

MemoryPressure: True

Then check the node's Linux/kernel evidence:

free -h

dmesg -T | grep -i oom

journalctl -k | grep -i oom

This distinction is extremely important in production troubleshooting.

17. Senior SRE Interview Answer

If an interviewer asks:

"Explain the difference between Container OOM and Node OOM."

A strong answer would be:

Container OOM is a container-level memory exhaustion condition, typically occurring when a container exceeds its configured memory limit. Kubernetes may report the container termination as OOMKilled, and the container may subsequently restart.
Node OOM is a node-level memory exhaustion condition where the overall system runs critically low on memory. The Linux kernel may invoke the OOM killer and terminate processes, potentially affecting multiple workloads. Kubernetes may also detect node memory pressure and evict Pods before the system reaches a hard kernel OOM condition.
During troubleshooting, I would correlate Pod termination reasons, resource requests and limits, node MemoryPressure, kubelet events, node memory utilization, and Linux kernel logs to determine the actual root cause.

That demonstrates production troubleshooting rather than simple Kubernetes theory.

18. Quick SRE Cheat Sheet

                 MEMORY INCIDENT
                       │
                       ▼
               Check Pod status
                       │
                       ▼
              kubectl describe pod
                       │
              ┌────────┴────────┐
              ▼                 ▼
          OOMKilled?          Other?
              │                 │
              ▼                 ▼
       Check limits          Continue
       Check usage           troubleshooting
       Check logs
              │
              ▼
       Check affected node
              │
              ▼
     MemoryPressure = True?
              │
        ┌─────┴─────┐
        ▼           ▼
       YES          NO
        │            │
        ▼            ▼
 Check node       Investigate
 memory           container/app
        │
        ▼
 Check kernel logs
        │
        ▼
 Root Cause

Conclusion

The most important lesson is:

OOMKilled does not automatically mean "the Kubernetes node ran out of memory."

Always determine the scope of the memory problem:

Container
   ↓
Pod
   ↓
Node
   ↓
Cluster

For a Kubernetes SRE, the goal is not simply to identify that "OOM happened."

The real goal is to answer:

What ran out of memory? Why did it happen? What evidence proves it? What was the impact? How do we fix it? And how do we prevent it from happening again?

Cloud DevOps Engineer ( Raj Gupta)

Wednesday, 2 September 2026

Container OOM vs Node OOM in Kubernetes

1. Container OOM vs Node OOM

2. What is Container OOM?

3. What does Exit Code 137 mean?

4. What is Node OOM?

5. How to identify Node Memory Pressure

6. Check Linux Kernel Logs

7. The SRE Mental Model

Container OOM

Node-level OOM

8. Don't confuse OOM with Pod Eviction

Situation 1 — Container OOM

Situation 2 — Node Memory Pressure

Situation 3 — Node-level Linux OOM

9. Production Troubleshooting Methodology

Step 1 — Check Pod status

Step 2 — Describe the Pod

Step 3 — Check container logs

10. Check Resource Requests and Limits

11. Check the Node

12. Check Node Memory from Linux

13. Check Kernel Evidence

14. Example Production RCA

15. Example Node-Level Incident

16. Interview Question

Correct answer

17. Senior SRE Interview Answer

18. Quick SRE Cheat Sheet

Conclusion

No comments:

Post a Comment