Short Description
Build a read-only AI-powered Kubernetes SRE agent that investigates pod incidents using status, logs, previous logs, and Kubernetes events. The agent dynamically collects evidence, forms and validates hypotheses, and generates a structured RCA with root-cause confidence—without making unsupported assumptions or performing remediation.
(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent> kubectl get pod
NAME READY STATUS RESTARTS AGE
app-crash-5cf8866df5-q6k4m 0/1 CrashLoopBackOff 8 (40s ago) 17m
broken-pod 0/1 ImagePullBackOff 0 258d
payment-api-987c667b8-xrqhf 0/1 CrashLoopBackOff 45 (3m1s ago) 20h
(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent> python -m agents.sre_agent
Enter namespace [default]: default
Enter pod name: app-crash-5cf8866df5-q6k4m
======================================================================
KUBERNETES SRE INVESTIGATION
======================================================================
Namespace: default
Pod: app-crash-5cf8866df5-q6k4m
======================================================================
AGENT ITERATION 1
======================================================================
======================================================================
TOOL CALL
======================================================================
Tool: get_pod_status
Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}
[K8S] Using local kubeconfig
======================================================================
AGENT ITERATION 2
======================================================================
======================================================================
TOOL CALL
======================================================================
Tool: get_pod_previous_logs
Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}
[K8S] Using local kubeconfig
======================================================================
TOOL CALL
======================================================================
Tool: get_pod_events
Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}
[K8S] Using local kubeconfig
======================================================================
AGENT ITERATION 3
======================================================================
======================================================================
TOOL CALL
======================================================================
Tool: get_pod_status
Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}
[K8S] Using local kubeconfig
======================================================================
AGENT ITERATION 4
======================================================================
======================================================================
FINAL SRE REPORT
======================================================================
SYMPTOM
- Pod default/app-crash-5cf8866df5-q6k4m is failing with CrashLoopBackOff; container is not ready and has multiple restarts.
EVIDENCE
- Pod status (facts):
- phase: Running
- container "busybox": ready=false, restart_count=8, state=waiting, reason=CrashLoopBackOff, message="back-off 5m0s restarting failed container=busybox pod=app-crash-5cf8866df5-q6k4m_default(69fb3e78-994f-42cd-98f2-981bbfa58147)"
- node: minikube, pod_ip: 10.244.0.14
- Pod previous container logs (facts):
- "Application started"
- "ERROR database connection refused"
- Pod events (facts):
- Scheduled on minikube successfully.
- Image pulls for "busybox" succeeded repeatedly.
- Warning BackOff: "Back-off restarting failed container busybox..." count=70 between 05:50:54 and 06:06:01 UTC.
- No warnings/errors about image pull failures, scheduling failures, or probe failures in the events listed.
HYPOTHESIS
- The application starts, immediately attempts a database connection, receives "connection refused," exits, and the container restarts. Repeated exits lead to CrashLoopBackOff.
- The underlying reason for "connection refused" could be incorrect DB host/port, missing/incorrect service/endpoints, network policy, credentials, or the DB process not accepting connections. This is not confirmed.
VALIDATION
- Correlation:
- Pod status shows CrashLoopBackOff with 8 restarts (FACT).
- Events show repeated back-off restarts and successful image pulls/scheduling (FACT), indicating the failure occurs during/after container start, not during scheduling or image pulling.
- Previous logs show the last messages before termination include "ERROR database connection refused" (FACT), consistent with the container exiting shortly after startup.
- What is not proven:
- We do not have the container’s last termination exit code or reason (e.g., Error vs OOMKilled), so we cannot assert the exact process exit details (FACT). However, there is no evidence of OOM or node pressure in events, and the log content points to an application-level exit after a DB connection failure (INFERENCE).
ROOT CAUSE
- Immediate cause confirmed: The container repeatedly exits after encountering "ERROR database connection refused" at startup, causing CrashLoopBackOff.
- Underlying cause of the database connection refusal (misconfiguration, network, service/endpoints, DB availability, credentials) is not established with the current evidence.
- Root cause not confirmed.
CONFIDENCE
- Medium. Logs and events clearly align with application exit due to a DB connection refusal, but lack of last termination details and no verification of the database endpoint/config prevents confirming the underlying root cause.
NEXT INVESTIGATION
- Retrieve container last termination details (exitCode, reason, signal) to confirm process exit conditions.
- Verify DB connectivity configuration used by this pod:
- Environment variables, config maps, and secrets for DB host, port, database name, and credentials.
- Container command/entrypoint to see if it exits on first connection failure.
- Validate the Kubernetes Service backing the database:
- Confirm the Service exists, DNS name is correct, and port matches what the app uses.
- Check Endpoints for the Service to ensure there are ready targets.
- Network path checks:
- Confirm no NetworkPolicy or firewall rules block traffic from this pod’s namespace to the DB.
- If possible, run a temporary debug pod in the same namespace to test TCP connectivity to the DB host:port.
- Inspect the database side:
- Ensure the DB process is listening on the expected interface/port and accepts connections from the pod’s network.
- Review DB logs for connection attempts/refusals at the corresponding timestamps.
- If liveness/readiness probes are configured for this pod, confirm they aren’t killing the container; correlate probe failures if any appear in describe/events.
(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent>
================================================
As a Kubernetes SRE, troubleshooting a production incident usually involves collecting information from multiple sources:
Pod status
Container restart count
Exit codes
Application logs
Previous container logs
Kubernetes events
Node information
Scheduling information
Application symptoms
The challenge is not simply collecting this information.
The real challenge is deciding:
What should I investigate next based on the evidence I already have?
This is where Agentic AI becomes interesting.
Instead of creating a Python script that always executes:
get pod status
↓
get logs
↓
get eventswe can build an AI agent that dynamically decides what information it needs next.
The investigation flow is:
Incident
↓
Observe
↓
Collect Evidence
↓
Reason
↓
Create Hypothesis
↓
Validate Hypothesis
↓
Collect More Evidence if Required
↓
Confirm Root Cause
↓
Generate RCAThis follows the structured investigation approach described : Incident → Evidence → Hypothesis → Validation → Root Cause → Confidence → Next Investigation.
1. What Are We Building?
We are going to build a Python-based Kubernetes SRE Agent.
The user provides:
Namespace
Pod NameThe AI agent then investigates the pod using Kubernetes tools.
The architecture is:
USER
|
v
+---------------+
| SRE AGENT |
| LLM |
+-------+-------+
|
Dynamic Planning
|
+-------------+-------------+
| | |
v v v
Pod Status Logs Events
| | |
+-------------+-------------+
|
v
Evidence
|
v
Hypothesis
|
v
Validation
|
v
Root Cause Confirmed?
/ \
NO YES
| |
v v
More Investigation RCA
|
v
Additional ToolsThe key difference from a traditional script is:
The LLM decides which tool to call based on the current evidence.
2. What Will Our Agent Be Able to Investigate?
Our first version will support four Kubernetes investigation tools:
1. get_pod_status()
2. get_pod_logs()
3. get_pod_previous_logs()
4. get_pod_events()These allow the agent to investigate common conditions such as:
CrashLoopBackOff
ImagePullBackOff
Pending
Container restart
Application startup failure
Scheduling failure3. Why FACT, HYPOTHESIS and ROOT CAUSE Must Be Different
This is one of the most important concepts in an SRE AI agent.
Suppose the application log says:
ERROR database connection refusedA weak AI agent might immediately report:
Root Cause:
Database is down.That is not necessarily correct.
The actual evidence only proves:
FACT:
Application received "connection refused".We can infer:
INFERENCE:
Application could not establish a database connection.We can form:
HYPOTHESIS:
There may be a database connectivity problem.But we cannot yet claim:
ROOT CAUSE:
Database is down.We need more evidence.
Potential causes could include:
Database server
Database Service
Service endpoints
DNS
NetworkPolicy
Firewall
Credentials
Wrong hostname
Wrong port
Application configurationTherefore:
FACT != HYPOTHESIS
HYPOTHESIS != ROOT CAUSE4. Project Directory Structure
Create the following project:
k8s-day5-sre-agent/
│
├── .venv/
│
├── agents/
│ ├── __init__.py
│ └── sre_agent.py
│
├── tools/
│ ├── __init__.py
│ └── agent_tools.py
│
├── models/
│ ├── __init__.py
│ └── incident.py
│
├── scenarios/
│ ├── app-pending.yaml
│ ├── create_incidents.ps1
│ └── cleanup.ps1
│
├── tests/
│ ├── __init__.py
│ └── test_tools.py
│
├── .env
├── .gitignore
├── requirements.txt
└── README.mdThe important components are:
agents/
AI agent
tools/
Kubernetes tools
models/
Structured incident model
scenarios/
Test incidents
tests/
Tool testing5. Create the Project
On Windows PowerShell:
mkdir k8s-day5-sre-agent
cd k8s-day5-sre-agentCreate the virtual environment:
python -m venv .venvActivate it:
.\.venv\Scripts\Activate.ps1Verify:
python --versionYou should see:
Python 3.x.x6. Install Python Dependencies
Create:
requirements.txtAdd:
openai
kubernetes
python-dotenv
pydanticInstall:
pip install -r requirements.txtVerify:
pip list7. Configure the OpenAI API Key
Create:
.envAdd:
OPENAI_API_KEY=YOUR_API_KEY_HEREFor example:
OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxDo not commit .env to Git.
8. Create .gitignore
Create:
.gitignoreAdd:
.venv/
.env
__pycache__/
*.pycThis prevents credentials and Python-generated files from being committed.
9. Kubernetes Tool Layer
The AI agent itself should not directly implement Kubernetes API calls everywhere.
Instead, we create a dedicated tool layer:
tools/agent_tools.pyThis provides a clean separation:
AI Agent
|
v
Tool Layer
|
v
Kubernetes API10. Complete tools/agent_tools.py
Create:
tools/agent_tools.pyUse the following complete code:
from kubernetes import client, config
from kubernetes.config.config_exception import ConfigException
def load_kubernetes():
"""
Load Kubernetes configuration.
First try in-cluster configuration.
If that fails, use the local kubeconfig.
"""
try:
config.load_incluster_config()
print("[K8S] Using in-cluster configuration")
except ConfigException:
config.load_kube_config()
print("[K8S] Using local kubeconfig")
def get_pod_status(
namespace: str,
pod_name: str
):
"""
Get Kubernetes pod status.
Returns:
- pod phase
- node
- pod IP
- container state
- restart count
- exit code
- termination reason
"""
load_kubernetes()
v1 = client.CoreV1Api()
pod = v1.read_namespaced_pod(
name=pod_name,
namespace=namespace
)
containers = []
if pod.status.container_statuses:
for container in pod.status.container_statuses:
state = {}
if container.state:
if container.state.waiting:
state = {
"state": "waiting",
"reason": container.state.waiting.reason,
"message": container.state.waiting.message
}
elif container.state.running:
state = {
"state": "running",
"started_at": str(
container.state.running.started_at
)
}
elif container.state.terminated:
state = {
"state": "terminated",
"reason": container.state.terminated.reason,
"exit_code": container.state.terminated.exit_code,
"signal": container.state.terminated.signal,
"message": container.state.terminated.message
}
containers.append(
{
"name": container.name,
"ready": container.ready,
"restart_count": container.restart_count,
"state": state
}
)
return {
"pod_name": pod.metadata.name,
"namespace": pod.metadata.namespace,
"phase": pod.status.phase,
"node": pod.spec.node_name,
"pod_ip": pod.status.pod_ip,
"containers": containers
}
def get_pod_logs(
namespace: str,
pod_name: str,
container: str = None
):
"""
Get current logs from a Kubernetes container.
"""
load_kubernetes()
v1 = client.CoreV1Api()
logs = v1.read_namespaced_pod_log(
name=pod_name,
namespace=namespace,
container=container,
tail_lines=100
)
return {
"pod_name": pod_name,
"namespace": namespace,
"log_type": "current",
"logs": logs
}
def get_pod_previous_logs(
namespace: str,
pod_name: str,
container: str = None
):
"""
Get logs from the previous terminated container.
Very useful for CrashLoopBackOff.
"""
load_kubernetes()
v1 = client.CoreV1Api()
logs = v1.read_namespaced_pod_log(
name=pod_name,
namespace=namespace,
container=container,
previous=True,
tail_lines=100
)
return {
"pod_name": pod_name,
"namespace": namespace,
"log_type": "previous",
"logs": logs
}
def get_pod_events(
namespace: str,
pod_name: str
):
"""
Get Kubernetes events associated with a pod.
"""
load_kubernetes()
v1 = client.CoreV1Api()
events = v1.list_namespaced_event(
namespace=namespace,
field_selector=(
f"involvedObject.name={pod_name}"
)
)
result = []
for event in events.items:
result.append(
{
"type": event.type,
"reason": event.reason,
"message": event.message,
"count": event.count,
"first_timestamp": str(
event.first_timestamp
),
"last_timestamp": str(
event.last_timestamp
)
}
)
return {
"pod_name": pod_name,
"namespace": namespace,
"events": result
}11. Important Troubleshooting Lesson
During implementation, one common mistake is accidentally creating:
tools/agent_tools.pyas an empty file.
For example:
agent_tools.py 0 bytesThen this command:
python -c "from tools.agent_tools import get_pod_status"will produce:
ImportError:
cannot import name 'get_pod_status'The reason is simple:
sre_agent.py
|
| imports
v
agent_tools.py
|
X
|
No functionsAfter adding the code above, verify:
Get-Item .\tools\agent_tools.pyThe file should have a size greater than zero.
12. Test the Kubernetes Tool Import
Run:
python -c "from tools.agent_tools import get_pod_status; print('TOOLS IMPORT OK')"Expected:
TOOLS IMPORT OKTest all four:
python -c "from tools.agent_tools import get_pod_status, get_pod_logs, get_pod_previous_logs, get_pod_events; print('ALL 4 TOOLS IMPORT OK')"Expected:
ALL 4 TOOLS IMPORT OK13. Incident Model
Create:
models/incident.pyCode:
from typing import List
from pydantic import BaseModel, Field
class Evidence(BaseModel):
fact: str
source: str
class IncidentReport(BaseModel):
incident: str
symptom: str
evidence: List[Evidence] = Field(
default_factory=list
)
hypothesis: str
validation: List[str] = Field(
default_factory=list
)
root_cause: str
confidence: str
next_investigation: List[str] = Field(
default_factory=list
)This allows us to represent the investigation in a structured format.
The intended report structure is:
SYMPTOM
EVIDENCE
HYPOTHESIS
VALIDATION
ROOT CAUSE
CONFIDENCE
NEXT INVESTIGATION14. Building the AI SRE Agent
Now create:
agents/sre_agent.pyThe agent will:
Accept namespace and pod name.
Ask the LLM to investigate.
Allow the LLM to select tools.
Execute the selected Kubernetes tool.
Send the result back to the LLM.
Allow the LLM to select another tool if required.
Continue until sufficient evidence exists.
Generate the final SRE report.
15. Complete agents/sre_agent.py
Use this complete code:
import json
import os
from dotenv import load_dotenv
from openai import OpenAI
from tools.agent_tools import (
get_pod_status,
get_pod_logs,
get_pod_previous_logs,
get_pod_events,
)
# ============================================================
# ENVIRONMENT
# ============================================================
load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError(
"OPENAI_API_KEY is missing. "
"Create a .env file in the project root."
)
# ============================================================
# OPENAI CLIENT
# ============================================================
client = OpenAI(
api_key=api_key
)
# ============================================================
# SYSTEM PROMPT
# ============================================================
SYSTEM_PROMPT = """
You are a senior Kubernetes SRE investigation agent.
Your job is to investigate Kubernetes incidents
using available READ-ONLY tools.
Investigation rules:
1. Never invent cluster state.
2. Always collect evidence before making conclusions.
3. Clearly distinguish FACT from INFERENCE.
4. Clearly distinguish HYPOTHESIS from CONFIRMED ROOT CAUSE.
5. Never claim a root cause without supporting evidence.
6. Use additional tools when evidence is insufficient.
7. Do not perform remediation.
8. Prefer the smallest number of tools necessary.
9. Correlate pod status, logs and Kubernetes events.
10. If the root cause cannot be proven, explicitly say:
Root cause not confirmed.
Important:
Exit code 137 does NOT automatically prove OOMKilled.
Database connection refused does NOT automatically prove
that the database is down.
ImagePullBackOff should be investigated using Kubernetes
events and pod status.
Pending pods should be investigated using pod status and
Kubernetes scheduling events.
The final answer must contain:
SYMPTOM
EVIDENCE
HYPOTHESIS
VALIDATION
ROOT CAUSE
CONFIDENCE
NEXT INVESTIGATION
"""
# ============================================================
# TOOL DEFINITIONS
# ============================================================
TOOLS = [
{
"type": "function",
"name": "get_pod_status",
"description": (
"Get Kubernetes pod status, container states, "
"restart counts, exit codes and node information."
),
"parameters": {
"type": "object",
"properties": {
"namespace": {
"type": "string"
},
"pod_name": {
"type": "string"
}
},
"required": [
"namespace",
"pod_name"
]
}
},
{
"type": "function",
"name": "get_pod_logs",
"description": (
"Get current logs from a Kubernetes pod."
),
"parameters": {
"type": "object",
"properties": {
"namespace": {
"type": "string"
},
"pod_name": {
"type": "string"
}
},
"required": [
"namespace",
"pod_name"
]
}
},
{
"type": "function",
"name": "get_pod_previous_logs",
"description": (
"Get logs from the previous terminated "
"container instance. Useful for CrashLoopBackOff."
),
"parameters": {
"type": "object",
"properties": {
"namespace": {
"type": "string"
},
"pod_name": {
"type": "string"
}
},
"required": [
"namespace",
"pod_name"
]
}
},
{
"type": "function",
"name": "get_pod_events",
"description": (
"Get Kubernetes events associated with a pod. "
"Useful for scheduling, image pull, startup and "
"restart failures."
),
"parameters": {
"type": "object",
"properties": {
"namespace": {
"type": "string"
},
"pod_name": {
"type": "string"
}
},
"required": [
"namespace",
"pod_name"
]
}
}
]
# ============================================================
# TOOL EXECUTION
# ============================================================
def execute_tool(name, arguments):
print()
print("=" * 70)
print("TOOL CALL")
print("=" * 70)
print("Tool:", name)
print(
"Arguments:",
json.dumps(
arguments,
indent=2
)
)
try:
if name == "get_pod_status":
return get_pod_status(
arguments["namespace"],
arguments["pod_name"]
)
elif name == "get_pod_logs":
return get_pod_logs(
arguments["namespace"],
arguments["pod_name"]
)
elif name == "get_pod_previous_logs":
return get_pod_previous_logs(
arguments["namespace"],
arguments["pod_name"]
)
elif name == "get_pod_events":
return get_pod_events(
arguments["namespace"],
arguments["pod_name"]
)
else:
return {
"error": f"Unknown tool: {name}"
}
except Exception as exc:
return {
"error": str(exc)
}
# ============================================================
# INVESTIGATION ENGINE
# ============================================================
def investigate(
namespace: str,
pod_name: str
):
user_prompt = f"""
Investigate this Kubernetes incident.
Namespace:
{namespace}
Pod:
{pod_name}
Determine why this pod is failing.
Use evidence.
Do not make assumptions.
Do not perform remediation.
Use additional tools if the available evidence
is insufficient.
At the end produce a structured SRE incident report.
"""
messages = [
{
"role": "system",
"content": SYSTEM_PROMPT
},
{
"role": "user",
"content": user_prompt
}
]
max_iterations = 10
for iteration in range(max_iterations):
print()
print("=" * 70)
print(
f"AGENT ITERATION {iteration + 1}"
)
print("=" * 70)
response = client.responses.create(
model="gpt-5",
input=messages,
tools=TOOLS
)
tool_outputs = []
for item in response.output:
if item.type == "function_call":
name = item.name
arguments = json.loads(
item.arguments
)
result = execute_tool(
name,
arguments
)
tool_outputs.append(
{
"type": "function_call_output",
"call_id": item.call_id,
"output": json.dumps(
result,
default=str
)
}
)
# ----------------------------------------------------
# No more tools required
# ----------------------------------------------------
if not tool_outputs:
return response.output_text
# ----------------------------------------------------
# Send assistant tool request back to conversation
# ----------------------------------------------------
messages.extend(
response.output
)
# ----------------------------------------------------
# Send tool results back to LLM
# ----------------------------------------------------
messages.extend(
tool_outputs
)
return (
"Investigation stopped because maximum "
"agent iterations were reached."
)
# ============================================================
# MAIN
# ============================================================
def main():
namespace = input(
"Enter namespace [default]: "
).strip()
if not namespace:
namespace = "default"
pod_name = input(
"Enter pod name: "
).strip()
if not pod_name:
print(
"Pod name is required."
)
return
print()
print("=" * 70)
print(
"KUBERNETES SRE INVESTIGATION"
)
print("=" * 70)
print(
"Namespace:",
namespace
)
print(
"Pod:",
pod_name
)
result = investigate(
namespace,
pod_name
)
print()
print("=" * 70)
print(
"FINAL SRE REPORT"
)
print("=" * 70)
print(result)
if __name__ == "__main__":
main()16. Why We Use python -m
Our project uses:
k8s-day5-sre-agent/
│
├── agents/
│ └── sre_agent.py
│
└── tools/
└── agent_tools.pyTherefore, run:
python -m agents.sre_agentDo not normally run:
python .\agents\sre_agent.pyThe module approach allows Python to correctly find:
from tools.agent_tools import ...17. Verify the Agent Import
Run:
python -c "import agents.sre_agent; print('AGENT IMPORT OK')"Expected:
AGENT IMPORT OKIf you get:
ModuleNotFoundError: No module named 'tools'you are probably executing the file directly instead of using:
python -m agents.sre_agent18. Verify the API Key
Run:
python -c "from dotenv import load_dotenv; import os; load_dotenv(); print('API KEY STATUS:', 'SET' if os.getenv('OPENAI_API_KEY') else 'NOT SET')"Expected:
API KEY STATUS: SETIf you get:
API KEY STATUS: NOT SETcheck:
k8s-day5-sre-agent/
│
└── .envand make sure it contains:
OPENAI_API_KEY=YOUR_API_KEYDo not share your actual API key publicly.
19. Verify Kubernetes
Before running the AI agent:
minikube statusThen:
kubectl get nodesExpected:
NAME STATUS ROLES AGE
minikube Ready control-plane ...Then:
kubectl get pods -A20. Create a CrashLoopBackOff Incident
We need a controlled incident for testing.
Run:
kubectl create deployment app-crash `
--image=busybox `
-- /bin/sh -c "echo Application started; echo ERROR database connection refused; exit 1"Check:
kubectl get podsAfter a while:
NAME READY STATUS RESTARTS
app-crash-xxxxxxxxxx 0/1 CrashLoopBackOff 3Copy the actual pod name.
21. Manually Investigate the Incident
Before allowing AI to investigate, it is useful for an SRE to understand the raw evidence.
Run:
kubectl get pod YOUR_POD_NAME -o wideThen:
kubectl describe pod YOUR_POD_NAMECurrent logs:
kubectl logs YOUR_POD_NAMEPrevious logs:
kubectl logs YOUR_POD_NAME --previousEvents:
kubectl get events --sort-by=.lastTimestampYou may see:
Application started
ERROR database connection refusedand events such as:
BackOff22. Understand the Evidence
The correct reasoning is:
FACT
Application logged:
database connection refusedThen:
INFERENCE
The application could not establish a database connection.Then:
HYPOTHESIS
Database connectivity is a possible cause.But:
ROOT CAUSE
Database is downis not yet proven.
This is the difference between an AI chatbot that generates plausible answers and an SRE investigation agent that works from evidence.
23. Run the AI Agent
Run:
python -m agents.sre_agentYou should see:
Enter namespace [default]:Enter:
defaultThen:
Enter pod name:Enter your actual pod:
app-crash-xxxxxxxxxx24. What Happens Inside the Agent?
The first iteration may look conceptually like:
AGENT ITERATION 1
|
v
LLM asks:
"What is the pod state?"
|
v
get_pod_status()The tool returns:
phase = Running
restart_count = 3
container state = terminated
exit_code = 1The LLM then reasons:
The container is restarting.
I need logs.It calls:
get_pod_previous_logs()The result:
Application started
ERROR database connection refusedThe LLM may then call:
get_pod_events()The result:
BackOffNow it has multiple pieces of evidence.
25. Dynamic Investigation
The important point is that the Python program does not explicitly contain:
if status == "CrashLoopBackOff":
get_logs()
if logs contain "database":
get_events()Instead:
LLM
|
+--> get_pod_status()
|
+--> inspect result
|
+--> decide next action
|
+--> get_pod_previous_logs()
|
+--> inspect result
|
+--> decide next action
|
+--> get_pod_events()
|
+--> inspect result
|
+--> generate reportThis is the beginning of agentic behavior.
26. Expected SRE Report
The exact wording generated by the LLM may vary, but the report should resemble:
SYMPTOM
-------
Pod app-crash-xxxx is repeatedly restarting
and is currently in CrashLoopBackOff.
EVIDENCE
--------
1. Container restart count is greater than zero.
2. Container terminated with exit code 1.
3. Previous container logs contain:
"ERROR database connection refused"
4. Kubernetes events show BackOff.
HYPOTHESIS
----------
The application is failing because it cannot
establish database connectivity.
VALIDATION
----------
The previous container logs confirm that the
application reported a database connection failure.
However, database availability itself has not
been independently verified.
ROOT CAUSE
----------
Root cause not confirmed.
The evidence confirms an application-level
database connection failure but does not prove
that the database server itself is down.
CONFIDENCE
----------
Medium
NEXT INVESTIGATION
------------------
1. Check database Service.
2. Check Service endpoints.
3. Check DNS resolution.
4. Check NetworkPolicy.
5. Check database availability.
6. Validate application database configuration.This is consistent with the Day-5 target report structure.
27. Incident 2 — ImagePullBackOff
Now create a bad image:
kubectl create deployment app-image `
--image=nginx:this-image-does-not-existCheck:
kubectl get podsYou should see something similar to:
app-image-xxxxxxxxxx 0/1 ImagePullBackOffNow run:
python -m agents.sre_agentEnter:
defaultand the actual pod name.
The expected investigation pattern is:
Pod status
|
v
ImagePullBackOff
|
v
Kubernetes events
|
v
Failed to pull image
|
v
DiagnosisThe agent should not treat this like a CrashLoopBackOff investigation.
That is the purpose of dynamic planning.
28. Incident 3 — Pending Pod
A normal Minikube cluster may successfully schedule:
nginxTherefore, simply creating an nginx deployment does not guarantee Pending.
Instead, deliberately create an impossible node selector.
Create:
scenarios/app-pending.yamlUse:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-pending
spec:
replicas: 1
selector:
matchLabels:
app: app-pending
template:
metadata:
labels:
app: app-pending
spec:
nodeSelector:
sre-test-node: "does-not-exist"
containers:
- name: nginx
image: nginxApply:
kubectl apply -f scenarios/app-pending.yamlCheck:
kubectl get podsExpected:
app-pending-xxxxxxxxxx 0/1 Pending29. Investigate Pending
Run:
kubectl describe pod YOUR_PENDING_PODNear the bottom you should see scheduling events similar to:
Events:
Warning FailedScheduling
0/1 nodes are available:
1 node(s) didn't match Pod's node affinity/selector.The agent should follow:
Pending
|
v
Pod Status
|
v
FailedScheduling
|
v
Kubernetes Events
|
v
Node selector / scheduling constraint30. Test Kubernetes Tools Independently
Create:
tests/test_tools.pyUse:
from tools.agent_tools import (
get_pod_status,
get_pod_logs,
get_pod_previous_logs,
get_pod_events
)
POD_NAME = "YOUR_POD_NAME"
NAMESPACE = "default"
print("=" * 70)
print("TEST 1: POD STATUS")
print("=" * 70)
result = get_pod_status(
NAMESPACE,
POD_NAME
)
print(result)
print()
print("=" * 70)
print("TEST 2: CURRENT LOGS")
print("=" * 70)
result = get_pod_logs(
NAMESPACE,
POD_NAME
)
print(result)
print()
print("=" * 70)
print("TEST 3: PREVIOUS LOGS")
print("=" * 70)
try:
result = get_pod_previous_logs(
NAMESPACE,
POD_NAME
)
print(result)
except Exception as exc:
print("Previous logs unavailable:")
print(exc)
print()
print("=" * 70)
print("TEST 4: EVENTS")
print("=" * 70)
result = get_pod_events(
NAMESPACE,
POD_NAME
)
print(result)Run:
python tests/test_tools.pyThis lets us separate:
Kubernetes problemfrom:
AI agent problemThat is an important engineering practice.
31. Cleanup Test Incidents
Create:
scenarios/cleanup.ps1Use:
Write-Host "Cleaning SRE test incidents..."
kubectl delete deployment app-crash --ignore-not-found
kubectl delete deployment app-image --ignore-not-found
kubectl delete deployment app-pending --ignore-not-found
Write-Host ""
Write-Host "Cleanup complete."
kubectl get podsRun:
.\scenarios\cleanup.ps132. Create an Automated Incident Generator
Create:
scenarios/create_incidents.ps1Code:
Write-Host "============================================"
Write-Host "Creating Kubernetes SRE Test Incidents"
Write-Host "============================================"
Write-Host ""
Write-Host "[1] Creating CrashLoopBackOff incident"
kubectl create deployment app-crash `
--image=busybox `
-- /bin/sh -c "echo Application started; echo ERROR database connection refused; exit 1"
Write-Host ""
Write-Host "[2] Creating ImagePullBackOff incident"
kubectl create deployment app-image `
--image=nginx:this-image-does-not-exist"
Write-Host ""
Write-Host "[3] Creating Pending incident"
kubectl apply -f scenarios/app-pending.yaml
Write-Host ""
Write-Host "============================================"
Write-Host "Incidents created"
Write-Host "============================================"
kubectl get podsRun:
.\scenarios\create_incidents.ps133. Complete End-to-End Test
Now the complete workflow becomes:
cd k8s-day5-sre-agent
.\.venv\Scripts\Activate.ps1
minikube status
kubectl get nodes
pip install -r requirements.txt
.\scenarios\create_incidents.ps1
kubectl get pods
python -m agents.sre_agentInvestigate the CrashLoopBackOff pod.
Then:
python -m agents.sre_agentInvestigate the ImagePullBackOff pod.
Then:
python -m agents.sre_agentInvestigate the Pending pod.
Finally:
.\scenarios\cleanup.ps134. Complete Architecture
At this point the project looks like:
USER
|
v
+-------------------+
| SRE AGENT |
| LLM |
+---------+---------+
|
|
Dynamic Planning
|
+----------------+----------------+
| | |
v v v
get_pod_status() get_pod_logs() get_pod_events()
|
|
v
get_pod_previous_logs()
|
+----------------+
|
v
EVIDENCE
|
v
HYPOTHESIS
|
v
VALIDATION
|
v
ROOT CAUSE CHECK
/ \
/ \
NO YES
| |
v v
More investigation RCA
|
v
More tools35. Traditional Script vs Agentic SRE
A traditional script may look like:
get status
↓
get logs
↓
get events
↓
print outputThe agentic approach is:
get status
↓
reason
↓
decide what evidence is missing
↓
get relevant tool
↓
reason again
↓
validate hypothesis
↓
decide whether root cause is proven
↓
generate RCAThis is a significant architectural difference.
36. Example: Exit Code 137
One of the important rules in our system is:
Exit code 137must not automatically become:
OOMKilledWhy?
Because:
137 = 128 + 9which corresponds to:
SIGKILLPossible explanations require additional evidence.
The investigation should therefore be:
Exit code 137
|
v
SIGKILL
|
v
Possible OOM
|
v
Collect evidence
|
+----> Container state
|
+----> Kubernetes events
|
+----> Node memory
|
+----> Container memory limit
|
v
Confirm / Reject OOM hypothesisThis evidence-first reasoning is specifically emphasized in the Day-5 material.
37. Example: Database Connection Refused
Consider:
ERROR database connection refusedThe agent should reason:
FACT
Application received connection refused.
↓
INFERENCE
Application could not establish DB connection.
↓
HYPOTHESIS
Database connectivity problem.
↓
VALIDATION
Need more evidence.
↓
INVESTIGATE
Service
Endpoints
DNS
NetworkPolicy
Database
Credentials
Configuration
↓
ROOT CAUSE
Only confirmed when evidence supports it.This is much safer than letting an LLM invent a root cause.
38. Read-Only Safety Model
Our Day-5 agent is intentionally read-only.
The tools can:
READ pod status
READ logs
READ previous logs
READ eventsThey cannot:
DELETE pod
DELETE deployment
RESTART pod
SCALE deployment
PATCH deployment
CHANGE node
CHANGE network policy
CHANGE storageThis is an important production design principle.
The investigation agent should first establish:
What happened?before eventually being allowed to answer:
What should we do?And remediation should be a separate controlled capability.
39. Why Previous Logs Matter
For a normal running container:
kubectl logs PODmay be enough.
But for:
CrashLoopBackOffthe currently running container may have little or no useful information.
The previous container may contain:
startup logs
exception
stack trace
configuration failure
connection failure
termination reasonTherefore:
CrashLoopBackOff
|
v
Previous Logs
|
v
Failure Evidenceis an important SRE troubleshooting pattern.
40. Why Kubernetes Events Matter
Application logs tell us what the application experienced.
Kubernetes events tell us what Kubernetes experienced.
For example:
Application logs
----------------
database connection refusedversus:
Kubernetes events
-----------------
FailedScheduling
Failed to pull image
BackOff
FailedMount
UnhealthyAn SRE agent should correlate both.
Therefore:
Application Evidence
+
Kubernetes Evidence
=
Better Incident Diagnosis41. Current Project Capabilities
After completing Day 5, our agent can:
✓ Connect to Kubernetes
✓ Inspect pod status
✓ Inspect container state
✓ Inspect restart counts
✓ Inspect exit codes
✓ Read current logs
✓ Read previous logs
✓ Read Kubernetes events
✓ Dynamically select tools
✓ Collect evidence
✓ Generate hypotheses
✓ Validate hypotheses
✓ Avoid unsupported root-cause claims
✓ Generate structured SRE reportsThe Day-5 goal is to establish this structured evidence-driven investigation loop before moving toward broader observability and multi-agent capabilities.
42. What We Have Learned
The most important lesson from this project is:
An AI SRE agent should not be a chatbot that guesses the root cause.
It should behave more like an experienced SRE.
An experienced SRE thinks:
What do I know?
↓
What don't I know?
↓
What evidence would reduce uncertainty?
↓
Which command should I run?
↓
What did that command tell me?
↓
Does my hypothesis still hold?
↓
What should I investigate next?Our AI agent is beginning to follow the same process.
43. Mental Model
Remember this:
OBSERVE
↓
REASON
↓
HYPOTHESIZE
↓
COLLECT EVIDENCE
↓
VALIDATE
↓
REASON AGAIN
↓
CONFIRM ROOT CAUSE
↓
REPORTThis is the core mental model for building an Agentic AI SRE system.
44. Final Project Tree
The completed project should look like:
k8s-day5-sre-agent/
│
├── .venv/
│
├── agents/
│ ├── __init__.py
│ └── sre_agent.py
│
├── tools/
│ ├── __init__.py
│ └── agent_tools.py
│
├── models/
│ ├── __init__.py
│ └── incident.py
│
├── scenarios/
│ ├── app-pending.yaml
│ ├── create_incidents.ps1
│ └── cleanup.ps1
│
├── tests/
│ ├── __init__.py
│ └── test_tools.py
│
├── .env
├── .gitignore
├── requirements.txt
└── README.md45. Final Execution Checklist
Use this checklist whenever you start the project:
# Activate environment
.\.venv\Scripts\Activate.ps1
# Check Python
python --version
# Check Kubernetes
minikube status
# Check node
kubectl get nodes
# Check API key
python -c "from dotenv import load_dotenv; import os; load_dotenv(); print('API KEY:', 'SET' if os.getenv('OPENAI_API_KEY') else 'NOT SET')"
# Check Kubernetes tools
python -c "from tools.agent_tools import get_pod_status, get_pod_logs, get_pod_previous_logs, get_pod_events; print('ALL 4 TOOLS IMPORT OK')"
# Check agent
python -c "import agents.sre_agent; print('AGENT IMPORT OK')"
# Run agent
python -m agents.sre_agent
