Sunday, 30 August 2026

Building a Kubernetes SRE Incident Investigation Agent with Python & Agentic AI

Short Description

Build a read-only AI-powered Kubernetes SRE agent that investigates pod incidents using status, logs, previous logs, and Kubernetes events. The agent dynamically collects evidence, forms and validates hypotheses, and generates a structured RCA with root-cause confidence—without making unsupported assumptions or performing remediation.





(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent> kubectl get pod

NAME                          READY   STATUS             RESTARTS        AGE

app-crash-5cf8866df5-q6k4m    0/1     CrashLoopBackOff   8 (40s ago)     17m

broken-pod                    0/1     ImagePullBackOff   0               258d

payment-api-987c667b8-xrqhf   0/1     CrashLoopBackOff   45 (3m1s ago)   20h

(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent> python -m agents.sre_agent

Enter namespace [default]: default

Enter pod name: app-crash-5cf8866df5-q6k4m


======================================================================

KUBERNETES SRE INVESTIGATION

======================================================================

Namespace: default

Pod: app-crash-5cf8866df5-q6k4m


======================================================================

AGENT ITERATION 1

======================================================================


======================================================================

TOOL CALL

======================================================================

Tool: get_pod_status

Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}

[K8S] Using local kubeconfig


======================================================================

AGENT ITERATION 2

======================================================================


======================================================================

TOOL CALL

======================================================================

Tool: get_pod_previous_logs

Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}

[K8S] Using local kubeconfig


======================================================================

TOOL CALL

======================================================================

Tool: get_pod_events

Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}

[K8S] Using local kubeconfig


======================================================================

AGENT ITERATION 3

======================================================================


======================================================================

TOOL CALL

======================================================================

Tool: get_pod_status

Arguments: {'namespace': 'default', 'pod_name': 'app-crash-5cf8866df5-q6k4m'}

[K8S] Using local kubeconfig


======================================================================

AGENT ITERATION 4

======================================================================


======================================================================

FINAL SRE REPORT

======================================================================

SYMPTOM

- Pod default/app-crash-5cf8866df5-q6k4m is failing with CrashLoopBackOff; container is not ready and has multiple restarts.


EVIDENCE

- Pod status (facts):

  - phase: Running

  - container "busybox": ready=false, restart_count=8, state=waiting, reason=CrashLoopBackOff, message="back-off 5m0s restarting failed container=busybox pod=app-crash-5cf8866df5-q6k4m_default(69fb3e78-994f-42cd-98f2-981bbfa58147)"

  - node: minikube, pod_ip: 10.244.0.14

- Pod previous container logs (facts):

  - "Application started"

  - "ERROR database connection refused"

- Pod events (facts):

  - Scheduled on minikube successfully.

  - Image pulls for "busybox" succeeded repeatedly.

  - Warning BackOff: "Back-off restarting failed container busybox..." count=70 between 05:50:54 and 06:06:01 UTC.

  - No warnings/errors about image pull failures, scheduling failures, or probe failures in the events listed.


HYPOTHESIS

- The application starts, immediately attempts a database connection, receives "connection refused," exits, and the container restarts. Repeated exits lead to CrashLoopBackOff.

- The underlying reason for "connection refused" could be incorrect DB host/port, missing/incorrect service/endpoints, network policy, credentials, or the DB process not accepting connections. This is not confirmed.


VALIDATION

- Correlation:

  - Pod status shows CrashLoopBackOff with 8 restarts (FACT).

  - Events show repeated back-off restarts and successful image pulls/scheduling (FACT), indicating the failure occurs during/after container start, not during scheduling or image pulling.

  - Previous logs show the last messages before termination include "ERROR database connection refused" (FACT), consistent with the container exiting shortly after startup.

- What is not proven:

  - We do not have the container’s last termination exit code or reason (e.g., Error vs OOMKilled), so we cannot assert the exact process exit details (FACT). However, there is no evidence of OOM or node pressure in events, and the log content points to an application-level exit after a DB connection failure (INFERENCE).


ROOT CAUSE

- Immediate cause confirmed: The container repeatedly exits after encountering "ERROR database connection refused" at startup, causing CrashLoopBackOff.

- Underlying cause of the database connection refusal (misconfiguration, network, service/endpoints, DB availability, credentials) is not established with the current evidence.

- Root cause not confirmed.


CONFIDENCE

- Medium. Logs and events clearly align with application exit due to a DB connection refusal, but lack of last termination details and no verification of the database endpoint/config prevents confirming the underlying root cause.


NEXT INVESTIGATION

- Retrieve container last termination details (exitCode, reason, signal) to confirm process exit conditions.

- Verify DB connectivity configuration used by this pod:

  - Environment variables, config maps, and secrets for DB host, port, database name, and credentials.

  - Container command/entrypoint to see if it exits on first connection failure.

- Validate the Kubernetes Service backing the database:

  - Confirm the Service exists, DNS name is correct, and port matches what the app uses.

  - Check Endpoints for the Service to ensure there are ready targets.

- Network path checks:

  - Confirm no NetworkPolicy or firewall rules block traffic from this pod’s namespace to the DB.

  - If possible, run a temporary debug pod in the same namespace to test TCP connectivity to the DB host:port.

- Inspect the database side:

  - Ensure the DB process is listening on the expected interface/port and accepts connections from the pod’s network.

  - Review DB logs for connection attempts/refusals at the corresponding timestamps.

- If liveness/readiness probes are configured for this pod, confirm they aren’t killing the container; correlate probe failures if any appear in describe/events.

(.venv) PS C:\Users\Raj Kumar Gupta\Desktop\Raj\minikube\k8s-day5-sre-agent> 

================================================

As a Kubernetes SRE, troubleshooting a production incident usually involves collecting information from multiple sources:

  • Pod status

  • Container restart count

  • Exit codes

  • Application logs

  • Previous container logs

  • Kubernetes events

  • Node information

  • Scheduling information

  • Application symptoms

The challenge is not simply collecting this information.

The real challenge is deciding:

What should I investigate next based on the evidence I already have?

This is where Agentic AI becomes interesting.

Instead of creating a Python script that always executes:

get pod status
    ↓
get logs
    ↓
get events

we can build an AI agent that dynamically decides what information it needs next.


The investigation flow is:

Incident
   ↓
Observe
   ↓
Collect Evidence
   ↓
Reason
   ↓
Create Hypothesis
   ↓
Validate Hypothesis
   ↓
Collect More Evidence if Required
   ↓
Confirm Root Cause
   ↓
Generate RCA

This follows the structured investigation approach described : Incident → Evidence → Hypothesis → Validation → Root Cause → Confidence → Next Investigation.

1. What Are We Building?

We are going to build a Python-based Kubernetes SRE Agent.

The user provides:

Namespace
Pod Name

The AI agent then investigates the pod using Kubernetes tools.

The architecture is:

                    USER
                      |
                      v
              +---------------+
              |   SRE AGENT   |
              |      LLM      |
              +-------+-------+
                      |
              Dynamic Planning
                      |
        +-------------+-------------+
        |             |             |
        v             v             v
   Pod Status       Logs          Events
        |             |             |
        +-------------+-------------+
                      |
                      v
                  Evidence
                      |
                      v
                 Hypothesis
                      |
                      v
                  Validation
                      |
                      v
             Root Cause Confirmed?
                /             \
              NO               YES
              |                 |
              v                 v
        More Investigation      RCA
              |
              v
        Additional Tools

The key difference from a traditional script is:

The LLM decides which tool to call based on the current evidence.

2. What Will Our Agent Be Able to Investigate?

Our first version will support four Kubernetes investigation tools:

1. get_pod_status()
2. get_pod_logs()
3. get_pod_previous_logs()
4. get_pod_events()

These allow the agent to investigate common conditions such as:

CrashLoopBackOff
ImagePullBackOff
Pending
Container restart
Application startup failure
Scheduling failure


3. Why FACT, HYPOTHESIS and ROOT CAUSE Must Be Different

This is one of the most important concepts in an SRE AI agent.

Suppose the application log says:

ERROR database connection refused

A weak AI agent might immediately report:

Root Cause:
Database is down.

That is not necessarily correct.

The actual evidence only proves:

FACT:
Application received "connection refused".

We can infer:

INFERENCE:
Application could not establish a database connection.

We can form:

HYPOTHESIS:
There may be a database connectivity problem.

But we cannot yet claim:

ROOT CAUSE:
Database is down.

We need more evidence.

Potential causes could include:

Database server
Database Service
Service endpoints
DNS
NetworkPolicy
Firewall
Credentials
Wrong hostname
Wrong port
Application configuration

Therefore:

FACT != HYPOTHESIS

HYPOTHESIS != ROOT CAUSE


4. Project Directory Structure

Create the following project:

k8s-day5-sre-agent/
│
├── .venv/
│
├── agents/
│   ├── __init__.py
│   └── sre_agent.py
│
├── tools/
│   ├── __init__.py
│   └── agent_tools.py
│
├── models/
│   ├── __init__.py
│   └── incident.py
│
├── scenarios/
│   ├── app-pending.yaml
│   ├── create_incidents.ps1
│   └── cleanup.ps1
│
├── tests/
│   ├── __init__.py
│   └── test_tools.py
│
├── .env
├── .gitignore
├── requirements.txt
└── README.md

The important components are:

agents/
    AI agent

tools/
    Kubernetes tools

models/
    Structured incident model

scenarios/
    Test incidents

tests/
    Tool testing

5. Create the Project

On Windows PowerShell:

mkdir k8s-day5-sre-agent
cd k8s-day5-sre-agent

Create the virtual environment:

python -m venv .venv

Activate it:

.\.venv\Scripts\Activate.ps1

Verify:

python --version

You should see:

Python 3.x.x

6. Install Python Dependencies

Create:

requirements.txt

Add:

openai
kubernetes
python-dotenv
pydantic

Install:

pip install -r requirements.txt

Verify:

pip list

7. Configure the OpenAI API Key

Create:

.env

Add:

OPENAI_API_KEY=YOUR_API_KEY_HERE

For example:

OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxx

Do not commit .env to Git.

8. Create .gitignore

Create:

.gitignore

Add:

.venv/
.env
__pycache__/
*.pyc

This prevents credentials and Python-generated files from being committed.

9. Kubernetes Tool Layer

The AI agent itself should not directly implement Kubernetes API calls everywhere.

Instead, we create a dedicated tool layer:

tools/agent_tools.py

This provides a clean separation:

AI Agent
   |
   v
Tool Layer
   |
   v
Kubernetes API

10. Complete tools/agent_tools.py

Create:

tools/agent_tools.py

Use the following complete code:

from kubernetes import client, config
from kubernetes.config.config_exception import ConfigException


def load_kubernetes():
    """
    Load Kubernetes configuration.

    First try in-cluster configuration.
    If that fails, use the local kubeconfig.
    """

    try:
        config.load_incluster_config()

        print("[K8S] Using in-cluster configuration")

    except ConfigException:

        config.load_kube_config()

        print("[K8S] Using local kubeconfig")


def get_pod_status(
    namespace: str,
    pod_name: str
):
    """
    Get Kubernetes pod status.

    Returns:
        - pod phase
        - node
        - pod IP
        - container state
        - restart count
        - exit code
        - termination reason
    """

    load_kubernetes()

    v1 = client.CoreV1Api()

    pod = v1.read_namespaced_pod(
        name=pod_name,
        namespace=namespace
    )

    containers = []

    if pod.status.container_statuses:

        for container in pod.status.container_statuses:

            state = {}

            if container.state:

                if container.state.waiting:

                    state = {
                        "state": "waiting",
                        "reason": container.state.waiting.reason,
                        "message": container.state.waiting.message
                    }

                elif container.state.running:

                    state = {
                        "state": "running",
                        "started_at": str(
                            container.state.running.started_at
                        )
                    }

                elif container.state.terminated:

                    state = {
                        "state": "terminated",
                        "reason": container.state.terminated.reason,
                        "exit_code": container.state.terminated.exit_code,
                        "signal": container.state.terminated.signal,
                        "message": container.state.terminated.message
                    }

            containers.append(
                {
                    "name": container.name,
                    "ready": container.ready,
                    "restart_count": container.restart_count,
                    "state": state
                }
            )

    return {
        "pod_name": pod.metadata.name,
        "namespace": pod.metadata.namespace,
        "phase": pod.status.phase,
        "node": pod.spec.node_name,
        "pod_ip": pod.status.pod_ip,
        "containers": containers
    }


def get_pod_logs(
    namespace: str,
    pod_name: str,
    container: str = None
):
    """
    Get current logs from a Kubernetes container.
    """

    load_kubernetes()

    v1 = client.CoreV1Api()

    logs = v1.read_namespaced_pod_log(
        name=pod_name,
        namespace=namespace,
        container=container,
        tail_lines=100
    )

    return {
        "pod_name": pod_name,
        "namespace": namespace,
        "log_type": "current",
        "logs": logs
    }


def get_pod_previous_logs(
    namespace: str,
    pod_name: str,
    container: str = None
):
    """
    Get logs from the previous terminated container.

    Very useful for CrashLoopBackOff.
    """

    load_kubernetes()

    v1 = client.CoreV1Api()

    logs = v1.read_namespaced_pod_log(
        name=pod_name,
        namespace=namespace,
        container=container,
        previous=True,
        tail_lines=100
    )

    return {
        "pod_name": pod_name,
        "namespace": namespace,
        "log_type": "previous",
        "logs": logs
    }


def get_pod_events(
    namespace: str,
    pod_name: str
):
    """
    Get Kubernetes events associated with a pod.
    """

    load_kubernetes()

    v1 = client.CoreV1Api()

    events = v1.list_namespaced_event(
        namespace=namespace,
        field_selector=(
            f"involvedObject.name={pod_name}"
        )
    )

    result = []

    for event in events.items:

        result.append(
            {
                "type": event.type,
                "reason": event.reason,
                "message": event.message,
                "count": event.count,
                "first_timestamp": str(
                    event.first_timestamp
                ),
                "last_timestamp": str(
                    event.last_timestamp
                )
            }
        )

    return {
        "pod_name": pod_name,
        "namespace": namespace,
        "events": result
    }

11. Important Troubleshooting Lesson

During implementation, one common mistake is accidentally creating:

tools/agent_tools.py

as an empty file.

For example:

agent_tools.py    0 bytes

Then this command:

python -c "from tools.agent_tools import get_pod_status"

will produce:

ImportError:
cannot import name 'get_pod_status'

The reason is simple:

sre_agent.py
     |
     | imports
     v
agent_tools.py
     |
     X
     |
     No functions

After adding the code above, verify:

Get-Item .\tools\agent_tools.py

The file should have a size greater than zero.

12. Test the Kubernetes Tool Import

Run:

python -c "from tools.agent_tools import get_pod_status; print('TOOLS IMPORT OK')"

Expected:

TOOLS IMPORT OK

Test all four:

python -c "from tools.agent_tools import get_pod_status, get_pod_logs, get_pod_previous_logs, get_pod_events; print('ALL 4 TOOLS IMPORT OK')"

Expected:

ALL 4 TOOLS IMPORT OK

13. Incident Model

Create:

models/incident.py

Code:

from typing import List
from pydantic import BaseModel, Field


class Evidence(BaseModel):

    fact: str

    source: str


class IncidentReport(BaseModel):

    incident: str

    symptom: str

    evidence: List[Evidence] = Field(
        default_factory=list
    )

    hypothesis: str

    validation: List[str] = Field(
        default_factory=list
    )

    root_cause: str

    confidence: str

    next_investigation: List[str] = Field(
        default_factory=list
    )

This allows us to represent the investigation in a structured format.

The intended report structure is:

SYMPTOM
EVIDENCE
HYPOTHESIS
VALIDATION
ROOT CAUSE
CONFIDENCE
NEXT INVESTIGATION

14. Building the AI SRE Agent

Now create:

agents/sre_agent.py

The agent will:

  1. Accept namespace and pod name.

  2. Ask the LLM to investigate.

  3. Allow the LLM to select tools.

  4. Execute the selected Kubernetes tool.

  5. Send the result back to the LLM.

  6. Allow the LLM to select another tool if required.

  7. Continue until sufficient evidence exists.

  8. Generate the final SRE report.

15. Complete agents/sre_agent.py

Use this complete code:

import json
import os

from dotenv import load_dotenv
from openai import OpenAI

from tools.agent_tools import (
    get_pod_status,
    get_pod_logs,
    get_pod_previous_logs,
    get_pod_events,
)


# ============================================================
# ENVIRONMENT
# ============================================================

load_dotenv()

api_key = os.getenv("OPENAI_API_KEY")

if not api_key:

    raise RuntimeError(
        "OPENAI_API_KEY is missing. "
        "Create a .env file in the project root."
    )


# ============================================================
# OPENAI CLIENT
# ============================================================

client = OpenAI(
    api_key=api_key
)


# ============================================================
# SYSTEM PROMPT
# ============================================================

SYSTEM_PROMPT = """
You are a senior Kubernetes SRE investigation agent.

Your job is to investigate Kubernetes incidents
using available READ-ONLY tools.

Investigation rules:

1. Never invent cluster state.

2. Always collect evidence before making conclusions.

3. Clearly distinguish FACT from INFERENCE.

4. Clearly distinguish HYPOTHESIS from CONFIRMED ROOT CAUSE.

5. Never claim a root cause without supporting evidence.

6. Use additional tools when evidence is insufficient.

7. Do not perform remediation.

8. Prefer the smallest number of tools necessary.

9. Correlate pod status, logs and Kubernetes events.

10. If the root cause cannot be proven, explicitly say:

    Root cause not confirmed.


Important:

Exit code 137 does NOT automatically prove OOMKilled.

Database connection refused does NOT automatically prove
that the database is down.

ImagePullBackOff should be investigated using Kubernetes
events and pod status.

Pending pods should be investigated using pod status and
Kubernetes scheduling events.


The final answer must contain:

SYMPTOM
EVIDENCE
HYPOTHESIS
VALIDATION
ROOT CAUSE
CONFIDENCE
NEXT INVESTIGATION
"""


# ============================================================
# TOOL DEFINITIONS
# ============================================================

TOOLS = [

    {
        "type": "function",

        "name": "get_pod_status",

        "description": (
            "Get Kubernetes pod status, container states, "
            "restart counts, exit codes and node information."
        ),

        "parameters": {

            "type": "object",

            "properties": {

                "namespace": {
                    "type": "string"
                },

                "pod_name": {
                    "type": "string"
                }
            },

            "required": [
                "namespace",
                "pod_name"
            ]
        }
    },


    {
        "type": "function",

        "name": "get_pod_logs",

        "description": (
            "Get current logs from a Kubernetes pod."
        ),

        "parameters": {

            "type": "object",

            "properties": {

                "namespace": {
                    "type": "string"
                },

                "pod_name": {
                    "type": "string"
                }
            },

            "required": [
                "namespace",
                "pod_name"
            ]
        }
    },


    {
        "type": "function",

        "name": "get_pod_previous_logs",

        "description": (
            "Get logs from the previous terminated "
            "container instance. Useful for CrashLoopBackOff."
        ),

        "parameters": {

            "type": "object",

            "properties": {

                "namespace": {
                    "type": "string"
                },

                "pod_name": {
                    "type": "string"
                }
            },

            "required": [
                "namespace",
                "pod_name"
            ]
        }
    },


    {
        "type": "function",

        "name": "get_pod_events",

        "description": (
            "Get Kubernetes events associated with a pod. "
            "Useful for scheduling, image pull, startup and "
            "restart failures."
        ),

        "parameters": {

            "type": "object",

            "properties": {

                "namespace": {
                    "type": "string"
                },

                "pod_name": {
                    "type": "string"
                }
            },

            "required": [
                "namespace",
                "pod_name"
            ]
        }
    }
]


# ============================================================
# TOOL EXECUTION
# ============================================================

def execute_tool(name, arguments):

    print()
    print("=" * 70)
    print("TOOL CALL")
    print("=" * 70)

    print("Tool:", name)

    print(
        "Arguments:",
        json.dumps(
            arguments,
            indent=2
        )
    )

    try:

        if name == "get_pod_status":

            return get_pod_status(
                arguments["namespace"],
                arguments["pod_name"]
            )


        elif name == "get_pod_logs":

            return get_pod_logs(
                arguments["namespace"],
                arguments["pod_name"]
            )


        elif name == "get_pod_previous_logs":

            return get_pod_previous_logs(
                arguments["namespace"],
                arguments["pod_name"]
            )


        elif name == "get_pod_events":

            return get_pod_events(
                arguments["namespace"],
                arguments["pod_name"]
            )


        else:

            return {
                "error": f"Unknown tool: {name}"
            }


    except Exception as exc:

        return {
            "error": str(exc)
        }


# ============================================================
# INVESTIGATION ENGINE
# ============================================================

def investigate(
    namespace: str,
    pod_name: str
):

    user_prompt = f"""
Investigate this Kubernetes incident.

Namespace:
{namespace}

Pod:
{pod_name}


Determine why this pod is failing.

Use evidence.

Do not make assumptions.

Do not perform remediation.

Use additional tools if the available evidence
is insufficient.

At the end produce a structured SRE incident report.
"""


    messages = [

        {
            "role": "system",

            "content": SYSTEM_PROMPT
        },

        {
            "role": "user",

            "content": user_prompt
        }
    ]


    max_iterations = 10


    for iteration in range(max_iterations):

        print()
        print("=" * 70)

        print(
            f"AGENT ITERATION {iteration + 1}"
        )

        print("=" * 70)


        response = client.responses.create(

            model="gpt-5",

            input=messages,

            tools=TOOLS
        )


        tool_outputs = []


        for item in response.output:

            if item.type == "function_call":

                name = item.name


                arguments = json.loads(
                    item.arguments
                )


                result = execute_tool(
                    name,
                    arguments
                )


                tool_outputs.append(

                    {
                        "type": "function_call_output",

                        "call_id": item.call_id,

                        "output": json.dumps(
                            result,
                            default=str
                        )
                    }

                )


        # ----------------------------------------------------
        # No more tools required
        # ----------------------------------------------------

        if not tool_outputs:

            return response.output_text


        # ----------------------------------------------------
        # Send assistant tool request back to conversation
        # ----------------------------------------------------

        messages.extend(
            response.output
        )


        # ----------------------------------------------------
        # Send tool results back to LLM
        # ----------------------------------------------------

        messages.extend(
            tool_outputs
        )


    return (
        "Investigation stopped because maximum "
        "agent iterations were reached."
    )


# ============================================================
# MAIN
# ============================================================

def main():

    namespace = input(
        "Enter namespace [default]: "
    ).strip()


    if not namespace:

        namespace = "default"


    pod_name = input(
        "Enter pod name: "
    ).strip()


    if not pod_name:

        print(
            "Pod name is required."
        )

        return


    print()

    print("=" * 70)

    print(
        "KUBERNETES SRE INVESTIGATION"
    )

    print("=" * 70)


    print(
        "Namespace:",
        namespace
    )

    print(
        "Pod:",
        pod_name
    )


    result = investigate(
        namespace,
        pod_name
    )


    print()

    print("=" * 70)

    print(
        "FINAL SRE REPORT"
    )

    print("=" * 70)


    print(result)


if __name__ == "__main__":

    main()

16. Why We Use python -m

Our project uses:

k8s-day5-sre-agent/
│
├── agents/
│   └── sre_agent.py
│
└── tools/
    └── agent_tools.py

Therefore, run:

python -m agents.sre_agent

Do not normally run:

python .\agents\sre_agent.py

The module approach allows Python to correctly find:

from tools.agent_tools import ...

17. Verify the Agent Import

Run:

python -c "import agents.sre_agent; print('AGENT IMPORT OK')"

Expected:

AGENT IMPORT OK

If you get:

ModuleNotFoundError: No module named 'tools'

you are probably executing the file directly instead of using:

python -m agents.sre_agent

18. Verify the API Key

Run:

python -c "from dotenv import load_dotenv; import os; load_dotenv(); print('API KEY STATUS:', 'SET' if os.getenv('OPENAI_API_KEY') else 'NOT SET')"

Expected:

API KEY STATUS: SET

If you get:

API KEY STATUS: NOT SET

check:

k8s-day5-sre-agent/
│
└── .env

and make sure it contains:

OPENAI_API_KEY=YOUR_API_KEY

Do not share your actual API key publicly.

19. Verify Kubernetes

Before running the AI agent:

minikube status

Then:

kubectl get nodes

Expected:

NAME       STATUS   ROLES           AGE
minikube   Ready    control-plane   ...

Then:

kubectl get pods -A

20. Create a CrashLoopBackOff Incident

We need a controlled incident for testing.

Run:

kubectl create deployment app-crash `
  --image=busybox `
  -- /bin/sh -c "echo Application started; echo ERROR database connection refused; exit 1"

Check:

kubectl get pods

After a while:

NAME                          READY   STATUS             RESTARTS
app-crash-xxxxxxxxxx          0/1     CrashLoopBackOff   3

Copy the actual pod name.

21. Manually Investigate the Incident

Before allowing AI to investigate, it is useful for an SRE to understand the raw evidence.

Run:

kubectl get pod YOUR_POD_NAME -o wide

Then:

kubectl describe pod YOUR_POD_NAME

Current logs:

kubectl logs YOUR_POD_NAME

Previous logs:

kubectl logs YOUR_POD_NAME --previous

Events:

kubectl get events --sort-by=.lastTimestamp

You may see:

Application started
ERROR database connection refused

and events such as:

BackOff

22. Understand the Evidence

The correct reasoning is:

FACT
Application logged:
database connection refused

Then:

INFERENCE
The application could not establish a database connection.

Then:

HYPOTHESIS
Database connectivity is a possible cause.

But:

ROOT CAUSE
Database is down

is not yet proven.

This is the difference between an AI chatbot that generates plausible answers and an SRE investigation agent that works from evidence.

23. Run the AI Agent

Run:

python -m agents.sre_agent

You should see:

Enter namespace [default]:

Enter:

default

Then:

Enter pod name:

Enter your actual pod:

app-crash-xxxxxxxxxx

24. What Happens Inside the Agent?

The first iteration may look conceptually like:

AGENT ITERATION 1
        |
        v
LLM asks:
"What is the pod state?"
        |
        v
get_pod_status()

The tool returns:

phase = Running
restart_count = 3
container state = terminated
exit_code = 1

The LLM then reasons:

The container is restarting.
I need logs.

It calls:

get_pod_previous_logs()

The result:

Application started
ERROR database connection refused

The LLM may then call:

get_pod_events()

The result:

BackOff

Now it has multiple pieces of evidence.

25. Dynamic Investigation

The important point is that the Python program does not explicitly contain:

if status == "CrashLoopBackOff":
    get_logs()

if logs contain "database":
    get_events()

Instead:

LLM
 |
 +--> get_pod_status()
 |
 +--> inspect result
 |
 +--> decide next action
 |
 +--> get_pod_previous_logs()
 |
 +--> inspect result
 |
 +--> decide next action
 |
 +--> get_pod_events()
 |
 +--> inspect result
 |
 +--> generate report

This is the beginning of agentic behavior.

26. Expected SRE Report

The exact wording generated by the LLM may vary, but the report should resemble:

SYMPTOM
-------
Pod app-crash-xxxx is repeatedly restarting
and is currently in CrashLoopBackOff.


EVIDENCE
--------
1. Container restart count is greater than zero.

2. Container terminated with exit code 1.

3. Previous container logs contain:
   "ERROR database connection refused"

4. Kubernetes events show BackOff.


HYPOTHESIS
----------
The application is failing because it cannot
establish database connectivity.


VALIDATION
----------
The previous container logs confirm that the
application reported a database connection failure.

However, database availability itself has not
been independently verified.


ROOT CAUSE
----------
Root cause not confirmed.

The evidence confirms an application-level
database connection failure but does not prove
that the database server itself is down.


CONFIDENCE
----------
Medium


NEXT INVESTIGATION
------------------
1. Check database Service.
2. Check Service endpoints.
3. Check DNS resolution.
4. Check NetworkPolicy.
5. Check database availability.
6. Validate application database configuration.

This is consistent with the Day-5 target report structure.

27. Incident 2 — ImagePullBackOff

Now create a bad image:

kubectl create deployment app-image `
  --image=nginx:this-image-does-not-exist

Check:

kubectl get pods

You should see something similar to:

app-image-xxxxxxxxxx    0/1    ImagePullBackOff

Now run:

python -m agents.sre_agent

Enter:

default

and the actual pod name.

The expected investigation pattern is:

Pod status
    |
    v
ImagePullBackOff
    |
    v
Kubernetes events
    |
    v
Failed to pull image
    |
    v
Diagnosis

The agent should not treat this like a CrashLoopBackOff investigation.

That is the purpose of dynamic planning.

28. Incident 3 — Pending Pod

A normal Minikube cluster may successfully schedule:

nginx

Therefore, simply creating an nginx deployment does not guarantee Pending.

Instead, deliberately create an impossible node selector.

Create:

scenarios/app-pending.yaml

Use:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: app-pending

spec:
  replicas: 1

  selector:
    matchLabels:
      app: app-pending

  template:

    metadata:
      labels:
        app: app-pending

    spec:

      nodeSelector:
        sre-test-node: "does-not-exist"

      containers:

        - name: nginx
          image: nginx

Apply:

kubectl apply -f scenarios/app-pending.yaml

Check:

kubectl get pods

Expected:

app-pending-xxxxxxxxxx    0/1    Pending

29. Investigate Pending

Run:

kubectl describe pod YOUR_PENDING_POD

Near the bottom you should see scheduling events similar to:

Events:

Warning  FailedScheduling

0/1 nodes are available:
1 node(s) didn't match Pod's node affinity/selector.

The agent should follow:

Pending
   |
   v
Pod Status
   |
   v
FailedScheduling
   |
   v
Kubernetes Events
   |
   v
Node selector / scheduling constraint

30. Test Kubernetes Tools Independently

Create:

tests/test_tools.py

Use:

from tools.agent_tools import (
    get_pod_status,
    get_pod_logs,
    get_pod_previous_logs,
    get_pod_events
)


POD_NAME = "YOUR_POD_NAME"

NAMESPACE = "default"


print("=" * 70)

print("TEST 1: POD STATUS")

print("=" * 70)

result = get_pod_status(
    NAMESPACE,
    POD_NAME
)

print(result)


print()

print("=" * 70)

print("TEST 2: CURRENT LOGS")

print("=" * 70)

result = get_pod_logs(
    NAMESPACE,
    POD_NAME
)

print(result)


print()

print("=" * 70)

print("TEST 3: PREVIOUS LOGS")

print("=" * 70)

try:

    result = get_pod_previous_logs(
        NAMESPACE,
        POD_NAME
    )

    print(result)

except Exception as exc:

    print("Previous logs unavailable:")

    print(exc)


print()

print("=" * 70)

print("TEST 4: EVENTS")

print("=" * 70)

result = get_pod_events(
    NAMESPACE,
    POD_NAME
)

print(result)

Run:

python tests/test_tools.py

This lets us separate:

Kubernetes problem

from:

AI agent problem

That is an important engineering practice.

31. Cleanup Test Incidents

Create:

scenarios/cleanup.ps1

Use:

Write-Host "Cleaning SRE test incidents..."

kubectl delete deployment app-crash --ignore-not-found

kubectl delete deployment app-image --ignore-not-found

kubectl delete deployment app-pending --ignore-not-found

Write-Host ""

Write-Host "Cleanup complete."

kubectl get pods

Run:

.\scenarios\cleanup.ps1

32. Create an Automated Incident Generator

Create:

scenarios/create_incidents.ps1

Code:

Write-Host "============================================"
Write-Host "Creating Kubernetes SRE Test Incidents"
Write-Host "============================================"


Write-Host ""

Write-Host "[1] Creating CrashLoopBackOff incident"

kubectl create deployment app-crash `
  --image=busybox `
  -- /bin/sh -c "echo Application started; echo ERROR database connection refused; exit 1"


Write-Host ""

Write-Host "[2] Creating ImagePullBackOff incident"

kubectl create deployment app-image `
  --image=nginx:this-image-does-not-exist"


Write-Host ""

Write-Host "[3] Creating Pending incident"

kubectl apply -f scenarios/app-pending.yaml


Write-Host ""

Write-Host "============================================"

Write-Host "Incidents created"

Write-Host "============================================"


kubectl get pods

Run:

.\scenarios\create_incidents.ps1

33. Complete End-to-End Test

Now the complete workflow becomes:

cd k8s-day5-sre-agent

.\.venv\Scripts\Activate.ps1

minikube status

kubectl get nodes

pip install -r requirements.txt

.\scenarios\create_incidents.ps1

kubectl get pods

python -m agents.sre_agent

Investigate the CrashLoopBackOff pod.

Then:

python -m agents.sre_agent

Investigate the ImagePullBackOff pod.

Then:

python -m agents.sre_agent

Investigate the Pending pod.

Finally:

.\scenarios\cleanup.ps1

34. Complete Architecture

At this point the project looks like:

                         USER
                           |
                           v
                 +-------------------+
                 |    SRE AGENT      |
                 |      LLM          |
                 +---------+---------+
                           |
                           |
                    Dynamic Planning
                           |
          +----------------+----------------+
          |                |                |
          v                v                v
  get_pod_status()   get_pod_logs()   get_pod_events()
          |
          |
          v
get_pod_previous_logs()
          |
          +----------------+
                           |
                           v
                     EVIDENCE
                           |
                           v
                     HYPOTHESIS
                           |
                           v
                     VALIDATION
                           |
                           v
                 ROOT CAUSE CHECK
                    /          \
                   /            \
                  NO            YES
                  |              |
                  v              v
           More investigation   RCA
                  |
                  v
             More tools

35. Traditional Script vs Agentic SRE

A traditional script may look like:

get status
   ↓
get logs
   ↓
get events
   ↓
print output

The agentic approach is:

get status
   ↓
reason
   ↓
decide what evidence is missing
   ↓
get relevant tool
   ↓
reason again
   ↓
validate hypothesis
   ↓
decide whether root cause is proven
   ↓
generate RCA

This is a significant architectural difference.

36. Example: Exit Code 137

One of the important rules in our system is:

Exit code 137

must not automatically become:

OOMKilled

Why?

Because:

137 = 128 + 9

which corresponds to:

SIGKILL

Possible explanations require additional evidence.

The investigation should therefore be:

Exit code 137
       |
       v
SIGKILL
       |
       v
Possible OOM
       |
       v
Collect evidence
       |
       +----> Container state
       |
       +----> Kubernetes events
       |
       +----> Node memory
       |
       +----> Container memory limit
       |
       v
Confirm / Reject OOM hypothesis

This evidence-first reasoning is specifically emphasized in the Day-5 material.

37. Example: Database Connection Refused

Consider:

ERROR database connection refused

The agent should reason:

FACT
Application received connection refused.

        ↓

INFERENCE
Application could not establish DB connection.

        ↓

HYPOTHESIS
Database connectivity problem.

        ↓

VALIDATION
Need more evidence.

        ↓

INVESTIGATE
Service
Endpoints
DNS
NetworkPolicy
Database
Credentials
Configuration

        ↓

ROOT CAUSE
Only confirmed when evidence supports it.

This is much safer than letting an LLM invent a root cause.

38. Read-Only Safety Model

Our Day-5 agent is intentionally read-only.

The tools can:

READ pod status
READ logs
READ previous logs
READ events

They cannot:

DELETE pod
DELETE deployment
RESTART pod
SCALE deployment
PATCH deployment
CHANGE node
CHANGE network policy
CHANGE storage

This is an important production design principle.

The investigation agent should first establish:

What happened?

before eventually being allowed to answer:

What should we do?

And remediation should be a separate controlled capability.

39. Why Previous Logs Matter

For a normal running container:

kubectl logs POD

may be enough.

But for:

CrashLoopBackOff

the currently running container may have little or no useful information.

The previous container may contain:

startup logs
exception
stack trace
configuration failure
connection failure
termination reason

Therefore:

CrashLoopBackOff
      |
      v
Previous Logs
      |
      v
Failure Evidence

is an important SRE troubleshooting pattern.

40. Why Kubernetes Events Matter

Application logs tell us what the application experienced.

Kubernetes events tell us what Kubernetes experienced.

For example:

Application logs
----------------
database connection refused

versus:

Kubernetes events
-----------------
FailedScheduling
Failed to pull image
BackOff
FailedMount
Unhealthy

An SRE agent should correlate both.

Therefore:

Application Evidence
          +
Kubernetes Evidence
          =
Better Incident Diagnosis

41. Current Project Capabilities

After completing Day 5, our agent can:

✓ Connect to Kubernetes
✓ Inspect pod status
✓ Inspect container state
✓ Inspect restart counts
✓ Inspect exit codes
✓ Read current logs
✓ Read previous logs
✓ Read Kubernetes events
✓ Dynamically select tools
✓ Collect evidence
✓ Generate hypotheses
✓ Validate hypotheses
✓ Avoid unsupported root-cause claims
✓ Generate structured SRE reports

The Day-5 goal is to establish this structured evidence-driven investigation loop before moving toward broader observability and multi-agent capabilities.

42. What We Have Learned

The most important lesson from this project is:

An AI SRE agent should not be a chatbot that guesses the root cause.

It should behave more like an experienced SRE.

An experienced SRE thinks:

What do I know?
        ↓
What don't I know?
        ↓
What evidence would reduce uncertainty?
        ↓
Which command should I run?
        ↓
What did that command tell me?
        ↓
Does my hypothesis still hold?
        ↓
What should I investigate next?

Our AI agent is beginning to follow the same process.

43.  Mental Model

Remember this:

OBSERVE
   ↓
REASON
   ↓
HYPOTHESIZE
   ↓
COLLECT EVIDENCE
   ↓
VALIDATE
   ↓
REASON AGAIN
   ↓
CONFIRM ROOT CAUSE
   ↓
REPORT

This is the core mental model for building an Agentic AI SRE system.

44. Final Project Tree

The completed project should look like:

k8s-day5-sre-agent/
│
├── .venv/
│
├── agents/
│   ├── __init__.py
│   └── sre_agent.py
│
├── tools/
│   ├── __init__.py
│   └── agent_tools.py
│
├── models/
│   ├── __init__.py
│   └── incident.py
│
├── scenarios/
│   ├── app-pending.yaml
│   ├── create_incidents.ps1
│   └── cleanup.ps1
│
├── tests/
│   ├── __init__.py
│   └── test_tools.py
│
├── .env
├── .gitignore
├── requirements.txt
└── README.md

45. Final Execution Checklist

Use this checklist whenever you start the project:

# Activate environment
.\.venv\Scripts\Activate.ps1


# Check Python
python --version


# Check Kubernetes
minikube status


# Check node
kubectl get nodes


# Check API key
python -c "from dotenv import load_dotenv; import os; load_dotenv(); print('API KEY:', 'SET' if os.getenv('OPENAI_API_KEY') else 'NOT SET')"


# Check Kubernetes tools
python -c "from tools.agent_tools import get_pod_status, get_pod_logs, get_pod_previous_logs, get_pod_events; print('ALL 4 TOOLS IMPORT OK')"


# Check agent
python -c "import agents.sre_agent; print('AGENT IMPORT OK')"


# Run agent
python -m agents.sre_agent


on

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)