Let's Break and Fix Our Kubernetes Investigation Agent Before Production

This chapter focuses on the critical phase of evaluating an AI-driven Kubernetes troubleshooting agent before deploying it to a production environment. We will explore how an agent that successfully resolves one class of errors can fail catastrophically and hallucinate on another due to "tool blindspots." By implementing a systematic, five-step evaluation workflow using the Google Agent Development Kit (ADK) and agent CLI, we will synthesize test datasets, grade agent performance, diagnose architectural gaps, and iteratively refine our agent to achieve production-grade reliability.

1. The Imperative of AI Agent Evaluation

In software engineering, passing a single test case does not guarantee production readiness. The same principle applies to AI agents. While our Kubernetes investigation agent successfully diagnosed an ImagePullBackOff error in previous runs, relying on a few manual tests creates a false sense of security.

Without a systematic evaluation framework, an agent is highly susceptible to hallucination when confronted with unfamiliar incidents. If the agent's tools cannot retrieve the true root cause of a failure, the LLM will attempt to construct a plausible-sounding but completely incorrect explanation.

To prevent these silent failures in production, we must establish an automated evaluation pipeline that tests the agent against a diverse suite of edge cases, measures its accuracy, and tracks performance regressions over time.

2. Anatomy of a Failure: The Out-Of-Memory (OOM) Blindspot

To demonstrate how easily an unevaluated agent can fail, we introduce a common production incident: an Out-Of-Memory (OOM) Kill on a database container.

The Faulty Deployment Configuration

We deploy a MySQL database for a payments microservice. The official MySQL initialization process requires at least 400 MB of memory. However, we intentionally restrict the container's memory limit to 128 MB in the deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-db
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments-db
  template:
    metadata:
      labels:
        app: payments-db
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "supersecretpassword"
        resources:
          limits:
            memory: "128Mi" # Intentionally low limit (Requires >= 400Mi)
          requests:
            memory: "64Mi"

The Naive Agent's Investigation Report

When we run the agent to investigate the failing payments-db pod, it outputs a highly confident but incorrect analysis:

• Identified Issue: Container startup failure and crash loop backoff.
• Confidence Score: 95%
• Agent's Summary & Root Cause:

  "The MySQL container process starts initialization of the DB files and InnoDB storage engine. Possible causes: missing mandatory environment variables, unhandled system signal, or container runtime file system write/permission restrictions."

• The Reality: The container was terminated by the Linux kernel's Out-Of-Memory killer (OOMKilled) because it exceeded its 128 MB memory limit during initialization. The agent completely missed this root cause.

3. The Root Cause of Agent Hallucination: Tool Limitations

Why did a highly capable LLM with a 95% confidence rating fail to identify an OOM kill? The answer lies in the information boundary of its tools.

An AI agent is only as good as the data its tools can retrieve. In our initial design, the agent was equipped with three tools:

1. get_pod_logs
2. get_pod_events
3. get_deployment_history

Let us examine what these tools actually see when a container is OOM-killed:

Because none of the active tools queried the pod's container status fields (where Kubernetes records the OOMKilled termination reason), the LLM was forced to guess. It analyzed the truncated logs, saw that initialization stopped, and hallucinated that a permission issue or missing environment variable was to blame.

Note: This highlights a fundamental rule of AI Agent Architecture: If your tools do not expose the ground-truth data, no amount of prompt engineering will prevent the agent from hallucinating.

4. The 5-Step End-to-End Evaluation Framework

To systematically identify and resolve these blindspots, we implement a structured evaluation workflow using the agent CLI (part of the Google Agent Development Kit).

5. Step-by-Step Execution: Synthesizing the Evaluation Dataset

Instead of manually writing test cases, we leverage our coding agent (configured with the agent CLI skills) to synthesize a comprehensive dataset using natural language.

The Synthesis Prompt

We instruct our coding agent to generate 10 distinct Kubernetes incident scenarios covering a wide range of production failures:

Synthesize 10 Kubernetes incident test cases for our evaluation data set related to:
- OOM kill
- CPU throttling
- Container crash / CrashLoopBackOff
- Node memory pressure
- Node not ready
- Readiness probe failures
- Incorrect resource requests/limits
- Service has no endpoints
- Service routing to unhealthy pods

The coding agent translates this prompt into the appropriate CLI command:

agent CLI eval dataset synthesize -n 10

Synthesized Dataset Structure

The output is saved to a JSON file (e.g., synthesized_10_incidents.json). Each test case contains a user scenario (input) and the expected diagnostic output (ground truth):

{
  "cases": [
    {
      "id": "case_001",
      "input": "I'm an on-call data scientist. My training job pod just died unexpectedly. It called data-processor in the backend namespace. Can you figure out what happened to it?",
      "expected_response": {
        "incident": "Pod termination investigation: training-job abrupt termination",
        "root_cause": "The training-job pod was terminated due to an Out-Of-Memory (OOM) event because the data-processor container exceeded its allocated memory limits.",
        "remediation": "Increase the memory limit in the pod specification or optimize the data processing script to reduce memory consumption."
      }
    },
    {
      "id": "case_007",
      "input": "The payment gateway just went down. The pod payment-gateway-v2 in the finance namespace is in CrashLoopBackOff according to ArgoCD. Please investigate.",
      "expected_response": {
        "incident": "Pod CrashLoopBackOff diagnostic",
        "root_cause": "The pod failed to schedule because the cluster has insufficient CPU resources to satisfy the resource requests of the new deployment.",
        "remediation": "Scale up the cluster nodes or adjust the CPU requests of the payment-gateway-v2 deployment downward."
      }
    }
  ]
}

6. Generating Traces and Establishing the Baseline Score

With our dataset synthesized, we run the agent against these 10 cases to capture its execution traces and grade its performance.

Step 1: Generate Traces

We prompt the coding agent to run the evaluation:

Execute the agent on this test data set and generate the traces.

This executes:

agent CLI eval generate --dataset synthesized_10_incidents.json --output_dir artifact/traces/

Step 2: Grade the Traces

Next, we score the generated traces against our evaluation metrics:

Grade the recorded traces and show me the test scorecard.

This runs the grading engine using the default response quality metric defined in test_eval/config.yaml:

agent CLI eval grade --traces_dir artifact/traces/ --config test_eval/config.yaml

Analyzing the Baseline Scorecard

The evaluation engine generates an interactive HTML report. Our baseline agent achieved the following performance metrics:

• Mean Score ($\mu$): $4.7 / 5.0$
• Standard Deviation ($\sigma$): $0.9487$

While a mean score of 4.7 appears high, the standard deviation indicates significant inconsistency. Looking closely at the individual test cases, we find a major failure in Case 7 (Pod Scheduling Failure):

• Case 7 Score: $2.0 / 5.0$
• The Agent's Hallucination: The agent claimed that the pod could not be scheduled because "three nodes have untolerated control-plane taints and three nodes have insufficient CPU/memory."
• The Reality: The test environment is a local, single-node kind cluster. There are no "three nodes" in existence.

Actual Cluster State: [Single Node]
Agent's Claim: "3 Nodes have taints, 3 Nodes have insufficient CPU" (Hallucination)

The agent hallucinated this multi-node topology because it lacked any tool to query the actual node configuration of the cluster. It assumed a standard multi-node production topology to make its explanation sound convincing.

7. Iterative Improvement: Adding the Node Investigation Tool

To resolve the node-related hallucinations, we must expand our agent's capability by adding a tool that retrieves node information.

Adding the Tool via the Coding Agent

We prompt our development agent to implement the new tool:

Add a tool called get_nodes_info to the Kubernetes investigation agent so that it can analyze node-related issues.

The coding agent uses the Google Agent Development Kit to modify our codebase. It updates tools.py and registers the new tool in agent.py.

Code Evolution: tools.py

The coding agent appends the implementation for get_nodes_info to query the Kubernetes API for cluster node statuses, taints, and capacities:

# tools.py
import subprocess
import json
from typing import Dict, Any

def get_nodes_info() -> Dict[str, Any]:
    """
    Retrieves detailed status, capacity, allocatable resources, 
    and taints for all nodes in the Kubernetes cluster.
    """
    try:
        # Execute kubectl get nodes in JSON format
        result = subprocess.run(
            ["kubectl", "get", "nodes", "-o", "json"],
            capture_output=True,
            text=True,
            check=True
        )
        nodes_data = json.loads(result.stdout)
        
        node_summary = []
        for node in nodes_data.get("items", []):
            metadata = node.get("metadata", {})
            status = node.get("status", {})
            spec = node.get("spec", {})
            
            node_summary.append({
                "name": metadata.get("name"),
                "status": [cond.get("type") for cond in status.get("conditions", []) if cond.get("status") == "True"],
                "capacity": status.get("capacity"),
                "allocatable": status.get("allocatable"),
                "taints": spec.get("taints", [])
            })
            
        return {"status": "success", "nodes": node_summary}
        
    except subprocess.CalledProcessError as e:
        return {"status": "error", "message": f"Failed to retrieve node info: {e.stderr}"}
    except Exception as e:
        return {"status": "error", "message": str(e)}

Registering the Tool in agent.py

The tool is then imported and registered within the agent's execution context:

# agent.py
from tools import get_pod_logs, get_pod_events, get_deployment_history, get_nodes_info

class KubernetesInvestigationAgent:
    def __init__(self):
        self.name = "K8s-SRE-Agent"
        # Registering the new tool alongside existing tools
        self.tools = {
            "get_pod_logs": get_pod_logs,
            "get_pod_events": get_pod_events,
            "get_deployment_history": get_deployment_history,
            "get_nodes_info": get_nodes_info # Added node analysis capability
        }
        
    def get_system_prompt(self) -> str:
        return """
        You are a senior Kubernetes Site Reliability Engineer (SRE) and forensic investigation specialist.
        Use the provided tools to diagnose issues with pods, deployments, and nodes.
        Always verify node capacities and taints using 'get_nodes_info' if a pod cannot be scheduled.
        """

8. Regression Testing and Final Evaluation Comparison

With the get_nodes_info tool integrated, we rerun our evaluation pipeline to verify the fix and ensure no existing capabilities were broken.

Step 1: Generate New Traces

We execute the agent against the same 10 test cases:

Execute the agent on our test data set and generate the traces.

Step 2: Grade and Compare Results

We grade the new traces and compare them against our baseline run:

Compare our latest trace results against the baseline using the eval compare tool.

This triggers:

agent CLI eval compare --baseline artifact/traces/baseline_report.json --candidate artifact/traces/new_report.json

Final Evaluation Metrics Comparison

Our iterative improvement yielded a perfect evaluation run:

By adding get_nodes_info, the agent successfully queried the single-node cluster, read its actual allocatable memory and CPU, and correctly diagnosed the scheduling failure without hallucinating non-existent nodes. The standard deviation dropped to $0$, proving that the agent's diagnostic behavior is now highly consistent and reliable across all 10 test scenarios.

Summary and Key Takeaways

• Evaluation is Mandatory: Successful execution of a single test case does not mean an agent is production-ready. Systematic evaluation is required to expose hidden hallucinations.
• The Information Boundary Rule: LLM hallucinations are often caused by tool limitations. If an agent's tools cannot access the ground-truth data (e.g., OOM status or node capacities), the agent will hallucinate plausible-sounding errors.
• Five-Step Evaluation Workflow: A robust agent lifecycle consists of synthesizing datasets, generating traces, configuring evaluation metrics, grading, and running regression comparisons.
• Leveraging Coding Agents for Testing: Using an Agent Development Kit (ADK) allows developers to automate complex CLI evaluation tasks using simple natural language prompts.
• Iterative Tool Expansion: When evaluations expose a diagnostic blindspot, the correct fix is to expand the agent's toolset (e.g., adding get_nodes_info to resolve node-scheduling hallucinations) rather than relying solely on prompt engineering.
• Quantifiable Progress: Tracking the mean score ($\mu$) and standard deviation ($\sigma$) across evaluation runs provides mathematical proof of an agent's stability and readiness for production deployment.