Day-4 | AI Agent Zero to Hero | Evaluation Before Deployment

Evaluating an autonomous AI agent before production deployment is critical to guarantee reliability, eliminate subtle hallucinations, and verify debugging logic across diverse failure modes. While an agent might resolve isolated test incidents correctly, shifts in runtime context, missing telemetry, or tool limitations can cause it to produce high-confidence hallucinations that disrupt production workflows.

1. Why Do We Need Agent Evaluation? (The Illusion of Success)

In agent development, successfully resolving a single bug creates a false sense of production readiness. An agent tested solely against basic scenarios (such as an obvious ImagePullBackOff error) can perform flawlessly: it queries the cluster, inspects events, identifies a misspelled image tag, and provides an actionable remediation plan.

However, real-world Kubernetes production incidents are significantly more complex:

• Pods crash due to out-of-memory errors (OOMKilled) without explicit error logging.
• Flaky networking causes intermittent Domain Name System (DNS) resolution drops.
• Insufficient Role-Based Access Control (RBAC) permissions silently block background secret reconciliation loops.

If an agent is exposed to an edge case where its defined tools fail to surface the root cause, an unconstrained Large Language Model (LLM) will often hallucinate a plausible-sounding root cause with extreme confidence rather than reporting missing telemetry.

The Real-World Risk

Deploying an unevaluated agent into an operations pipeline risks sending Site Reliability Engineers (SREs) down incorrect diagnostic paths, wasting critical minutes during high-severity outages. Systematic evaluation ensures:

1. Diagnostic Accuracy: The agent reaches the precise ground-truth root cause.
2. Trajectory Efficiency: The agent invokes appropriate tools in a logical order without redundant tool calls or extraneous reasoning loops.
3. Regression Safety: Adding a new tool or altering system prompts does not degrade performance across previously verified failure scenarios.

2. Anatomy of a Production Failure: The MySQL OOMKilled Blindspot

To demonstrate why single-test verifications fail, consider a payments service database deployment (payments-db) using an official MySQL image.

The Manifest Misconfiguration

The MySQL container is deployed with severely constrained resource limits ($128\text{ MB}$):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments-db
  template:
    metadata:
      labels:
        app: payments-db
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password"
        resources:
          limits:
            memory: "128Mi" # Official requirement during initialization is >= 400Mi

The Kubernetes Runtime Behavior

When applied to the cluster via kubectl apply -f deployment.yaml:

1. The container engine boots the MySQL process.
2. MySQL starts internal initialization routines (creating InnoDB data files, buffer pools, and system tables).
3. Memory consumption quickly spikes past $128\text{ MB}$.
4. The Linux kernel's Out-Of-Memory Killer immediately sends SIGKILL ($9$).
5. Kubernetes marks the pod as terminated with reason OOMKilled and restarts it into a CrashLoopBackOff state.

The Agent's Blind Failure Analysis

When executing the agent CLI:

agent-cli run "Identify issue with the pod payments-db-7d8b9c-xyz"

The agent runs with three standard diagnostic tools:

• get_pod_logs
• get_pod_events
• get_deployment_history

The Hallucination Output

Because none of the three tools exposed the kernel-level termination status (lastState.terminated.reason: OOMKilled), the agent synthesized a response based on incomplete information:

{
  "status": "Container startup failure and CrashLoopBackOff",
  "confidence": "95%",
  "summary": "Pod is failing to reach ready state and has entered a continuous restart loop immediately after container startup.",
  "root_cause": "The MySQL container process starts initialization of DB files and InnoDB storage engine. Possible causes: missing mandatory environment variables, unhandled system signal, or container runtime filesystem write/permission restrictions.",
  "remediation": "Verify filesystem permissions and check database initialization environment variables."
}

The agent was 95% confident, yet its diagnostic conclusion was completely wrong. It failed to mention memory starvation or OOMKilled, misguiding engineers away from the actual fix: increasing resources.limits.memory.

Note: LLMs do not fail by stopping; they fail by generating convincing rationalizations. If an agent's available tools do not provide sufficient signals, it will hallucinate plausible hypotheses unless systematically tested against an evaluation suite.

3. The 5-Stage Agent Evaluation Lifecycle

To prevent silent failures in production, autonomous agents require a structured evaluation pipeline comparable to software unit, integration, and regression testing.

4. Dataset Synthesis: Building Ground-Truth Benchmarks

An evaluation dataset consists of curated failure cases where the underlying root cause is known in advance.

Leveraging the Coding Agent & Skills Infrastructure

Instead of hand-crafting hundreds of lines of complex JSON mock data, developers can leverage an orchestrating coding agent configured with agent-cli skills:

1. The developer issues a natural-language prompt:
   CODE

   Copy

   Synthesize 5 Kubernetes incident test cases for our evaluation data set.
   

2. The coding agent parses its skills.md file, identifying the eval tool capability.
3. It inspects the agent code (agent.py, tools.py) to discover the problem domain and tool capabilities.
4. It executes the parameterized CLI command:
   BASH

   Copy

   agent-cli eval dataset synthesize -n 5 --instruction "Kubernetes incident investigation for OOMKilled, CrashLoopBackOff, ImagePullBackOff, RBAC 403 Forbidden, and DNS failure"
   

Dataset Structure

The synthesized dataset (data_sets/k8s_eval_dataset.json) defines structured incident scenarios:

[
  {
    "case_id": "case_001_oomkilled",
    "description": "Payment processor memory saturation leading to kernel termination",
    "input": {
      "pod_name": "payment-processor-6b4f7-9lmno",
      "namespace": "default",
      "mock_logs": "java.lang.OutOfMemoryError: Java heap space\n... dumping heap to disk\nTerminated",
      "mock_events": "Warning BackOff 3s (x4 over 45s) kubelet Back-off restarting failed container"
    },
    "expected_output": {
      "root_cause": "Container terminated due to OOMKilled",
      "confidence_threshold": 0.90,
      "remediation_contains": "Increase memory limits"
    }
  },
  {
    "case_id": "case_004_rbac_forbidden",
    "description": "Vault collector failing secret reconciliation due to RBAC denial",
    "input": {
      "pod_name": "vault-collector-98cd-11ab",
      "namespace": "default",
      "mock_logs": "Failed to synchronize API secrets from default namespace. HTTP 403 Forbidden: User system:serviceaccount:default:vault-sa cannot list resource 'secrets'",
      "mock_events": "Warning FailedMount 12s kubelet MountVolume.SetUp failed for volume"
    },
    "expected_output": {
      "root_cause": "RBAC authorization denied 403 Forbidden",
      "confidence_threshold": 0.85,
      "remediation_contains": "Update ClusterRole and RoleBinding permissions"
    }
  }
]

Essential Edge Cases for Kubernetes Diagnostics

To make an evaluation dataset robust for cluster troubleshooting, include at least 25 to 30 scenarios covering major infrastructure failure modes:

• Container memory starvation (OOMKilled)
• Missing environment variables / configuration failures (CrashLoopBackOff)
• Private registry authentication and invalid image tags (ImagePullBackOff / ErrImagePull)
• API Server authorization failures (RBAC 403 Forbidden)
• CoreDNS resolution timeouts and misconfigured upstream resolvers (DNSLookupFailure)
• Failed readiness/liveness probe thresholds (Unhealthy Probe)
• Node resource exhaustion and unschedulable pod states (FailedScheduling)

5. Execution Trajectory & Trace Analysis

Once the dataset is constructed, the agent must be evaluated against each scenario in a controlled environment to record its internal execution traces.

Generating Traces via Natural Language

Execute the agent on our test data set and generate the traces.

The underlying execution engine maps each synthesized scenario into the agent's runtime loop, storing execution data in eval/traces.json.

Trace Inspector Architecture

A trace record captures the complete context of an execution, enabling post-run analysis:

• Case Reference ID: Associates the trace with the specific test case in the dataset.
• System Prompt Applied: The active prompt injected into the LLM context window.
• Tool Invocations: The sequence of tools queried, arguments passed, and outputs received.
• LLM Reasoning Trajectory: The model's chain-of-thought tokens preceding tool calls.
• Final Payload Output: The formatted investigation report presented to the operator.

Reviewing raw traces helps pinpoint where an investigation went off track: whether the model selected the wrong tool, misinterpreted tool output, or hallucinated a conclusion despite receiving accurate data.

6. Automated Scoring, Metrics, and Evaluation Criteria

Manually reviewing JSON trace files becomes impractical as test suites grow to dozens of cases. Automated scoring pipelines evaluate output quality consistently at scale.

Grade the recorded traces and show me the test scorecard.

Evaluation Configurations (eval_config.yaml)

Automated grading is governed by configuration profiles that define assessment algorithms:

version: "1.0"
eval_suite:
  target_agent: "k8s-troubleshooter"
  metrics:
    - name: "response_quality"
      evaluator: "evaluators/response_quality.py"
      weight: 1.0
      criteria:
        semantic_similarity_threshold: 0.85
        require_exact_root_cause_match: true
    - name: "tool_quality"
      evaluator: "evaluators/tool_quality.py"
      weight: 0.5
      criteria:
        penalize_redundant_calls: true
        max_allowed_tool_calls: 5

Core Evaluation Metrics

1. Response Quality (response_quality.py):

   • Assesses the final diagnostic report against ground truth.
   • Evaluates whether the identified root cause matches the underlying failure mechanism (e.g., matching an OOMKilled classification rather than a generic startup crash).
   • Validates that remediation recommendations are technically sound.

2. Tool Selection Quality (tool_quality.py):

   • Evaluates tool usage efficiency throughout the investigation trajectory.
   • Penalizes redundant tool execution (e.g., calling get_pod_logs three consecutive times without changing input arguments).
   • Flags attempts to invoke non-existent tools or pass invalid parameters.

3. Formatting & Schema Adherence:

   • Ensures the agent returns a valid, parser-compatible JSON schema instead of unformatted conversational text.

7. Versioning, Regressions, and Baseline Comparison

Agent engineering is an iterative lifecycle. Updating a system prompt, adding diagnostic tools, or switching foundational models can fix one failure mode while unintentionally degrading others.

The Baseline Comparison Workflow

1. Establish a Baseline: Save a verified evaluation run against a standardized dataset as the reference standard (e.g., baseline_v1.0).
2. Implement Modifications: Update diagnostic tools (for example, adding a get_container_termination_reason tool to fix the OOMKilled blindspot) or refine system prompts.
3. Execute Comparative Runs:
   CODE

   Copy

   Compare our latest test results against the baseline using eval compare.
   

4. Analyze the Regression Delta:
   • Confirm that previously passing test cases continue to pass with the changes applied.
   • Verify that the target bug is resolved without introducing regressions elsewhere in the diagnostic pipeline.

Summary and Key Takeaways

• Single-Test Success is Deceptive: Resolving an isolated incident (like an invalid image tag) does not guarantee production readiness; complex edge cases frequently trigger confident hallucinations.
• Inadequate Tools Drive Hallucination: When an agent lacks the tools to observe critical system signals (such as container termination reasons in OOMKilled loops), LLMs often generate plausible-sounding but incorrect root causes.
• Dataset Synthesis Scales Ground-Truth Creation: Automated evaluation tools allow developers to rapidly generate representative incident suites covering memory starvation, permission denials, and networking failures.
• Traces Expose Diagnostic Trajectories: Trace records document prompts, tool usage sequences, and intermediate reasoning steps, simplifying post-incident debugging.
• Multi-Metric Scoring Measures Reliability: Automated evaluation evaluates both outcome accuracy (response quality) and execution efficiency (tool quality), moving beyond manual inspection.
• Regression Testing Protects Agent Baselines: Running comparative evaluations against established baselines prevents prompt refinements and tool additions from degrading overall diagnostic performance.