Building an Enterprise AI Agent: Kubernetes Log Investigation

Modern cloud infrastructure demands rapid incident response, yet site reliability engineers (SREs) spend countless hours manually extracting logs, checking event streams, and correlating error signatures. This chapter demonstrates the foundational step in evolving from manual triage to autonomous operations: building the first iteration of an enterprise-grade Kubernetes investigation agent using Google's open-source Agent Development Kit (ADK), Gemini Enterprise Agent Platform, and Agent CLI driven by natural language prompting.

1. The Operational Bottleneck: SRE Triage in Modern Kubernetes

In high-velocity production environments running on Kubernetes, microservice failures occur continuously—ranging from configuration mismatches and image registry authentication failures to deadlocks and node evictions.

The traditional troubleshooting lifecycle follows a repetitive, manual loop:

This diagnostic phase is critical, but it is repetitive, time-consuming, and prone to human error under high-pressure outage scenarios.

What Defines an AI Agent?

An AI Agent is an autonomous software system capable of reasoning, breaking down complex objectives into sequential plans, and performing multi-step actions across disparate tools and environments to achieve a concrete goal.

In this implementation, the primary objective of the agent is:

• Accept raw system diagnostics (Kubernetes pod descriptions, logs, or cluster event dumps).
• Parse and isolate failure signatures without manual human filtering.
• Produce a structured root-cause analysis backed by concrete evidence and actionable remediation steps.

2. Architectural Paradigm: ADK vs. Ad-Hoc Scripting

A common anti-pattern when entering the AI engineering space is attempting to create agents by issuing ad-hoc code generation prompts to an LLM without an underlying framework. While an isolated script generated this way may function for a basic prototype, it fails within enterprise environments.

Real-Life Analogy: The Freelance Handyman vs. Standardized Construction Code

Building an AI agent using raw, prompt-generated Python scripts is like hiring an unvetted handyman to wire an electrical grid with custom extension cords. It may turn on a light bulb today, but it lacks circuit breakers, inspection certificates, or standardized conduit.

Using the Agent Development Kit (ADK) is like building according to certified commercial building codes: every wire is housed in standard conduits, load-balancing panels are pre-installed, and certified inspectors can plug monitoring equipment into standard sockets at any point.

Enterprise AI agents require standardized infrastructure for:

1. Context and State Management: Retaining long-running diagnostic state without exceeding context windows.
2. Session Management: Isolating independent user queries and multi-tenant troubleshooting workflows.
3. Model Interchangeability: Seamlessly switching underlying foundation models (e.g., swapping between specialized inference models or low-latency lightweight models like Gemini 2.5/3.8 Flash) without rewriting core business logic.
4. Tool and Protocol Integration: Natively binding external systems via standardized protocols such as the Model Context Protocol (MCP).
5. Full Lifecycle Governance: Enforcing enterprise policy checks, observability pipelines, and evaluation metrics before production rollout.

3. The Toolchain Architecture: Agent CLI, Skills, and Coding Agents

Rather than manually authoring hundreds of lines of boilerplate framework code, we orchestrate three interconnected architectural layers:

The Three Architectural Pillars

1. The AI Coding Agent: The developer-facing intelligence layer (e.g., Google Antigravity CLI, Cursor, Claude Code, GitHub Copilot). It receives natural language prompts from the engineer and translates them into file operations and commands.
2. Agent CLI: A specialized tool distributed as part of the Gemini Enterprise Agent Platform. It serves as the bridge between developer intent and the underlying ADK framework.
3. Agent Skills: Pre-packaged, structured instruction sets that teach the AI coding agent how to use the Agent CLI and ADK. When the coding agent detects an instruction to "build an agent," the skill provides the precise architectural patterns, schemas, and best practices needed to generate robust code.

4. Environment & Platform Setup: Google Cloud and Gemini Enterprise Agent Platform

Enterprise agent development requires dedicated project isolation to track API usage, control access, and segregate billing.

Step 1: Project Creation and Isolation

In the Google Cloud Platform (GCP) Console:

1. Navigate to the project selection dropdown and create a new project: k8s-investigation-agent.
2. Attach the designated billing account (utilizing GCP credits where applicable).
3. Isolate all subsequent agent resources, vector stores, and endpoints under this dedicated project ID.

Step 2: Enabling Agent Platform APIs

Search for Gemini Enterprise Agent Platform within the GCP Console. Upon initial access, initialize the platform by enabling the core API suite. This process provisions access to:

• Model Garden: Access to foundation models (Gemini Flash, Gemini Pro, and supported third-party LLMs).
• Agent Registry & Studio: Cataloging agent instances, workflows, and visual authoring pipelines.
• MCP Servers & Vector Search: Contextual grounding infrastructure for Retrieval-Augmented Generation (RAG).
• Evaluation & Observability: Real-time logging, telemetry, and automated output evaluation.

Step 3: Application Default Credentials (ADC) Setup

Instead of generating long-lived static API key strings—which introduce significant credential leakage risks in engineering environments—authenticate using Application Default Credentials (ADC).

Run the following command in your terminal to authenticate your local environment against the newly created GCP project:

gcloud auth application-default login

During login, specify your project identifier:

gcloud config set project k8s-investigation-agent

Once authenticated, the local runtime issues a handshake request against the Gemini API (e.g., testing gemini-2.5-flash or gemini-3.8-flash) to confirm that local credentials and project billing linkages are valid.

5. Configuring the Coding Agent and Agent CLI Skills

To enable our coding agent to synthesize ADK-compliant architectures, we install and configure the Google Agent CLI and its associated skill catalog.

Step 1: Initialize the Coding Agent

In this workflow, we leverage the Antigravity CLI (agy), an agentic coding environment backed by Gemini models:

mkdir k8s-investigation-agent
cd k8s-investigation-agent
agy

Upon launch, verify that the CLI detects your active GCP credentials, binds to the k8s-investigation-agent project, and registers the active inference engine (e.g., Gemini 3.8 Flash).

Step 2: Bootstrapping Agent CLI and Skills via UVX

Run the automated installation script using uvx (UV tool runner) to deploy the Agent CLI and register its skill manifests with your coding environment:

uvx google-agent-cli-setup

This installation performs three critical operations:

1. Validates the active Google user and cloud identity.
2. Installs the core agent-cli runtime binary.
3. Injects seven standardized skill manifests directly into the coding agent’s runtime directory.

The Seven Agent CLI Skills

6. Synthesizing the Agent via Natural Language Prompting

With the environment initialized and skills loaded into memory, the entire agent application can be scaffolded using a single structured natural language prompt.

The Natural Language Prompt

Provide the following prompt to the coding agent:

Build a simple Kubernetes investigation agent in Python that takes logs or diagnostic 
pod output as input to perform investigation. Post investigation, it should provide a 
detailed analysis including root cause, confidence score, and supporting evidence.

The Autonomous Synthesis Process

Upon receiving this prompt, the coding agent executes a coordinated sequence of actions guided by its skills:

Generated Project Layout

The scaffolding process produces a fully decoupled, production-ready directory structure:

Deconstructed Source Architecture (agent.py)

The generated ADK agent relies on modular primitives, cleanly isolating model declarations, prompt templates, and input parsers:

"""
Kubernetes Investigation Agent - Version 1
Built using the Agent Development Kit (ADK).
"""

from google.genai import types
from google.agent_development_kit import Agent, Runner

SYSTEM_INSTRUCTION = """
You are an expert Site Reliability Engineer (SRE) specializing in Kubernetes troubleshooting.
Your objective is to analyze the provided Kubernetes diagnostic information (pod logs,
describe outputs, or cluster events), determine the precise failure signature, and output
a structured report containing:
1. Incident Classification
2. Root Cause Analysis
3. Confidence Score (expressed as a percentage)
4. Supporting Evidence extracted directly from the logs
5. Recommended Remediation Steps
"""

def create_k8s_agent() -> Agent:
    """Initializes and configures the ADK Kubernetes Investigation Agent."""
    return Agent(
        name="k8s_investigation_agent",
        model="gemini-2.5-flash",
        system_instruction=SYSTEM_INSTRUCTION,
        # Future iterations dynamically inject MCP tools here:
        tools=[]
    )

agent = create_k8s_agent()

7. Execution Modalities: Playground, Direct Python, and Agent CLI

Once the agent is scaffolded, the developer has three distinct methods for running and validating it locally:

For enterprise workflows, agent-cli run is the standard approach because it preserves environment parity with downstream deployment, evaluation, and observability commands.

8. Real-World Validation: Triaging a Pod Failure in Production

To validate the agent against actual cluster failures, consider a live Kubernetes cluster with a pod stuck in an error state.

Step 1: Capturing Pod Failure Signatures

Checking pod status reveals an NGINX pod in an unhealthy state:

kubectl get pods

NAME                     READY   STATUS             RESTARTS   AGE
nginx-6799fc88d8-7xklw   0/1     ImagePullBackOff   0          4m12s

Extract the complete failure context using describe pod and route the output to a diagnostic file:

kubectl describe pod nginx-6799fc88d8-7xklw > error.txt

The captured error.txt file contains typical deployment diagnostic metadata:

Step 2: Executing the Agent via Agent CLI

Supply the diagnostic file to the local agent instance:

agent-cli run --input "Perform an investigation on error.txt and provide the analysis in simple words."

Step 3: Diagnostic Output Analysis

The agent executes locally, processes the file context through the configured Gemini inference pipeline, and returns a structured post-incident evaluation:

The confidence score ($\approx 98\%$) is derived directly from the explicit RPC error code (code = NotFound) matched with the spec configuration, illustrating the agent's ability to extract ground truth without hallucinating ambiguous network failure modes.

9. Evolution Roadmap: From Passive Triage to Autonomous Remediation

The architecture built in this chapter represents Version 1 (Static Diagnostics). In this stage, the human operator acts as the intermediary, copying log text from the cluster and passing it to the agent.

In subsequent iterations, this architecture will scale into a fully autonomous system:

• Direct Cluster Connectivity via MCP: Eliminating manual file transfers by providing the agent with standardized read/write Kubernetes MCP tools.
• Dynamic Cluster Queries: Empowering the agent to inspect adjacent services, query Prometheus metrics, and evaluate ingress health autonomously.
• Automated Remediation: Moving from passive recommendations to active self-healing—such as issuing rollout rollbacks, scaling node pools, or updating ConfigMaps under strict governance policies.
• Enterprise Lifecycle Policies: Enforcing evaluation checkpoints, rate limits, and OpenTelemetry-based audit logging before any cluster mutations are committed.

10. Summary and Key Takeaways

• The Manual Troubleshooting Tax: Kubernetes triage requires substantial manual effort across logs, events, and metrics; automating this flow with AI agents frees up SREs for higher-value engineering tasks.
• Agents vs. Traditional Scripts: While scripts execute rigid, linear logic, an AI agent autonomously plans, correlates disparate diagnostic signals, and provides structured reasoning with confidence ratings.
• ADK Framework Over Custom Code: Building enterprise agents requires the open-source Agent Development Kit (ADK) to standardize context management, session persistence, model portability, and tool protocols.
• Decoupled Architecture via Agent CLI: The combination of AI Coding Agents, Agent CLI, and structured Skills allows developers to scaffold enterprise-grade agents using simple natural language prompts without writing manual framework boilerplate.
• Identity and Security Best Practices: Avoid insecure, long-lived API key strings in local environments; always favor Google Cloud Application Default Credentials (ADC) tied to an isolated GCP project.
• Standardized Skills Accelerate Delivery: The Agent CLI bundles seven targeted skills (ranging from adk-code and scaffold to eval and observability) that instruct coding assistants on framework best practices.
• Production-Ready Artifacts: The scaffolding process automatically generates production artifacts—including pyproject.toml, Docker containers, and test suites—ensuring full lifecycle readiness from day one.
• Actionable Diagnostic Outputs: A properly grounded SRE agent does not merely summarize logs; it identifies root causes, assigns confidence metrics, extracts verbatim supporting evidence, and provides concrete remediation commands.
• Extensibility via MCP: Version 1 ingests static diagnostic outputs (error.txt), establishing the architectural foundation for subsequent iterations to interact directly with live clusters via the Model Context Protocol (MCP).