Day 5: Deploying the Kubernetes Investigation AI Agent to Production

In this chapter, we explore how to take our Kubernetes investigation AI agent from a local development environment and deploy it into a production-ready ecosystem on Google Cloud Platform (GCP). Using the Agent CLI and our coding agent, we will examine three distinct deployment targets—Agent Runtime, Cloud Run, and Google Kubernetes Engine (GKE)—complete with networking, security configurations, and Infrastructure as Code (IaC).

1. Why Do We Need Production Deployment?

Up to Day 4, our Kubernetes investigation agent ran locally via the Agent Development Kit (ADK) framework. While local execution is ideal for iterative development, testing, and evaluation, real-world Site Reliability Engineering (SRE) workflows demand enterprise-grade production characteristics:

• High Availability & Scalability: The agent must handle concurrent incident investigations across multiple production clusters.
• Strict Security & Access Control: Access must be governed via Identity and Access Management (IAM), Virtual Private Cloud (VPC) peering, and CIDR allowlisting.
• Enterprise Integrations: Native connectivity to Gemini enterprise agent platforms, secure secret management, and zero-trust networking.

2. Core Mental Model & The Agent CLI Manifest

The deployment process is orchestrated entirely through the Agent CLI and a central configuration file located at the root of the project repository: agent-cla-manifest.yaml.

Real-Life Analogy: The Universal Freight Shipping Container

Think of the agent and its dependencies (Python runtime, ADK web UI, system prompts, and Kubernetes tools) as cargo packed into a standardized shipping container (via Docker). The agent-cla-manifest.yaml acts as the manifest document that tells the freight carrier (Agent CLI) which terminal (deployment target) to deliver the container to, and how to wire up its utilities (networking, IAM, and scaling rules).

When you execute deployment commands, the coding agent reads this manifest, generates or updates necessary infrastructure scripts (such as Terraform or gcloud commands), builds a Docker container from the base Dockerfile, and provisions the target.

3. Overview of the Three Production Deployment Targets

The Agent CLI supports three production-grade targets. Choosing the correct target depends entirely on your organization's infrastructure strategy and operational overhead constraints.

4. Method 1: Deploying to Agent Runtime (Gemini Enterprise Agent Platform)

Agent Runtime is the fastest path to production when you want Google to manage the underlying infrastructure so you can focus entirely on agent logic and tools.

Step-by-Step Workflow

1. Configure the Manifest: Instruct the coding agent to set the target to agent-runtime.
   BASH

   Copy

   # Prompt given to the coding agent:
   "Set agent deployment target to agent runtime."
   

   The agent updates agent-cla-manifest.yaml:
   YAML

   Copy

   deployment:
     target: agent-runtime
     region: us-central1
   

2. Trigger Deployment: The Agent CLI builds the container image using the project's root Dockerfile and deploys it to the Gemini Enterprise Agent Platform.
3. Track Logs: You can monitor the container build layer-by-layer using Google Cloud Log Explorer.
4. Access & Verification: Once deployed, navigate to the Gemini Enterprise Agent Platform UI, open your agent deployment (Kubernetes Investigator), and initiate a session in the playground.

Access Management in Agent Runtime

Security on Agent Runtime is strictly enforced via GCP IAM. You manage read-only, administrative, and execution permissions directly through GCP IAM service configurations without managing low-level firewall rules.

5. Method 2: Deploying to Cloud Run with Custom Networking & VPC

Cloud Run allows you to deploy containerized agents with complete serverless convenience while retaining granular control over networking, private egress, and ingress restrictions.

Network Architecture Setup

Before deploying to Cloud Run, we configure a secure enterprise networking layout:

1. Custom VPC: Create a dedicated VPC network (k8s-agent-prod-vpc).
2. Private Egress Subnet: Set up a private subnet in us-east1 with Private Google Access enabled.
3. Cloud Router & NAT Gateway: Provision a Cloud Router and NAT gateway (k8s-agent-nat-router) to allow secure outbound traffic.
4. Ingress CIDR Allowlisting: Implement environment variable validation (ALLOWED_CIDRS) in Python to reject unauthorized incoming IP addresses with an HTTP 403 Forbidden status.

# Conceptual Python Ingress Validation Middleware
from fastapi import Request, HTTPException
import ipaddress

ALLOWED_CIDRS = ["192.168.1.0/24", "10.0.0.0/8"]

async def verify_client_ip(request: Request):
    client_ip = ipaddress.ip_address(request.client.host)
    allowed = any(client_ip in ipaddress.ip_network(cidr) for cidr in ALLOWED_CIDRS)
    if not allowed and ALLOWED_CIDRS != ["*"]:
        raise HTTPException(status_code=403, detail="Access Forbidden: IP not whitelisted")

Execution Flow for Cloud Run

1. Create Custom VPC & Subnet:
   BASH

   Copy

   gcloud compute networks create k8s-agent-prod-vpc --subnet-mode=custom
   gcloud compute networks subnets create k8s-agent-prod-egress-subnet \
     --network=k8s-agent-prod-vpc --region=us-east1 --range=10.10.1.0/24 \
     --enable-private-ip-google-access
   

2. Create Cloud Router and NAT:
   BASH

   Copy

   gcloud compute routers create k8s-agent-nat-router \
     --network=k8s-agent-prod-vpc --region=us-east1
   gcloud compute routers nats create k8s-agent-nat-gw \
     --router=k8s-agent-nat-router --region=us-east1 \
     --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges
   

3. Deploy to Cloud Run with Direct VPC Egress: Instruct the coding agent:

   "Deploy the agent to cloud run with direct VPC egress attached to the k8s-agent-prod-egress-subnet."

4. Grant Public or Restricted Access: By default, newly deployed Cloud Run services return 403 Forbidden until unauthenticated public access is explicitly granted (if required for testing):
   BASH

   Copy

   gcloud run services add-iam-policy-binding k8s-investigator \
     --region=us-east1 --member="allUsers" --role="roles/run.invoker"
   

6. Method 3: Deploying to Google Kubernetes Engine (GKE) via Terraform

For organizations standardized on Kubernetes, deploying the AI agent to GKE provides ultimate orchestration flexibility, autoscaling capabilities, and deep cluster-level telemetry.

Infrastructure as Code (IaC) with Terraform

Instead of manual gcloud commands, we use Terraform to provision our GKE infrastructure. The coding agent generates a modular Terraform workspace inside a deployment/terraform/ folder:

• network.tf: VPC, subnets, routers, and firewall rules.
• variables.tf: Project IDs, region selectors, and CIDR blocks.
• network_outputs.tf: Exported infrastructure handles.
• apis.tf: Enabling required GCP service APIs.

Workload Identity Configuration

When deploying an AI agent pod inside GKE, the agent needs to securely communicate with the Gemini Enterprise Agent Platform. To avoid risky static service account keys stored in container environment variables, we configure Workload Identity:

• Mechanism: Binds a Kubernetes Service Account (KSA) inside the cluster to a Google Cloud Service Account (GSA).
• Result: Pods running in GKE automatically inherit securely vended GCP credentials via token projection.

Kubernetes Service Account (KSA)
        |
        | mapped via GKE Workload Identity
        v
Google Cloud Service Account (GSA)
        |
        | authenticates securely (no keys on disk)
        v
Gemini Enterprise Agent Platform & GCP APIs

Steps to Deploy on GKE

1. Initialize and Apply Terraform:
   BASH

   Copy

   cd deployment/terraform
   terraform init
   terraform apply -auto-approve
   

2. Build & Push Container to Artifact Registry: The agent CLI provisions an Artifact Registry repository (k8s-agent-repo), builds the agent Docker image, tags it, and pushes it.
3. Deploy Agent Pod & Load Balancer: The coding agent applies Kubernetes manifests to create the deployment, service account, and a LoadBalancer service type.
   YAML

   Copy

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: k8s-investigator
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: k8s-investigator
     template:
       metadata:
         labels:
           app: k8s-investigator
       spec:
         serviceAccountName: k8s-agent-ksa
         containers:
         - name: agent
           image: us-east1-docker.pkg.dev/k8s-investigation-agent/k8s-agent-repo/k8s-investigator:latest
           ports:
           - containerPort: 8080
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: k8s-investigation-service
   spec:
     type: LoadBalancer
     selector:
       app: k8s-investigator
     ports:
     - port: 80
       targetPort: 8080
   

4. Verification: Retrieve the external load balancer IP address and open the ADK Web UI in your browser to test interactive agent sessions.

7. Summary and Key Takeaways

• Agent CLI Manifest (agent-cla-manifest.yaml): Acts as the single source of truth for declaring deployment targets and configuring scaffolds.
• Three Production Paths:
  1. Agent Runtime: Zero infrastructure management, serverless multi-turn memory, and native IAM.
  2. Cloud Run: Serverless DX combined with custom VPC networking, direct VPC egress, and customizable ingress allowlists.
  3. GKE: Full cluster control, custom Kubernetes load balancing, and secure Workload Identity integration.
• Security Best Practices: Never embed static credentials in container images; leverage IAM and Workload Identity. Use CIDR allowlisting and HTTP status codes (403 Forbidden) to protect sensitive agent endpoints.
• ADK Web UI: Regardless of the deployment target chosen (Agent Runtime, Cloud Run, or GKE), the underlying agent exposes an interactive web interface supporting session management, state inspection, artifact tracking, and evaluation set creation.