Day-6: Model Context Protocol (MCP) and Memory Bank in Production AI Agents

When moving an AI troubleshooting agent from a local development environment into production, two critical architectural gaps emerge: secure remote system execution and persistent cross-session memory. This guide details how to resolve both gaps using the Model Context Protocol (MCP) to decouple tool execution via an in-cluster server on Google Kubernetes Engine (GKE), and Gemini Enterprise Agent Platform's Memory Bank to persist root-cause findings across isolated user sessions.

1. The Production Gap: Local Development vs. Production Realities

During early local development, an AI agent operates in a privileged, co-located bubble. The agent code and the target workload typically run on the same physical host:

In this environment:

• The agent accesses the Kubernetes API directly through a static ~/.kube/config credential file stored on disk.
• Execution latency between tool calls and the local API server is negligible.
• Memory across conversations is bounded only by the lifespan of the running terminal or active runtime process.

Moving this architecture to production introduces two major architectural barriers:

1. The Remote Credential Dilemma: Deploying an agent to a managed environment (such as Agent Runtime) strips away local file access. Storing static, high-privilege kubeconfig files inside the agent runtime violates enterprise security baselines, creates key-rotation headaches, and leaves the agent vulnerable to credential exfiltration.
2. Session Amnesia: Default agent runtimes provide session-bound memory. Conversations maintain context within an isolated session thread. However, when an alert clears and a new session starts days later for the same recurring issue (e.g., an application running into CrashLoopBackOff or ImagePullBackOff), the agent retains zero context. It is forced to re-run expensive diagnostic tools, re-examine logs, and re-derive known solutions from scratch.

2. Core Mental Models

Real-Life Analogy: The Emergency Room Physician vs. The General Practitioner

Analogy:
Imagine an emergency room (ER) physician who treats a patient suffering from an acute allergic reaction. The physician runs blood tests, performs an airway exam, stabilizes the patient, and writes a detailed diagnosis in the hospital's central electronic health record (EHR).

One week later, the same patient returns with identical symptoms.

• Without Memory Bank (Session-Only Memory): A new doctor attends to the patient with complete amnesia. The doctor orders the full panel of blood tests again, re-runs diagnostics from scratch, and takes an hour to deduce what was already solved last week.
• With Memory Bank (Cross-Session Persistent Memory): The attending doctor queries the central EHR first, observes the diagnosis from seven days prior, immediately checks if the same allergen is present, and resolves the incident in minutes.

The MCP Client-Server Boundary

The Model Context Protocol (MCP) provides an open standard for agents to discover and invoke tools over standard communication protocols (such as Streamable HTTP or standard input/output).

Instead of embedding domain-specific SDKs and cluster credentials directly inside the AI agent:

• The agent becomes a lightweight MCP Client.
• An MCP Server runs co-located with the target infrastructure (inside the GKE cluster).
• The MCP server uses native, in-cluster Kubernetes Service Account tokens (RBAC) to run diagnostics and exposes structured tool definitions over a network interface.

AI Agent (Agent Runtime)
   |
   | 1. Discovers tools & sends tool invocation over HTTP
   v
GKE Load Balancer (MCP Endpoint)
   |
   | 2. Routes request to internal Pod
   v
In-Cluster MCP Server (FastMCP / Python)
   |
   | 3. Queries API via local In-Cluster ServiceAccount
   v
Kubernetes Control Plane (Kube API Server)

3. High-Level Implementation Roadmap

The transition from a stateless, locally bounded agent to a secure, stateful production agent follows a structured five-step lifecycle:

Step 1: Standardize Deployment Target
   | (Switch to Agent Runtime for native session management and Memory Bank support)
   v
Step 2: Deploy In-Cluster MCP Server
   | (Deploy FastMCP inside GKE, expose inspection tools over LoadBalancer)
   v
Step 3: Connect MCP Client & Smoke Test
   | (Redeploy Agent Runtime agent with remote MCP URL; verify node discovery)
   v
Step 4: Configure Enterprise Memory Bank
   | (Enable cross-session memory ingestion, simulate and resolve incident)
   v
Step 5: Automated Evaluation & Regression
     (Run quantitative evaluation for memory recall accuracy and turn-count reduction)

4. Step 1: Standardizing on Agent Runtime

The Gemini Enterprise Agent Platform supports multiple hosting targets, including Cloud Run, Google Kubernetes Engine (GKE), and Agent Runtime. For this deployment, Agent Runtime is the preferred control plane:

• Zero Infrastructure Maintenance: Abstracted compute layer with zero manual node or container management.
• Native Observability: Deep tracing of LLM reasoning traces, tool-invocation latency, and input/output payloads out of the box.
• First-Class Platform Integrations: Built-in support for conversational session management and direct integration with Enterprise Memory Bank.

The project configuration is aligned using the Agent CLI:

# Scaffold and point agent deployment target to Agent Runtime
agent-cli scaffold --target agent-runtime --update-config

This updates agent.yaml to ensure dependencies and platform configurations point to the managed runtime infrastructure rather than local container runtimes.

5. Step 2: Architecting and Deploying the In-Cluster MCP Server

Architecture Decision: Off-the-Shelf vs. Custom FastMCP Server

When deploying an MCP server to Kubernetes, engineers typically choose between two patterns:

1. Off-the-shelf Generic MCP Image: Generic images expose arbitrary kubectl primitives. While flexible, they risk violating the principle of least privilege by exposing wide APIs to LLM hallucination or misuse.
2. Tailored FastMCP Python Server: Exposes explicit, hardened, domain-specific troubleshooting functions (e.g., get_pod_events, get_pod_logs) over Streamable HTTP. This encapsulates diagnostic logic and enforces strict parameter validation.

We select the custom FastMCP Python Server implementation.

Implementation: The FastMCP Server (mcp_server.py)

The server wraps the official Kubernetes Python client, exposing five core troubleshooting tools:

"""
In-cluster MCP Server exposing targeted Kubernetes diagnostic tools.
"""
from fastmcp import FastMCP
from kubernetes import client, config

# Initialize FastMCP Server
mcp = FastMCP("k8s-investigator-mcp-server")

# Load in-cluster service account credentials
try:
    config.load_incluster_config()
except config.ConfigException:
    config.load_kube_config()

core_v1 = client.CoreV1Api()
apps_v1 = client.AppsV1Api()

@mcp.tool()
def get_node_info() -> dict:
    """Fetch node health, capacity, and system specifications."""
    nodes = core_v1.list_node()
    summary = []
    for node in nodes.items:
        conditions = {c.type: c.status for c in node.status.conditions}
        summary.append({
            "name": node.metadata.name,
            "status": "Ready" if conditions.get("Ready") == "True" else "NotReady",
            "kubelet_version": node.status.node_info.kubelet_version,
            "os_image": node.status.node_info.os_image
        })
    return {"total_nodes": len(nodes.items), "nodes": summary}

@mcp.tool()
def get_pod_events(namespace: str, pod_name: str) -> list:
    """Retrieve Kubernetes events associated with a specific pod."""
    events = core_v1.list_namespaced_event(
        namespace=namespace,
        field_selector=f"involvedObject.name={pod_name}"
    )
    return [
        {
            "reason": event.reason,
            "message": event.message,
            "type": event.type,
            "count": event.count
        }
        for event in events.items
    ]

@mcp.tool()
def get_deployment_history(namespace: str, deployment_name: str) -> dict:
    """Check rollout status and replica state for a given deployment."""
    dep = apps_v1.read_namespaced_deployment(name=deployment_name, namespace=namespace)
    return {
        "name": dep.metadata.name,
        "replicas": dep.status.replicas,
        "ready_replicas": dep.status.ready_replicas,
        "updated_replicas": dep.status.updated_replicas
    }

@mcp.tool()
def get_pod_logs(namespace: str, pod_name: str, tail_lines: int = 100) -> str:
    """Fetch trailing container logs for root-cause analysis."""
    return core_v1.read_namespaced_pod_log(
        name=pod_name,
        namespace=namespace,
        tail_lines=tail_lines
    )

@mcp.tool()
def get_pod_info(namespace: str, pod_name: str) -> dict:
    """Fetch pod status, phase, restarts, and container states."""
    pod = core_v1.read_namespaced_pod(name=pod_name, namespace=namespace)
    container_statuses = []
    for cs in pod.status.container_statuses or []:
        state = "running" if cs.state.running else ("waiting" if cs.state.waiting else "terminated")
        reason = cs.state.waiting.reason if cs.state.waiting else None
        container_statuses.append({
            "name": cs.name,
            "state": state,
            "reason": reason,
            "restart_count": cs.restart_count
        })
    return {
        "pod_name": pod.metadata.name,
        "phase": pod.status.phase,
        "container_statuses": container_statuses
    }

if __name__ == "__main__":
    # Start streamable HTTP server on port 8080
    mcp.run(transport="sse", host="0.0.0.0", port=8080)

In-Cluster Kubernetes Manifest (mcp_server.yaml)

The MCP server runs inside its own namespace, attached to a distinct ServiceAccount bounded by a read-only ClusterRole:

apiVersion: v1
kind: Namespace
metadata:
  name: k8s-investigator
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-mcp-sa
  namespace: k8s-investigator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-mcp-reader
rules:
- apiGroups: [""]
  resources: ["nodes", "pods", "pods/log", "events", "services"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-mcp-reader-binding
subjects:
- kind: ServiceAccount
  name: k8s-mcp-sa
  namespace: k8s-investigator
roleRef:
  kind: ClusterRole
  name: k8s-mcp-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-mcp-server
  namespace: k8s-investigator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-mcp-server
  template:
    metadata:
      labels:
        app: k8s-mcp-server
    spec:
      serviceAccountName: k8s-mcp-sa
      containers:
      - name: mcp-server
        image: gcr.io/my-gcp-project/k8s-mcp-server:v1.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-mcp-server-service
  namespace: k8s-investigator
spec:
  type: LoadBalancer
  selector:
    app: k8s-mcp-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Applying this manifest binds the service account, assigns the RBAC policies, launches the container, and provisions a public external IP via Google Cloud Network Load Balancing:

kubectl apply -f mcp_server.yaml

6. Step 3: Configuring the Remote Agent Client and Smoke Testing

Agent Client Registration

With the MCP server operational and exposed via the load balancer endpoint, the agent running in Agent Runtime must be reconfigured to communicate with it.

We update the agent's tool configuration to consume the remote MCP service and trigger an automated deployment via the CLI:

agent-cli deploy \
  --target=agent-runtime \
  --project=my-gcp-project \
  --mcp-server-url="http://34.121.X.X/sse"

The underlying deployment engine:

1. Re-packages the agent logic.
2. Updates tool definitions to target the remote Server-Sent Events (SSE) stream.
3. Performs a dynamic handshake with the endpoint to discover exposed tools (get_node_info, get_pod_events, etc.).

Verification and Smoke Testing

To verify the handshake without local credentials, open the agent interactive playground and submit a query:

Prompt: "Get the Kubernetes cluster node information."

Execution Trace:

1. The agent interprets the user intent.
2. It detects that get_node_info matches the requirement.
3. It dispatches an MCP remote execution request over HTTP.
4. The MCP server queries the Kubernetes API and returns the node attributes.
5. The agent synthesizes the response:

The matching node identifier confirms that the agent successfully inspected the remote cluster without retaining local cluster access credentials.

7. Step 4: Enabling Agent Memory Bank for Cross-Session Continuity

The Limits of Default Session Isolation

In standard agent architectures:

• Each user interaction or alert context is allocated a unique session_id.
• Context windows are retained only for the duration of that specific session_id.
• Once an alert resolves and the session closes, the context window resets.

If an application enters an error state repeatedly, the agent re-executes the exact same diagnostic loops:

Configuring Memory Bank on Gemini Enterprise Agent Platform

Memory Bank introduces an asynchronous memory engine. When enabled:

1. Extraction (Post-Triage): After an investigation, a background job extracts key artifacts (workload name, error type, root cause, remediation steps).
2. Indexing: Findings are embedded and stored in an enterprise-managed vector index linked to the agent's identity.
3. Retrieval (Pre-Triage): Upon receiving any new prompt or alert, the platform automatically queries the Memory Bank. Relevant past incident summaries are injected into the agent's context before tools are evaluated.

# Conceptual memory configuration inside agent setup
agent_config = {
    "name": "k8s-investigator",
    "deployment_target": "agent-runtime",
    "memory_bank": {
        "enabled": True,
        "extraction_policy": "AUTO_ON_SESSION_COMPLETE",
        "retrieval_strategy": "SEMANTIC_SIMILARITY",
        "max_memories_retrieved": 3
    }
}

Deploying this configuration wires the memory persistence layer directly into the runtime:

agent-cli deploy --update-memory-config

8. Incident Lifecycle Analysis: Cold vs. Warm Investigation Flow

To validate Memory Bank, we deploy an intentionally broken deployment that fails image pulling:

kubectl create deployment nginx-broken --image=nginx:non-existent-tag-v99

The pod enters ImagePullBackOff / ErrImagePull.

Session A: Cold Investigation (No Prior Memory)

Session B: Warm Investigation (With Memory Bank Active)

A completely new session (session_id: session-9941) is initiated:

Total tool turns drop from three round-trips to zero, reducing investigation latency and token consumption significantly.

9. Architectural Comparison: Context Strategies

10. Step 5: Automated Evaluation and Regression Testing for Memory Recall

To ensure that future agent prompt adjustments or MCP schema updates do not degrade memory retrieval, we add an automated evaluation scenario using the Agent CLI evaluation skill (eval).

Evaluation Configuration and Criteria

The evaluation framework scores three primary metrics:

1. Memory Recall Accuracy: Did the agent correctly cite the historical root cause without inventing new diagnostic data?
2. Custom Response Quality: Semantic correctness and clarity of the remediation plan.
3. Agent Turn Count: Did the agent bypass redundant tool invocations when answering from memory?

agent-cli eval run --scenario="memory-bank-recall-evaluation"

Evaluation Output Trace

Running this test as part of a CI/CD deployment pipeline guarantees that changes to system prompts, underlying foundation models, or tool definitions will not break long-term context recall.

11. Production Pitfalls, Security Realities, and Hardening

Deploying an MCP server and Memory Bank to production requires addressing several operational and security concerns:

1. Ingress Network Protection

Note:
The demonstration architecture exposes the FastMCP server over a public GKE LoadBalancer service to simplify setup. Do not use an unauthenticated public Load Balancer in enterprise production environments.

Production Hardening Patterns:

• Internal Load Balancer (ILB) + Private Service Connect (PSC): Place the MCP server behind an Internal Passthrough Network Load Balancer. Use Google Cloud Private Service Connect to expose the MCP service directly to the Agent Runtime VPC without traversing the public internet.
• Mutual TLS (mTLS) and Shared Bearer Tokens: Enforce an API Gateway or Envoy proxy ahead of the MCP server that requires a validated OAuth2 / IAM bearer token on every JSON-RPC/SSE frame.

2. Fine-Grained Kubernetes RBAC

Never bind the MCP server to cluster-admin.

• Restrict read permissions exclusively to the resources the agent must observe (pods, events, deployments, logs).
• If remediation capabilities (e.g., restarting pods, scaling deployments) are added later, split them into a separate, audited MCP server with explicit human-in-the-loop approval workflows.

3. Memory Bank Invalidation and Stale State

Cluster states change dynamically:

• If a developer fixes an invalid image tag and redeploys, an agent relying solely on historical memory might falsely report that the service is broken.
• Mitigation: The agent's reasoning framework must include a conditional verification step: If the user asks to diagnose current health, perform a single lightweight status check first; if status is healthy, mark historical memory as resolved.

12. Complete Architectural Flow

The diagram below maps the complete interaction loop between the human operator, Agent Runtime, the GKE cluster running the MCP server, and the Enterprise Memory Bank:

Summary and Key Takeaways

• Decoupled Architecture: Hosting the agent in Agent Runtime while running an MCP server inside GKE eliminates the need to distribute static kubeconfig files, ensuring clean separation between reasoning and execution.
• FastMCP Protocol Implementation: The Python FastMCP server encapsulates standard Kubernetes SDK calls into five focused troubleshooting tools exposed over a streamable transport.
• RBAC Containment: The MCP server uses native Kubernetes Service Accounts and ClusterRoles, bounding the agent's blast radius to read-only diagnostic operations.
• Solving Session Amnesia: Standard agent runtimes drop context when sessions terminate; Memory Bank introduces a persistent vector memory layer that stores root-cause findings across distinct sessions.
• Efficiency and Latency Improvements: Storing historical incident resolutions allows the agent to recall root causes for recurring issues immediately, reducing tool round-trips from multiple turns to zero.
• Automated Regression Prevention: Integrating an evaluation suite via the Agent CLI verifies memory recall accuracy, response quality, and reduced turn counts across iterations.
• Production Hardening Requirements: Production enterprise deployments must secure the MCP endpoint using Internal Load Balancers and Private Service Connect, while implementing cache-invalidation rules to avoid acting on stale incident memories.