Observability and Metrics Monitoring with Grafana

This chapter covers the deployment, orchestration, scaling, and observability of a production-grade Retrieval-Augmented Generation (RAG) API. We will explore how to configure a Kubernetes cluster on AWS EKS, implement Horizontal and Vertical Pod Autoscaling, establish robust CI/CD pipelines with automated evaluation gates, and execute real-time load testing using Locust to observe system behavior under stress.

1. Infrastructure Architecture & Cluster Configuration

Deploying production-grade AI applications requires a highly secure, scalable, and resilient infrastructure. In this implementation, the cluster is hosted on Amazon Elastic Kubernetes Service (EKS) and provisioned using eksctl.

Infrastructure Provisioning Options

While eksctl is highly accessible for developers and rapid prototyping, production environments typically leverage enterprise-grade Infrastructure as Code (IaC) tools:

• Terraform / OpenTofu: Best for declarative, multi-cloud infrastructure state management.
• AWS Cloud Development Kit (CDK): Ideal for defining infrastructure using familiar programming languages (TypeScript, Python, Java).
• Pulumi: Excellent for imperative, code-first infrastructure deployment with native language support.

Security & Identity Management

To enforce the principle of least privilege, the cluster implements IAM Roles for Service Accounts (IRSA). Instead of assigning broad AWS IAM permissions directly to the EC2 worker nodes, IRSA maps specific AWS IAM roles directly to Kubernetes Service Accounts. This ensures that the RAG API pod can only access the specific AWS resources it needs (such as S3 buckets or Vector Databases) without exposing the underlying node infrastructure.

Kubernetes Service Account (RAG API)
        |
        | annotated with AWS IAM Role ARN
        v
   AWS STS (Security Token Service)
        |
        | assumes role & returns temporary credentials
        v
   Secure Access to AWS Resources (e.g., S3, RDS, DynamoDB)

Node Specifications

The cluster is provisioned with two primary worker nodes. To prevent resource exhaustion and manage cloud spend, resource limits are hardcoded at the node level:

• Node Count: 2 active worker nodes.
• Memory Limit: Hardcapped at $16\text{ GB}$ of RAM per node.

2. The Core Mental Model: Horizontal vs. Vertical Scaling

To maintain application reliability during traffic spikes, Kubernetes provides two primary scaling vectors: Horizontal Pod Autoscaling (HPA) and Vertical Pod Autoscaler (VPA).

Real-Life Analogy

Think of a busy restaurant kitchen.

• Vertical Scaling is like training your existing chef to work faster, handle more complex tasks, and giving them better tools (increasing their individual capacity from $6\text{ GB}$ to $8\text{ GB}$). However, a single chef has physical limits and will eventually burn out.
• Horizontal Scaling is like hiring additional chefs (spinning up more pods) to work side-by-side at identical stations when the order queue gets too long. This allows the kitchen to scale out dynamically to handle hundreds of customers.

3. Workload Isolation: Airflow vs. RAG API

A critical architectural pattern in AI system design is separating asynchronous background processing from synchronous user-facing APIs.

Airflow (Data Ingestion Pipeline)

• Replica Count: Fixed at $1$ replica.
• Access Scope: Internal only.
• Rationale: Apache Airflow orchestrates data ingestion, document parsing, and vector database synchronization. Because it runs scheduled, non-interactive background tasks, it does not require high-availability replication or low-latency scaling. Keeping it at a single replica minimizes resource overhead.

RAG API (User-Facing Service)

• Replica Count: Minimum of $2$ replicas, scaling up to $6$ replicas.
• Access Scope: Exposed to end clients via an Ingress controller / Load Balancer.
• Rationale: The RAG API processes live user queries, performs vector searches, and interacts with LLM providers. It requires high availability ($N+1$ redundancy) and dynamic scaling to handle unpredictable traffic spikes without degrading user experience.

4. CI/CD Pipeline & Automated Governance

Continuous Integration and Continuous Deployment (CI/CD) pipelines act as the quality and security gatekeepers for AI applications.

Automated Evaluation & "Golden Datasets"

To prevent regressions in LLM performance, the CI pipeline does not simply check if the code compiles; it runs evaluation testing against a Golden Dataset.

• Golden Dataset: A curated set of high-quality, representative prompt-response pairs.
• CI Gate: Every code commit triggers integration tests that run the RAG pipeline against this dataset. If the retrieval accuracy, context precision, or response quality falls below predefined thresholds, the pipeline fails, preventing the deployment of a degraded model or application.

Deployment & Rollbacks

• GitHub Actions: Orchestrates the build, test, and deployment phases.
• Multi-Image Builds: The pipeline builds two distinct Docker images:
  1. The primary FastAPI application image (RAG API).
  2. The Airflow worker/scheduler image.
• Rollback Strategy: If a deployment fails post-merge, Kubernetes allows for instant rollbacks to the previous stable replica set state via EKS, minimizing downtime.

Audit Trails & Observability

• Langfuse: Captures detailed traces of LLM calls, prompt templates, latency, token usage, and guardrail violations.
• Structured Log Files: Standardized JSON logs are written to stdout/stderr, aggregated by log collectors, and shipped to centralized logging platforms for compliance and security auditing.

5. Real-World Resilience & Cloud Outages

Production systems must be designed to handle downstream dependency failures. During the deployment demonstrated in this lecture, a real-world outage occurred in the Neon Serverless Postgres service within the us-east-1 region.

Impact of Database Outages on AI Agents

If the relational database hosting metadata, user sessions, or Model Context Protocol (MCP) configurations goes offline:

• Airflow ingestion pipelines fail immediately.
• The RAG API cannot resolve metadata queries or retrieve session history.
• Downstream LLM calls fail due to missing context.

Mitigation & Fallback Strategies

To prevent a complete system crash during database or third-party API outages, implement the following patterns:

1. Graceful Degradation: If the database is unreachable, serve cached responses or a static fallback message to the user instead of throwing a raw 500 Internal Server Error.
2. Circuit Breakers: Temporarily halt requests to the failing downstream service (e.g., Neon DB or LLM API) to prevent resource exhaustion (thread pool starvation) on the API gateway.
3. Health Checks & Probes: Configure Kubernetes liveness and readiness probes to detect when a pod cannot reach its database dependencies, preventing traffic from being routed to unhealthy pods.

6. Step-by-Step Load Testing Workflow with Locust

To validate the Horizontal Pod Autoscaler (HPA) and observe how the system scales under load, we use Locust, an open-source, Python-based load testing framework.

1. The Locust Test Script (locustfile.py)

This script simulates concurrent users sending queries to the RAG API.

from locust import HttpUser, task, between

class RAGApiUser(HttpUser):
    # Simulate a delay of 1 to 3 seconds between tasks per user
    wait_time = between(1, 3)

    @task
    def query_rag_endpoint(self):
        headers = {
            "Content-Type": "application/json",
            "Authorization": "Bearer mock-prod-token-123"
        }
        payload = {
            "query": "What are the security compliance requirements for HIPAA in our system?",
            "stream": False
        }
        self.client.post("/api/v1/query", json=payload, headers=headers)

2. Launching the Load Test

Start the Locust web interface locally or in-cluster:

locust -f locustfile.py

Navigate to http://localhost:8089 in your browser to configure the test parameters:

• Number of Users (Peak Concurrency): $10$
• Ramp Up Rate (Users started per second): $2$
• Host: http://<your-eks-ingress-loadbalancer-url>

3. Monitoring the Horizontal Pod Autoscaler (HPA)

While the load test is running, open a terminal to monitor the real-time scaling behavior of the Kubernetes pods.

# Watch the HPA status to see target vs. actual memory usage
kubectl get hpa -n production --watch

Expected Output Evolution:

Initially, the system runs with the minimum number of replicas:

NAME      REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
rag-api   Deployment/rag-api   3% / 70%  2         6         2          4h

As Locust ramps up concurrent requests, memory usage increases. Once the average memory utilization crosses the $70\%$ threshold, the HPA controller triggers the creation of new pods:

NAME      REFERENCE            TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
rag-api   Deployment/rag-api   78% / 70%  2         6         4          4h5m

Verify the new pods are spinning up successfully:

kubectl get pods -n production

NAME                       READY   STATUS    RESTARTS   AGE
rag-api-84bfb9765f-abc12   1/1     Running   0          4h
rag-api-84bfb9765f-xyz78   1/1     Running   0          4h
rag-api-84bfb9765f-new01   0/1     Pending   0          10s
rag-api-84bfb9765f-new02   1/1     Running   0          10s

7. Summary and Key Takeaways

• EKS Provisioning: EKS clusters can be provisioned using eksctl, Terraform, or AWS CDK. Securing these clusters requires mapping IAM roles directly to Kubernetes service accounts using IRSA.
• Scaling Paradigms: Horizontal scaling (HPA) adds more pod instances to distribute load, while Vertical scaling (VPA) increases resource allocations (CPU/Memory) of existing pods.
• Workload Separation: Isolate background tasks (e.g., Airflow data ingestion) from user-facing APIs (e.g., RAG API) to optimize resource allocation and ensure high availability.
• CI/CD Quality Gates: Implement automated evaluation testing against a Golden Dataset in the deployment pipeline to prevent LLM performance regressions.
• Resilience Planning: Real-world cloud outages (such as Neon DB in us-east-1) highlight the necessity of implementing circuit breakers, fallbacks, and robust health probes.
• Load Testing: Tools like Locust allow developers to simulate production traffic spikes, validating that HPA configurations scale pods seamlessly before going live.