Horizontal Pod Autoscaling and Concurrency Under Stress

This chapter explores the practical architecture, deployment, and scaling of a production-grade Retrieval-Augmented Generation (RAG) application. It covers the transition of standard APIs into Model Context Protocol (MCP) servers, multi-channel integrations (such as Telegram bots), robust fallback mechanisms, and container orchestration on AWS Elastic Kubernetes Service (EKS) monitored via Grafana Cloud.

1. Architectural Overview: From Fast API to Model Context Protocol (MCP)

Exposing application capabilities to LLMs and agentic workflows requires transitioning from traditional REST endpoints to tool-based interfaces. The Model Context Protocol (MCP) standardizes how LLM agents interact with external data sources and tools.

Instead of building custom integration layers for every new agentic framework (e.g., LangGraph, CrewAI), the core application is exposed as an MCP server. This allows any MCP-compliant client to dynamically discover and execute application routes as tools.

Tool Discovery and Intent-Based Selection

When an MCP server is registered (for example, via a local or remote connection like npx mcp-cli pointing to http://localhost:8000), it exposes its available tools along with detailed descriptions.

• Dynamic Loading: The client does not load all tools into memory or execute them simultaneously.
• Intent Routing: Based on the user's natural language query, the LLM analyzes the tool descriptions, selects the most relevant tool, and generates the required arguments.
• Enterprise Integration: Organizations can run their main business logic inside a secure FastAPI application, wrap it with an MCP interface, and expose it internally or externally using API tokens.

2. The RAG Data Layer: Postgres, Neo4j, and OpenSearch Integration

A production RAG system separates structured metadata queries from unstructured semantic vector searches to optimize performance and reduce latency.

Hybrid Storage Architecture

Direct Database Lookups vs. Vector Search

If a user asks to "Show me the 3 most recent papers," routing this request to a vector database is inefficient. Instead, the agent selects a structured metadata tool:

1. The LLM maps the query to the list_recent_papers(limit=3) tool.
2. The application executes a direct database query against Postgres or Neo4j.
3. The metadata is returned instantly, bypassing the vector search pipeline and saving computational overhead.

3. Real-World Integration: Telegram Bot and Human-in-the-Loop Feedback

To demonstrate multi-channel accessibility, the RAG application integrates with messaging platforms like Telegram and observability backends like LangFuse.

Telegram Bot Workflow

Using Telegram's BotFather, a bot token is generated and configured within the application's environment variables. The integration operates as follows:

Human-in-the-Loop Feedback with LangFuse

To continuously evaluate and improve the RAG system, human feedback is captured directly from the user interface (e.g., thumbs up/down or rating submissions) and sent to LangFuse via the submit_feedback tool.

# Conceptual FastAPI Route converted to an MCP Tool for Feedback
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
import uuid

router = APIRouter()

class FeedbackSchema(BaseModel):
    generation_id: str
    score: float  # e.g., 1.0 for positive, 0.0 for negative
    comment: str | None = None

@router.post("/tools/submit-feedback")
async def submit_feedback(feedback: FeedbackSchema):
    """
    Submits human-in-the-loop feedback directly to LangFuse 
    to track answer relevancy and generation quality.
    """
    try:
        # Code to send metrics to LangFuse SDK
        # langfuse.score(name="user-feedback", value=feedback.score, trace_id=feedback.generation_id)
        return {"status": "success", "message": "Feedback successfully logged to LangFuse"}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

4. Resilience Patterns: Multi-Level Fallback Mechanisms

Production AI systems must be resilient to API rate limits, network partitions, and model degradation. This architecture implements three distinct layers of fallback mechanisms.

1. Retrieval Fallback

If the primary vector database (e.g., OpenSearch) fails or returns zero relevant chunks, the system retries the query. If the second attempt fails, it falls back to a keyword-based search or a cached database of historical queries before returning an error.

2. Model Fallback

If the primary LLM provider (e.g., OpenAI GPT-4o) returns a rate limit error (HTTP 429) or experiences an outage:

• The system catches the exception.
• It dynamically routes the payload to a secondary provider (e.g., AWS Bedrock running Anthropic Claude).
• This transition is transparent to the end user.

3. Chunk-Level Fallback

If the retrieved document chunks exceed the context window limit of the target LLM or cause parsing errors, the system falls back to a smaller chunk size or summarizes the chunks before generation.

Cost-Optimized Model Routing

In addition to fallbacks, Model Routing can be implemented to optimize costs. An intent classifier evaluates incoming queries:

• Simple Queries (e.g., greeting, metadata lookup) $\rightarrow$ Routed to cheaper models (e.g., GPT-4o-mini, Claude Haiku).
• Complex Queries (e.g., multi-paper synthesis, mathematical analysis) $\rightarrow$ Routed to premium models (e.g., GPT-4o, Claude Opus).

5. Kubernetes Deployment on AWS EKS

For production scaling, the application is deployed on AWS Elastic Kubernetes Service (EKS).

Environment Configuration (.env)

The application is driven by a comprehensive environment file (often exceeding 120 lines) that manages feature flags, API tokens, and infrastructure endpoints:

# Infrastructure Configuration
AWS_ACCOUNT_ID=123456789012
AWS_DEFAULT_REGION=us-east-1
EKS_CLUSTER_NAME=rag-production-cluster

# Feature Flags
ENABLE_MCP=true
ENABLE_TELEGRAM=true
PRIMARY_LLM_PROVIDER=openai # Options: openai, bedrock

# Third-Party API Tokens
OPENAI_API_KEY=sk-proj-...
TELEGRAM_BOT_TOKEN=748392...
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

# Database Endpoints
OPENSEARCH_HOST=https://opensearch.production.svc.cluster.local:9200
POSTGRES_HOST=postgres.production.svc.cluster.local

Cluster Configuration (cluster.yaml)

The EKS cluster is defined declaratively. Below is a sample configuration using cost-effective instance types for development and testing:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: rag-production-cluster
  region: us-east-1
  version: "1.30"

managedNodeGroups:
  - name: general-purpose-nodes
    instanceType: m5.xlarge # 4 vCPUs, 16 GB RAM
    desiredCapacity: 2
    minSize: 1
    maxSize: 5
    volumeSize: 50
    privateNetworking: true
    labels: { role: workers }
    tags:
      nodegroup-role: workers

Kubernetes Namespaces

To maintain clean separation of concerns, the cluster is divided into logical namespaces:

• production: Runs the core application workloads, including the FastAPI RAG API, Apache Airflow DAGs, and database connectors.
• monitoring: Dedicated to observability agents, running Prometheus, Grafana Alloy, and log forwarders that stream metrics to Grafana Cloud.

6. Monitoring and Observability with Grafana Cloud

While the default Kubernetes dashboard provides basic resource metrics, it lacks the deep observability required to debug distributed AI applications. Integrating EKS with Grafana Cloud provides unified monitoring across metrics, logs, and traces.

Key Observability Components

• Grafana Mimir (Metrics): Tracks CPU utilization, memory saturation, and network I/O across nodes and individual pods.
• Grafana Loki (Logs): Aggregates stdout/stderr streams from all containerized applications, enabling real-time log analysis and debugging of runtime exceptions (e.g., database connection timeouts or LLM API failures).
• Grafana Tempo (Traces): Tracks request lifecycles across microservices, helping identify bottlenecks in the RAG pipeline (such as slow vector database queries or high LLM generation latency).

7. Scaling Strategies: Horizontal and Vertical Pod Autoscaling under Stress

When an application experiences high concurrency (e.g., 10,000 concurrent requests), the infrastructure must scale dynamically to prevent resource exhaustion and cascading failures.

Horizontal vs. Vertical Scaling

Designing for 10,000 Concurrent Requests

To handle high-concurrency workloads, the underlying infrastructure must be sized and configured appropriately.

1. Node Instance Sizing

Running on low-spec instances (such as m5.xlarge with 16 GB RAM) limits maximum concurrency due to hardware constraints. For high-concurrency production workloads, compute-optimized or memory-optimized instances are recommended:

• Compute-Optimized: c6i.4xlarge (16 vCPUs, 32 GB RAM) for CPU-heavy workloads like local embedding generation or tokenization.
• Memory-Optimized: r6i.4xlarge (16 vCPUs, 128 GB RAM) for memory-intensive workloads like caching and vector indexing.

2. Horizontal Pod Autoscaler (HPA) Configuration

The HPA scales the number of pod replicas based on resource utilization. Below is a sample configuration that scales the RAG API deployment when CPU utilization exceeds 70%:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rag-api-deployment
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

3. Concurrency Bottlenecks

Scaling pods alone may not resolve performance issues if downstream dependencies are bottlenecked. High concurrency requires tuning the entire application stack:

Summary and Key Takeaways

• MCP Standardization: Converting FastAPI routes into Model Context Protocol (MCP) tools allows agentic frameworks to dynamically discover and execute application capabilities.
• Hybrid Data Layer: Direct database lookups (Postgres/Neo4j) bypass expensive vector search engines (OpenSearch) for structured metadata queries, reducing latency and compute costs.
• Multi-Channel Delivery: Integrating with platforms like Telegram provides accessible user interfaces, while backends like LangFuse capture human-in-the-loop feedback for continuous evaluation.
• Multi-Level Fallbacks: Production resilience requires fallback mechanisms at the retrieval, model, and chunk levels to handle API rate limits and infrastructure failures.
• Kubernetes Orchestration: Deploying workloads on AWS EKS with dedicated namespaces (production and monitoring) ensures clean resource isolation and management.
• Unified Observability: Grafana Cloud integration (Mimir, Loki, Tempo) provides deep visibility into cluster health, container logs, and request traces, simplifying debugging under stress.
• Dynamic Scaling: Combining Horizontal Pod Autoscaling (HPA) with properly sized AWS instance types allows the application to scale dynamically to meet high-concurrency demands.