Kubernetes (EKS) Infrastructure and Cluster Deployment

This chapter covers the end-to-end infrastructure, application services, and data pipelines that form the core of a production-grade, secure Agentic Retrieval-Augmented Generation (RAG) system. We will explore how guardrails, orchestration engines (Apache Airflow), vector databases (OpenSearch), serverless databases (Neon DB), and caching layers (Upstash Redis) integrate to build a resilient, enterprise-ready AI application designed for deployment on Kubernetes (EKS).

1. Why Do We Need This? (The Production AI Infrastructure Challenge)

Deploying a basic RAG application on a local machine is straightforward, but transitioning it to a production-ready, secure, and scalable enterprise system introduces significant architectural challenges:

• Vulnerability to Malicious Inputs: Unfiltered LLM applications are susceptible to prompt injections, requests for dangerous content (e.g., weapon construction, self-harm), and leaks of Personally Identifiable Information (PII).
• Data Ingestion Bottlenecks: Ingesting, parsing, chunking, and indexing thousands of academic or corporate documents (such as arXiv papers) cannot be done synchronously within user request threads. It requires a robust, distributed orchestration pipeline.
• State and Cache Management: To prevent redundant, expensive LLM API calls and speed up response times, systems need high-performance caching layers and persistent databases to track ingestion states and user sessions.
• Scale and High Availability: Running these workloads requires an orchestrator like Kubernetes (EKS) to manage container lifecycles, handle auto-scaling, and ensure high availability across multiple availability zones.

2. Core Mental Model: The Agentic RAG & Guardrail Architecture

To address these challenges, we decouple the system into three primary layers: the Security Guardrail Layer, the Asynchronous Data Ingestion Pipeline, and the Stateful/Caching Infrastructure.

3. AI Guardrails in Action: Content Filtering, PII Redaction, and Tracing

A production-grade AI gateway must validate both incoming user queries and outgoing LLM responses. The guardrail system implemented here uses specialized classifiers to detect and mitigate risks across several categories.

Real-Life Analogy: The Nightclub Bouncer and the Redactor

Think of the guardrail system as a two-stage security team at a high-security facility:

1. The Bouncer (Blocking): Instantly turns away anyone trying to bring in contraband (e.g., requests for violence, NSFW content, or bomb-making instructions).
2. The Compliance Officer (Redacting): Allows people inside but blackouts sensitive documents (e.g., replacing credit card numbers or social security numbers with [REDACTED]) before they can be viewed by unauthorized personnel.

Guardrail Categories & Behaviors

Tracing and Observability

Every single interaction—whether blocked, redacted, or allowed—is traced back to two locations:

• LangFuse: An open-source LLM engineering platform that tracks latency, prompt templates, token usage, and guardrail evaluations.
• Local Log Files: Structured JSON logs written to disk for ingestion by log aggregators.

4. The Data Ingestion Pipeline: Orchestrating arXiv with Apache Airflow

To keep our vector database populated with the latest research papers, we use Apache Airflow (running locally on port 8080 with default credentials admin/admin for development) to orchestrate an asynchronous Directed Acyclic Graph (DAG).

Ingestion State Tracking in Neon DB

To prevent duplicate processing, the pipeline updates a tracking table in Neon DB (a serverless Postgres database). Before parsing a paper, the DAG queries Neon DB to check if the paper's unique identifier (e.g., arXiv ID) already exists.

5. Database & Cache Topology: Neon DB, OpenSearch, and Upstash Redis

Our infrastructure relies on three distinct database technologies, each optimized for a specific access pattern:

1. OpenSearch (Vector Database)

• Port: 5601 (Dashboard) / 9200 (API)
• Role: Stores high-dimensional vector embeddings of document chunks.
• Scale: In our deployment, the archive_paper_chunks index contains approximately $6,100$ chunks derived from $317$ parsed documents.
• Optimization: Optimized for fast $K$-Nearest Neighbors ($k$-NN) vector similarity searches using hierarchical navigable small world (HNSW) graphs.

2. Neon DB (Serverless Postgres)

• Role: Relational storage for structured application state, user sessions, DAG run workflows, and document ingestion metadata.
• Why Serverless?: Neon decouples compute from storage, allowing the database to scale down to zero when inactive, saving cloud costs during development and low-traffic periods.

3. Upstash Redis (Serverless Cache)

• Role: High-performance, low-latency key-value cache.
• Mechanism: When a user submits a query, the system hashes the query and checks Redis. If a cache hit occurs, the cached response is returned instantly, bypassing the vector search and LLM generation steps.
• Why Serverless?: Upstash provides a fully managed Redis interface that scales dynamically without requiring manual cluster provisioning.

6. Software Design Patterns: Factory Pattern for Decoupled Services

To maintain a clean, maintainable codebase, the application layer uses the Factory Pattern to instantiate services, chunkers, and database clients. This pattern provides high levels of encapsulation and abstraction, allowing developers to switch underlying technologies (e.g., swapping OpenSearch for Weaviate, or changing chunking strategies) with zero changes to the core business logic.

Example: Factory Pattern Implementation for Document Chunkers

The following code demonstrates how the system dynamically instantiates different chunking strategies using a factory pattern.

# services/chunking/factory.py
from typing import Protocol, Dict, Type

class ChunkingStrategy(Protocol):
    def chunk_text(self, text: str, chunk_size: int, overlap: int) -> list[str]:
        ...

class CharacterChunker:
    def chunk_text(self, text: str, chunk_size: int, overlap: int) -> list[str]:
        # Naive character-based splitting
        chunks = []
        start = 0
        while start < len(text):
            end = start + chunk_size
            chunks.append(text[start:end])
            start += chunk_size - overlap
        return chunks

class SemanticChunker:
    def chunk_text(self, text: str, chunk_size: int, overlap: int) -> list[str]:
        # Advanced semantic-boundary splitting (e.g., using sentence embeddings)
        print("Performing semantic chunking based on sentence boundaries...")
        # Implementation details omitted for brevity
        return [text] # Mocked return

class ChunkerFactory:
    _registry: Dict[str, Type[ChunkingStrategy]] = {
        "character": CharacterChunker,
        "semantic": SemanticChunker
    }

    @classmethod
    def register_chunker(cls, name: str, chunker_class: Type[ChunkingStrategy]):
        cls._registry[name] = chunker_class

    @classmethod
    def get_chunker(cls, strategy_name: str) -> ChunkingStrategy:
        chunker_class = cls._registry.get(strategy_name.lower())
        if not chunker_class:
            raise ValueError(f"Unknown chunking strategy: {strategy_name}")
        return chunker_class()

# Usage in Application
chunker = ChunkerFactory.get_chunker("semantic")
chunks = chunker.chunk_text("Sample document text...", chunk_size=500, overlap=50)

7. Vector Databases Demystified: Indexes, Collections, and Dimensions

For developers transitioning from relational databases, understanding how vector databases store and query data is critical.

Concept Comparison

What is a Vector Record?

If a single PDF document is parsed and split into $10$ distinct text chunks, the vector database will store $10$ separate records. Each record consists of:

1. The Vector Embedding: An array of floating-point numbers generated by an embedding model (e.g., [0.123, -0.456, 0.987, ...]).
2. Metadata: The raw text chunk, source document URL, page number, and creation timestamp.

Vector Dimensions

The size of the vector array (its dimensionality) is strictly determined by the embedding model used. For example:

• text-embedding-3-small (OpenAI): $1536$ dimensions.
• text-embedding-3-large (OpenAI): $3072$ dimensions.
• all-MiniLM-L6-v2 (SentenceTransformers): $384$ dimensions.

Note: You cannot mix dimensions within a single index. If an index is configured for $1536$ dimensions, every vector written to or queried against that index must have exactly $1536$ dimensions.

8. The Evolution of Search: From BM25 to Hybrid, Graph, and Vectorless RAG

Search technology has evolved rapidly, moving from simple keyword matching to multi-dimensional, context-aware retrieval systems.

1. Keyword Search (BM25)

• Mechanism: An advanced evolution of TF-IDF (Term Frequency-Inverse Document Frequency). It scores documents based on how often query terms appear relative to the document length and term frequency across the entire index.
• Formula:

Where:
*   $f(q_i, D)$ is the term frequency of query term $q_i$ in document $D$.
*   $\vert D \vert$ is the length of document $D$ in words.
*   $\text{avgdl}$ is the average document length in the text collection.
*   $k_1$ and $b$ are free parameters (typically $k_1 \in [1.2, 2.0]$ and $b = 0.75$).

2. Dense Vector Search

• Mechanism: Converts queries and documents into dense vectors and calculates semantic similarity (typically using Cosine Similarity or Dot Product). It excels at capturing conceptual meaning and synonyms (e.g., matching "feline" with "cat").

3. Hybrid Search

• Mechanism: Combines BM25 and Dense Vector search scores using algorithms like Reciprocal Rank Fusion (RRF) to deliver highly accurate results that capture both exact keyword matches (like serial numbers or specific names) and semantic intent.

4. Graph-RAG (The Modern Trend)

• Mechanism: Integrates Knowledge Graphs (using databases like Neo4j or FalkorDB) with vector search. This allows the system to perform complex, multi-hop reasoning (e.g., "Find all papers written by co-authors of Dr. Smith that discuss quantum computing").

5. Vectorless Solutions

• Mechanism: Platforms like Weaviate are pioneering hybrid architectures that allow developers to run highly optimized search queries without managing raw vector pipelines manually, making them highly suitable for complex financial and enterprise compliance documents.

9. Production Pitfalls & Edge Cases: Telegram Replicas and Serverless Limits

When deploying these services to a production Kubernetes (EKS) cluster, developers must design around several critical edge cases:

The Telegram Bot Replica Crash Loop

• The Problem: The application includes a Telegram integration. In a local environment with a single process, this works perfectly. However, when deployed to EKS with multiple pod replicas, each replica attempts to call the Telegram getUpdates API endpoint simultaneously.
• The Failure Mode: Telegram's API only supports one active long-polling connection per bot token. When Replica B calls getUpdates, Replica A's connection is forcefully closed, causing Replica A to throw an unhandled exception and crash. Kubernetes detects the crash and restarts Replica A, which then terminates Replica B's connection, creating a continuous CrashLoopBackOff cycle.
• The Solution: Never use long-polling (getUpdates) in a multi-replica production environment. Instead, configure Telegram to use Webhooks, where Telegram pushes updates to a single, load-balanced API endpoint (e.g., https://api.yourdomain.com/telegram/webhook) exposed by your EKS ingress controller.

Serverless Database Cold Starts and Connection Limits

• The Problem: Serverless databases like Neon DB and Upstash Redis scale down to zero when idle.
• The Failure Mode: The first user request after a period of inactivity can experience latency spikes (cold starts) of several seconds while the database compute instance provisions. Additionally, serverless databases often have lower concurrent connection limits than provisioned instances.
• The Solution: Implement robust connection pooling (using tools like PgBouncer for Postgres) and configure aggressive connection timeouts and retry policies in your FastAPI application.

10. Complete Request-Response Flow Walkthrough

To understand how these components interact, let's trace a single user request through the entire system:

[User] 
  |
  | 1. Sends query: "How do I build a secure RAG system?"
  v
[FastAPI Gateway]
  |
  | 2. Intercepts request and runs Guardrail checks
  |    - Checks for violence, NSFW, PII, and off-topic queries
  v
[Guardrail Layer]
  |
  | 3. Query passes safety checks (No violations found)
  v
[Upstash Redis Cache]
  |
  | 4. Checks for existing cache hit
  |    - Cache Miss! (Query has not been asked recently)
  v
[OpenSearch Vector DB]
  |
  | 5. Converts query to vector embedding and queries 'archive_paper_chunks'
  |    - Retrieves top 3 most relevant document chunks
  v
[Neon DB]
  |
  | 6. Fetches metadata and verifies document ingestion status
  v
[LLM (OpenAI via LangFuse)]
  |
  | 7. Sends prompt: [System Instructions] + [Retrieved Chunks] + [User Query]
  |    - Generates natural language response
  |    - Logs execution trace, latency, and token usage to LangFuse
  v
[Upstash Redis Cache]
  |
  | 8. Writes generated response to cache for future requests
  v
[FastAPI Gateway]
  |
  | 9. Returns final response to User
  v
[User]

11. Summary and Key Takeaways

• Multi-Layered Guardrails: Production LLM systems must implement robust input and output filtering. Use blocking for severe violations (violence, NSFW, credit cards) and redaction for standard PII (phone numbers, emails) to preserve user experience.
• Asynchronous Ingestion: Never parse or index documents synchronously within user request threads. Use an orchestrator like Apache Airflow to manage complex, multi-stage document processing pipelines.
• Decoupled Architecture: Utilize the Factory Pattern to abstract database, chunking, and LLM services. This ensures your codebase remains modular and adaptable to changing technology stacks.
• Hybrid Search is Standard: Modern RAG systems combine the semantic power of Dense Vector search with the exact-match precision of BM25 keyword search using Reciprocal Rank Fusion (RRF).
• Kubernetes Deployment Gotchas: Be aware of stateful integrations (like Telegram long-polling) that fail when scaled to multiple replicas. Always design stateless application pods and offload state to robust, external serverless or clustered databases.