Load Testing LLM Microservices with Locust

This chapter covers the production-grade Retrieval-Augmented Generation (RAG) architecture, caching strategies, agentic workflows, and Model Context Protocol (MCP) tool integrations that lay the foundation for high-performance LLM microservices. Before load testing an LLM application with tools like Locust, you must understand the underlying pipeline mechanics, latency profiles, and state-machine transitions that dictate system performance under load.

1. The Core RAG Architecture: Parsing, Chunking, and Embeddings

A production-grade RAG pipeline is only as good as its ingestion foundation. If the initial parsing of documents is flawed, subsequent chunking, embedding, and retrieval steps will suffer from a "garbage in, garbage out" cascade.

Parsing vs. Chunking

Many developers overemphasize chunking strategies while neglecting parsing.

Creator's Core Insight: "Chunking is a really important thing, but before chunking comes the main part: parsing. If your parsing is not good, your chunking will never be good."

To handle complex documents containing tables, images, and structured layouts, this architecture utilizes Docling for high-fidelity parsing, paired with a custom section-based text chunker. Rather than blindly slicing text at arbitrary character limits, the custom chunker respects document boundaries and structural sections to preserve contextual integrity.

Embedding Generation

Once sections are parsed and chunked, they are converted into dense vector representations:

• Model Provider: GenAI embedding models (from Elastic).
• Dimensionality: $1024$ dimensions.
• Flexibility: Designed as a plug-and-play interface, allowing seamless swapping with alternative providers (such as OpenAI embeddings) via environment configurations.

2. Hybrid Search Mechanics: BM25, Dense Vectors, and Reciprocal Rank Fusion (RRF)

To maximize retrieval accuracy, this architecture implements a Hybrid Search strategy that combines lexical (keyword-based) search and semantic (dense vector) search.

Lexical Search (BM25)

BM25 matches exact keywords, terminology, and specific codes. This ensures that highly specific jargon or unique identifiers are not missed due to vector space smoothing.

Dense Vector Search

Dense search captures the semantic intent of the query, mapping synonyms and conceptually related ideas even if they do not share exact keywords.

Fusion Strategies

To merge the disparate scoring systems of BM25 and dense vector search, the pipeline supports multiple fusion algorithms:

• Reciprocal Rank Fusion (RRF): A rank-based fusion method that evaluates the relative position of a document in both result sets rather than relying on raw scores.
• Relative Score Fusion (RSF): A score-normalization approach that scales and combines raw scores from both search engines.

The mathematical formulation for Reciprocal Rank Fusion (RRF) is:

Where:

• $M$ is the set of retrieval systems (e.g., BM25 and Dense Vector).
• $r_m(d)$ is the rank of document $d$ in system $m$.
• $k$ is a constant (typically set to $60$) that mitigates the impact of low-ranked outliers.

Metadata Filtering

Metadata filtering is critical when scaling to production datasets (e.g., $1,000,000+$ documents). By extracting the user's intent and applying metadata filters before or during vector search, the search space is drastically reduced, leading to highly optimized response times and more relevant context.

3. Optimizing Latency: Hash-Based Redis Caching vs. Semantic Caching

LLM API calls and vector database queries are highly latent operations. To deliver acceptable user experiences, caching is integrated at the API gateway layer.

Redis Cache Configuration

• Provider: Upstash Redis.
• Time-to-Live (TTL): Configured to $6$ hours (customizable via the .env file).
• Matching Strategy: Hash-based matching.

Hash-Based vs. Semantic Caching

4. The Complete RAG Pipeline Flow

The diagram below illustrates the complete execution path of a user query, contrasting the fast-path cache hit with the slow-path cache miss.

5. Transitioning to Agentic Workflows with LangGraph

To handle complex, non-deterministic user queries, the application transitions from a linear RAG pipeline to an agentic state machine built on LangGraph.

LangGraph State Machine Nodes

1. Guardrails Node: Evaluates incoming queries against safety and domain-relevance thresholds (e.g., a $60\%$ threshold configured via AWS Bedrock Guardrails).
2. Out of Scope Node: Catches queries that fall outside the application's domain.
3. Rewrite Query Node: If a query is ambiguous or fails to retrieve relevant chunks, this node reformulates the query. It raises the LLM temperature (e.g., to $0.7$) to encourage diverse, creative search variations rather than deterministic, rigid outputs.
4. Retrieve Node: Coordinates context retrieval.
5. OpenSearch Tool: Implemented as an external tool rather than a tightly coupled node. This decouples the core state machine from third-party infrastructure.

6. Exposing RAG as a Tool: Model Context Protocol (MCP) Integration

To make the RAG pipeline accessible to external AI assistants, IDE extensions (like Cursor), and coding agents (like Claude), the entire FastAPI application is exposed as a Model Context Protocol (MCP) server.

FastMCP Implementation

The MCP server is built using the FastMCP framework, sitting directly on top of the existing FastAPI application. This architecture exposes standard API endpoints as executable tools for MCP-compatible clients.

# Conceptual implementation of FastMCP integration within FastAPI
from fastapi import FastAPI
from mcp.server.fastmcp import FastMCP

app = FastAPI(title="RAG Microservice")
mcp = FastMCP("RAG-Agent-Service")

@mcp.tool()
async def ask_question(query: str) -> dict:
    """
    Queries the hybrid RAG pipeline to retrieve context and generate
    an answer with source PDF URLs.
    """
    # Internal call to the hybrid search and LLM generation pipeline
    response = await execute_rag_pipeline(query)
    return {
        "answer": response.answer,
        "sources": response.pdf_urls,
        "guardrail_score": response.guardrail_score,
        "trace_id": response.trace_id
    }

# Mount or run the MCP server alongside FastAPI
# Access via http://localhost:8000/mcp

Testing with the MCP Inspector

To debug and interact with the MCP tools without a frontend, developers use the MCP Inspector:

# Start the MCP Inspector pointing to the local FastAPI server
npx @modelcontextprotocol/inspector http://localhost:8000/mcp

Once connected, developers can list available tools (such as ask_question), execute queries, and inspect real-time trace payloads, including guardrail scores and LangFuse trace IDs.

Kubernetes Deployment

In production, the MCP-enabled FastAPI application is containerized and deployed to a Kubernetes cluster. The application exposes its standard HTTP endpoints alongside the /mcp transport route, allowing internal and external agentic systems to consume the RAG pipeline as a standardized tool.

7. Summary and Key Takeaways

• Parsing is Foundational: High-quality chunking is impossible without robust parsing. Tools like Docling preserve document structure (tables, images) before text segmentation.
• Hybrid Search is Standard: Combining BM25 lexical search with dense vector search (1024 dimensions) via Reciprocal Rank Fusion (RRF) yields superior retrieval accuracy.
• Caching Saves Budgets: Implementing a Redis cache with a 6-hour TTL reduces response latency from $20-30\text{ seconds}$ (cache miss) to $200-300\text{ milliseconds}$ (cache hit).
• Agentic Evolution: Moving from linear pipelines to LangGraph state machines allows for dynamic query rewriting, guardrail checks, and tool-based retrieval.
• Decoupled Tools: External databases like OpenSearch should be integrated as tools rather than hardcoded state nodes to maintain architectural flexibility.
• MCP Standardization: Exposing LLM microservices via the Model Context Protocol (using FastMCP) allows any modern AI agent or IDE to seamlessly consume your RAG pipeline as an out-of-the-box tool.