Building an Agentic RAG System from Scratch with LangChain

A practical guide to transitioning from static Retrieval-Augmented Generation (RAG) pipelines to autonomous, tool -calling Agentic RAG architectures. This guide covers how to handle compound queries, execute dynamic similarity searches, and build an end-to-end agent in Python using LangChain and OpenAI.

1. The Bottleneck of Standard (Naive) RAG

Standard RAG follows a rigid, linear pipeline:

1. Ingest and parse documents.
2. Split documents into text chunks.
3. Compute vector embeddings and store them in a Vector Database.
4. Take a user prompt, compute its embedding, and run a vector similarity search ($K$-nearest neighbors) to pull top-$K$ chunks.
5. Inject retrieved chunks into a static context window and send them to the Large Language Model (LLM) for synthesis.

The Problem: Multi-Intent and Compound Queries

Consider a vector database containing heterogeneous documents: Course Catalogs, Company Policies, and Movie Releases.

Suppose a user submits a compound prompt:

"What is the fee for the course, and what are the latest movies released this month?"

In a naive RAG setup, this entire string is embedded into a single high-dimensional vector. That single vector is compared against all stored document embeddings.

This creates several failure modes:

• Vector Averaging / Semantic Smearing: The combined embedding sits halfway between the "Education" cluster and the "Entertainment" cluster. It may match neither topic well.
• Top-$K$ Context Starvation: If $K = 4$, the database might return three course documents and one movie document, or all four about courses. Critical context for the other half of the prompt is dropped.
• Context Dilution: The LLM receives partial data and either hallucinates the missing half or explicitly states it cannot find the information.

2. Core Mental Model: Agentic RAG

Agentic RAG replaces the static retrieval pipeline with a reasoning-and-action loop driven by an AI agent. The retriever is wrapped as a Tool exposed to the agent.

Key Differences: Naive RAG vs. Agentic RAG

3. Data Ingestion, Chunking, and In-Memory Vector Storage

Before introducing an agent, the vector database must be seeded with indexed chunks.

Environment Setup and Document Parsing

Install the necessary packages and load environment variables:

import os
from dotenv import load_dotenv

# Load OpenAI API Key from .env
load_dotenv()

Load the source document (medical_report.pdf):

from langchain_community.document_loaders import PyPDFLoader

# Load PDF (9 pages)
loader = PyPDFLoader("data/medical_report.pdf")
docs = loader.load()

print(f"Total pages loaded: {len(docs)}")  # Output: 9

Chunking Strategy

Raw pages often exceed optimal embedding limits or combine unrelated topics. We use RecursiveCharacterTextSplitter to enforce boundaries:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Initialize text splitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

split_docs = splitter.split_documents(docs)
print(f"Total chunks created: {len(split_docs)}")  # Output: 26

Why Overlap Matters: A chunk_overlap of 200 characters prevents semantic loss when entities or sentences are sliced across chunk boundaries.

Embeddings and Vector Store Setup

We use OpenAI's text-embedding-3-small and LangChain's InMemoryVectorStore to index the chunks:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import InMemoryVectorStore

# Instantiate embedding model
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Create vector store directly from split chunks
vector_store = InMemoryVectorStore.from_documents(
    documents=split_docs,
    embedding=embeddings
)

Verify index functionality using standard similarity search:

# Validation test
records = vector_store.similarity_search("patient name", k=4)
print(f"Retrieved {len(records)} chunks.")
print(f"Sample Content:\n{records[0].page_content[:200]}...")

4. Converting a Retriever into an Agent Tool

For an agent to query the vector store dynamically, the retrieval logic must be exposed as a Tool.

In LangChain, a tool requires three elements:

1. Name: Inferred from the function name.
2. Type Hinting: Tells the LLM what data types to emit (e.g., query: str).
3. Docstring (__doc__): Used directly by the model to determine when and why to call this tool.

Implementing the Tool

from langchain_core.tools import tool

@tool
def retriever_tool(query: str) -> str:
    """
    Retrieves relevant medical report context based on a search query.
    Use this tool to find information about patient records, doctor details,
    diagnoses, and medical test results within the uploaded documents.
    """
    # Debug print to monitor query deconstruction during agent runtime
    print(f"[TOOL EXECUTION] retriever_tool called with query: '{query}'")
    
    # Run similarity search against our in-memory vector store
    matched_docs = vector_store.similarity_search(query=query, k=4)
    
    # Normalize context into a single string with paragraph separators
    context_chunks = [doc.page_content for doc in matched_docs]
    combined_context = "\n\n".join(context_chunks)
    
    return combined_context

Note: The @tool decorator converts standard Python functions into StructuredTool instances compatible with OpenAI's function/tool calling API.

5. Wiring the Complete Agent Pipeline

Once the tool is registered, we wire together the LLM, the System Prompt, and the Agent Controller.

Agent Construction

from langchain_openai import ChatOpenAI
from langchain.agents import create_agent

# 1. Initialize the chat model supporting tool calls
llm = ChatOpenAI(model="gpt-4o", temperature=0.0)

# 2. Register tools
tools = [retriever_tool]

# 3. Define the system prompt
system_prompt = (
    "You are an advanced medical assistant. Your job is to answer questions "
    "accurately using only the retrieved context from the provided tools. "
    "Always invoke the retriever_tool when external facts are required to answer "
    "questions regarding patient records, doctors, or lab results."
)

# 4. Create the agent instance
agent = create_agent(
    model=llm,
    tools=tools,
    system_prompt=system_prompt
)

6. Execution Trace: Compound Query Deconstruction

To see Agentic RAG in action, submit a multi-intent compound query:

query = "What is the name of the patient and what are the names of the doctors?"

# Invoke the agent passing conversational state
response = agent.invoke({
    "messages": [
        {"role": "user", "content": query}
    ]
})

# Extract final content from message history
final_answer = response["messages"][-1].content
print("\n=== FINAL ANSWER ===")
print(final_answer)

Execution Output Trace

When running this query, the agent splits the problem into distinct sub-queries rather than making a single unguided search:

[TOOL EXECUTION] retriever_tool called with query: 'patient name and doctor name'
[TOOL EXECUTION] retriever_tool called with query: 'doctor name'
[TOOL EXECUTION] retriever_tool called with query: 'patient name'

=== FINAL ANSWER ===
Based on the medical report context:
- Patient Name: Miss Nikita Choudhary
- Doctor Names: Dr. Nitin Nahar, Dr. A. K. Asthana, and Dr. R. M. Sharma

Why Did the Agent Query Multiple Times?

1. Initial Search: The agent first ran an exploratory query combining both intents: 'patient name and doctor name'.
2. Context Gap Detection: Reading the returned chunks, the agent recognized it had solid context for the patient, but the doctor references were incomplete or ambiguous across the top-$4$ results.
3. Targeted Sub-Queries: The reasoning loop triggered two targeted lookups: 'doctor name' and 'patient name'.
4. Final Synthesis: With both retrieved contexts in hand, the agent merged the facts into a clean, accurate answer.

A naive RAG pipeline would have stopped after the first retrieval, likely returning an incomplete doctor list.

7. Production Edge Cases & Architectural Gotchas

Deploying Agentic RAG into production introduces challenges not seen in standard pipelines:

Real-World Pitfalls

• Infinite Tool-Calling Loops: If the vector store does not contain the answer, an agent may repeatedly call the retriever with slightly different query strings.

  • Fix: Enforce an execution budget (e.g., max_iterations=5) and instruct the system prompt: "If information is missing after 2 retrieval attempts, declare it unavailable."

• Context Window Flooding: Each tool execution appends chunks to the conversation history. Three tool calls with $K = 4$ chunks can add 12,000+ characters of context.

  • Fix: Implement chunk deduplication, reduce top-$K$ for sub-queries ($K=2$), or use an intermediate summarization step before returning tool output to the agent.

• Tool Docstring Misdirection: If docstrings are ambiguous, the LLM may pass poorly formatted queries or invoke the wrong tool when multiple indices exist.

  • Fix: Clearly document the scope of each index:
    PYTHON

    Copy

    """
    Search specifically for internal HR benefits, leaves, and payroll. 
    DO NOT use this tool for customer-facing product pricing.
    """
    

8. Summary and Key Takeaways

• Standard RAG Fails on Complex Queries: Embedding a multi-part user query into a single vector often misses chunks for one or more sub-topics due to semantic smearing.
• Agentic RAG Decouples Search from Retrieval: Wrapping vector searches inside an LLM tool allows an agent to break queries down, call retrievers iteratively, and gather targeted context.
• Tool Docstrings Drive Routing: The function's __doc__ string acts as runtime documentation for the agent. Clear, descriptive docstrings directly improve retrieval routing.
• Dynamic Multi-Hop Search: When an agent detects missing or partial information, it can run follow-up searches targeting specific entities (e.g., isolating 'doctor name').
• Supports Multi-Vector Architectures: The agentic pattern easily scales across multiple vector stores (e.g., medical records vs. billing policies) by exposing each index as an independent tool.
• Budget Control is Essential: Production implementations must constrain iteration counts and context sizes to prevent runaway costs, token limits, and infinite reasoning loops.