Agentic AI: Building a RAG-Powered Multi-Utility Chatbot in LangGraph
Retrieval-Augmented Generation (RAG) empowers large language models (LLMs) to query external, dynamic, and private datasets without costly retraining. By encapsulating RAG as an executable tool within a LangGraph state graph, we transform a basic conversational interface into a production-ready, multi-utility AI agent capable of handling real-time tools, private document Q&A, and persistent conversation workflows.
1. Evolution of the Multi-Utility Chatbot
Across this learning trajectory, our running chatbot architecture has evolved incrementally. Rather than rebuilding the agent from scratch for every feature, each step builds modularly on top of previous state capabilities:
By adding RAG capabilities, the agent evolves into a Multi-Utility Chatbot. It can natively decide whether to handle a query via standard conversational parametric memory, invoke an external API tool (e.g., live stock prices), or query user-uploaded document stores (e.g., custom PDFs, e-books, or corporate expense reports).
2. Why Do We Need RAG? The Three Core Limitations of LLMs
While modern LLMs display impressive reasoning capabilities, relying solely on their pre-trained weights introduces three distinct structural failures in production applications:
3. Core Mental Model: In-Context Learning & Context Filtering
RAG operates fundamentally on the principle of In-Context Learning. Instead of attempting to update the model’s internal parametric knowledge (weights), we inject explicit information into the prompt as non-parametric knowledge (context).
The Context Window Bottleneck
When a user asks a question about a personal dataset (e.g., an internal expense report or a 500-page textbook), the simplest naive solution would be pasting the entire document directly into the LLM prompt alongside the query.
An LLM's context window is strictly bounded by token limits. Dumping massive documents, company codebases, or full e-book libraries into a single prompt triggers performance degradation, extreme token costs, or immediate rate-limit failures.
The Solution: Targeted Context Filtering
RAG solves this context-window overflow by performing Context Filtering. Instead of feeding the entire context, a search subsystem retrieves only the top- most semantically relevant passages matching the user query, appending just those target chunks to the prompt context.
4. End-to-End RAG Architecture & Ingestion Pipeline
A production-grade RAG system operates in two distinct phases: Ingestion (offline/indexing phase) and Retrieval & Generation (online query phase).
5. Architectural Design: RAG as a Tool in LangGraph
In complex agentic applications, building RAG as a monolithic graph node limits flexibility. The cleanest design pattern in LangGraph is wrapping the RAG retriever inside an Executable Tool (@tool).
By treating RAG as a tool:
- The agent's LLM routing node dynamically decides when to execute document retrieval versus when to answer directly or call other tools.
- The RAG pipeline seamless coexists alongside other functional tools (e.g., search tools, math calculators, database queries).
- The graph structure remains clean, modular, and easy to trace.
6. Standalone Implementation: Vector Store & Ingestion Setup
Below is the standalone Python code (rag.ipynb) demonstrating how to ingest a document, generate vector embeddings, store them in a FAISS vector store, and construct a retriever.
Prerequisites & Package Installation
pip install --upgrade langchain langchain-openai langchain-community faiss-cpu pypdf langgraph
Document Loading, Chunking, and Indexing
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
# Load environment variables (OPENAI_API_KEY, LANGCHAIN_API_KEY, etc.)
load_dotenv()
# Initialize LLM
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
# Step 1: Load Document
pdf_path = "intro_to_ml.pdf"
loader = PyPDFLoader(pdf_path)
docs = loader.load()
# Step 2: Split Document into Chunks
# Use chunk_overlap to maintain semantic context across chunk boundaries
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = text_splitter.split_documents(docs)
# Step 3: Embed Chunks and Index in Vector Store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(documents=chunks, embedding=embeddings)
# Step 4: Expose Vector Store as a Retriever
# Set search_type="similarity" to retrieve top k=4 relevant chunks
retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": 4}
)
7. Defining the Custom RAG Tool & LLM Binding
With the retriever initialized, we wrap its invocation inside a LangChain @tool function. We format the function signature and docstrings clearly so the LLM knows precisely when and how to invoke it.
from langchain_core.tools import tool
@tool
def rag_tool(query: str) -> dict:
"""
Search and retrieve relevant context from the uploaded PDF document.
Use this tool whenever the user asks questions about the contents of the document.
"""
# Invoke vector store retriever
retrieved_docs = retriever.invoke(query)
context_list = []
metadata_list = []
# Parse individual retrieved Document objects
for doc in retrieved_docs:
context_list.append(doc.page_content)
metadata_list.append(doc.metadata)
return {
"query": query,
"context": context_list,
"metadata": metadata_list
}
# Bind tool with LLM
tools = [rag_tool]
llm_with_tools = llm.bind_tools(tools)
8. Building and Executing the LangGraph Agent
We construct a execution state graph with two core nodes: a chat_node (invoking the LLM) and a tool_node (executing tool calls). Conditional routing logic automatically directs the control flow based on whether the LLM requests a tool execution.
from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition
from langchain_core.messages import SystemMessage, HumanMessage
# Define State Structure
class AgentState(TypedDict):
messages: Annotated[list, add_messages]
# Define Chat Node
def chat_node(state: AgentState):
messages = state["messages"]
response = llm_with_tools.invoke(messages)
return {"messages": [response]}
# Define Graph Workflow
builder = StateGraph(AgentState)
# Add Nodes
builder.add_node("chat_node", chat_node)
builder.add_node("tools", ToolNode(tools))
# Add Edges
builder.add_edge(START, "chat_node")
# Conditional Edge: Checks if LLM generated a tool call
builder.add_conditional_edges(
"chat_node",
tools_condition,
path_map={"tools": "tools", END: END}
)
# Route execution back to chat_node after executing tool
builder.add_edge("tools", "chat_node")
# Compile Graph
graph = builder.compile()
Sample Query Execution
# Query requiring document retrieval
query = "Using the PDF notes explain how to find the ideal value of K in K Nearest Neighbor."
inputs = {"messages": [HumanMessage(content=query)]}
result = graph.invoke(inputs)
# Print final synthesized response
print(result["messages"][-1].content)
9. Observability Walkthrough: Deep-Diving with LangSmith
By linking the workflow to LangSmith, we gain complete visibility into how the agent routes messages, fires tool calls, retrieves vectors, and constructs final outputs.
10. End-to-End System Integration: Dynamic Frontend & Production Backend
To convert this prototype into a full multi-utility application, we separate the system into a dynamic modular backend (langgraph_rag_backend.py) capable of parsing uploaded PDFs on-the-fly and a responsive Streamlit frontend (streamlit_rag_frontend.py).
System Architecture
Backend Implementation (langgraph_rag_backend.py)
import os
from typing import TypedDict, Annotated
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_core.tools import tool
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition
load_dotenv()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
retriever = None # Global retriever instance managed dynamically
def ingest_pdf(file_path: str):
"""Dynamically ingests a user-uploaded PDF into a local FAISS index."""
global retriever
loader = PyPDFLoader(file_path)
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(docs)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = FAISS.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_type="similarity", search_kwargs={"k": 4})
return True
@tool
def rag_tool(query: str) -> str:
"""Queries the dynamically uploaded document for context."""
global retriever
if retriever is None:
return "No document has been uploaded or indexed yet."
docs = retriever.invoke(query)
context = "\n\n".join([f"[Source Page {d.metadata.get('page', 'N/A')}]: {d.page_content}" for d in docs])
return context
# Additional tools (e.g., web search, math, stock price) can be appended here
tools = [rag_tool]
llm_with_tools = llm.bind_tools(tools)
class AgentState(TypedDict):
messages: Annotated[list, add_messages]
def chat_node(state: AgentState):
return {"messages": [llm_with_tools.invoke(state["messages"])]}
builder = StateGraph(AgentState)
builder.add_node("chat_node", chat_node)
builder.add_node("tools", ToolNode(tools))
builder.add_edge(START, "chat_node")
builder.add_conditional_edges("chat_node", tools_condition, {"tools": "tools", END: END})
builder.add_edge("tools", "chat_node")
graph = builder.compile()
Frontend Implementation (streamlit_rag_frontend.py)
import streamlit as st
import os
import tempfile
from langchain_core.messages import HumanMessage, AIMessage
from langgraph_rag_backend import graph, ingest_pdf
st.set_page_config(page_title="Multi-Utility RAG Chatbot", layout="wide")
st.title("Multi-Utility AI Chatbot")
# Sidebar for PDF Upload and Configuration
with st.sidebar:
st.header("Document Knowledge Base")
uploaded_file = st.file_uploader("Upload a PDF document", type=["pdf"])
if uploaded_file is not None:
if "file_processed" not in st.session_state or st.session_state.file_processed != uploaded_file.name:
with st.spinner("Indexing uploaded document..."):
# Save uploaded file to a temporary location
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp_file:
tmp_file.write(uploaded_file.getvalue())
tmp_file_path = tmp_file.name
# Trigger Backend Ingestion Pipeline
ingest_pdf(tmp_file_path)
os.remove(tmp_file_path) # Clean up temp file
st.session_state.file_processed = uploaded_file.name
st.success("Document successfully processed and indexed!")
# Initialize Chat History
if "messages" not in st.session_state:
st.session_state.messages = []
# Display Chat Messages
for msg in st.session_state.messages:
role = "user" if isinstance(msg, HumanMessage) else "assistant"
with st.chat_message(role):
st.write(msg.content)
# User Query Input
if user_input := st.chat_input("Ask a question about your document or general queries:"):
st.session_state.messages.append(HumanMessage(content=user_input))
with st.chat_message("user"):
st.write(user_input)
with st.chat_message("assistant"):
with st.spinner("Thinking & Retrieving..."):
response = graph.invoke({"messages": st.session_state.messages})
final_reply = response["messages"][-1].content
st.write(final_reply)
st.session_state.messages.append(AIMessage(content=final_reply))
11. Edge Cases, Pitfalls & Production Debugging
When deploying RAG tools within agentic architectures, several failure modes frequently emerge:
1. The Dynamic Retriever NoneType Edge Case
- Issue: A user asks a question about an uploaded document before actually uploading a PDF file via the UI.
- Mitigation: Implement robust defensive checks inside the
@toolimplementation returning explicit instruction text (e.g.,"No document indexed. Ask user to upload a PDF.") rather than throwing an unhandled python exception.
2. High Search Latency & Overhead
- Issue: Vector retrieval over large indexes introduces noticeable system delays (e.g., per turn).
- Mitigation:
- Restrict retriever depth ( or ).
- Utilize lighter, fast embedding models (e.g.,
text-embedding-3-small). - Keep vector indexes persistent or cached in memory via FAISS/Chroma rather than re-creating embeddings on every message turn.
3. Chunk Splitting Context Fracture
- Issue: Overly small
chunk_sizevalues fragment critical sentences, splitting a formula or code block in half. Overly largechunk_sizevalues degrade vector search precision. - Mitigation: Use
RecursiveCharacterTextSplitterwith balanced parameters (e.g.,chunk_size=1000,chunk_overlap=200) to guarantee continuous overlap across boundaries.
Summary and Key Takeaways
- Multi-Utility Chatbot Paradigm: Adding RAG upgrades a standard chatbot into a multi-utility agent capable of querying dynamic, uploaded, or private datasets alongside normal API tools.
- RAG Solves Core LLM Vulnerabilities: RAG overcomes LLM cut-off dates, protects data privacy, and minimizes hallucinations by grounding model responses in explicitly retrieved context.
- In-Context Learning Principle: RAG injects non-parametric retrieved data into the LLM prompt rather than fine-tuning or modifying underlying model parameters.
- Context Filtering: Instead of blowing past context window token limits, RAG filters large document collections down to the top- most semantically relevant text chunks.
- RAG as a LangGraph Tool: Decorating the RAG retriever as a LangChain
@toolprovides the cleanest architecture in LangGraph, enabling the LLM agent to selectively trigger document retrieval alongside other tools. - Ingestion Pipeline: Ingestion follows a strict progression: Document Loading Text Chunking Vector Embedding Generation Vector Indexing (FAISS) Retriever Object Creation.
- Observability with LangSmith: LangSmith traces illustrate the precise flow of execution: User Prompt Chat Node tool call decision Tool Node FAISS retrieval Chat Node final grounded response synthesis.