Generative AI using LangChain: Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) is an architectural framework designed to bridge the gap between static foundational Large Language Models (LLMs) and dynamic, private, or domain-specific real-world data. By marrying classical Information Retrieval (IR) algorithms with generative neural networks, RAG dynamically injects external factual context directly into an LLM's prompt at inference time, effectively eliminating knowledge cutoffs, reducing hallucinations, and eliminating the high compute costs associated with continuous model fine-tuning.

1. Why Do We Need RAG? The Fundamental Limitations of Standard LLMs

Large Language Models (LLMs) are massive, transformer-based neural network architectures characterized by billions of parameters—the numerical weights ($\mathbf{W}$) and biases ($\mathbf{b}$) acquired during training.

Massive Web Corpus (Terabytes/Petabytes)
   |
   | Pre-training Stage (Unsupervised next-token prediction)
   v
Base LLM (e.g., 7B, 13B, 70B, 175B Parameters)
   |
   | Knowledge stored exclusively within weights
   v
Parametric Knowledge

When a user submits a query (a prompt), the LLM processes the sequence through its transformer layers to predict subsequent tokens, navigating its parametric knowledge. While this mechanism exhibits impressive reasoning and general factual recall, it fails systematically in three production scenarios:

1. Private Data Isolation: LLMs are pre-trained on publicly available internet data. They possess zero awareness of proprietary internal databases, corporate records, or paywalled resources (such as video transcripts and private learning management portals).
2. Static Knowledge Cut-Off: Every base model snapshot has a temporal boundary. An LLM trained up to January cannot answer queries about events occurring in February. Standard foundation models cannot spontaneously assimilate fresh, real-time news or intraday documentation updates.
3. Probabilistic Hallucinations: LLMs are autoregressive token predictors governed by probability distributions:

They do not verify facts against an immutable truth table. If a prompt falls into a low-probability or ambiguous state space within the weights, the model may fabricate plausibly sounding, completely false assertions with high confidence (e.g., asserting that Albert Einstein played professional football in his youth).

2. The Core Mental Model: Pre-Training vs. Fine-Tuning

To resolve these shortcomings, early generative AI architectures relied primarily on Fine-Tuning. To understand where fine-tuning succeeds and where it breaks down, consider the trajectory of an engineering student.

Real-Life Analogy: The Engineering Graduate

Imagine an undergraduate student who completes a four-year engineering degree. Over four years, the student studies a wide array of foundational subjects: mathematics, physics, computing basics, and communication. This broad, general-purpose schooling represents Pre-Training.

When the graduate is hired by an enterprise, the company does not immediately assign them to production mission-critical tasks. Instead, the company conducts a 2- to 3-month focused onboarding program. This corporate training does not re-teach basic arithmetic or general sentence structure; it teaches internal project architectures, codebase conventions, and specialized tooling. This onboarding process represents Fine-Tuning.

Fine-tuning adapts a base model by taking a pre-trained network and continuing gradient updates on a curated, domain-specific dataset.

3. Naive Fine-Tuning: Methodologies, Execution, and Production Bottlenecks

Fine-tuning primarily takes two distinct forms:

• Supervised Fine-Tuning (SFT): The model is trained on labeled pairs consisting of explicit inputs and target outputs:

where $x_i$ represents a task prompt and $y_i$ represents the gold-standard completion.

• Continued Pre-Training: Unsupervised token prediction performed over raw domain-specific corpora (e.g., medical textbooks, proprietary technical transcripts) without prompt-response pairing.

The Standard SFT Execution Flow

Data Collection
   | Curate 1,000 to 1,000,000 domain-specific prompt-output pairs
   v
Method Selection
   | Choose Full Parameter Tuning vs. PEFT (e.g., LoRA, QLoRA)
   v
Domain Training
   | Run backpropagation over low epoch counts to prevent catastrophic forgetting
   v
Evaluation & Safety
   | Benchmark exact matches, factuality, hallucination rates, and perplexity

The Architectural Pitfalls of Fine-Tuning for Factual Knowledge

While fine-tuning is an effective strategy for adjusting tone, style, output formatting (e.g., structured JSON), and task-specific reasoning behaviors, relying on it as a factual knowledge-storage mechanism introduces severe bottlenecks:

• Compute and Cost Extremes: Continuous backpropagation across 7B to 70B parameters requires enterprise-tier GPU clusters (e.g., NVIDIA H100s). Frequent retraining cycles quickly become commercially non-viable.
• Dynamic Data Inefficiency: If proprietary data changes hourly or daily (such as inventory catalogs, price sheets, or new tutorial modules), the fine-tuned model becomes instantly obsolete. Retraining the network for every minor update is impossible.
• Knowledge Deletion Impossibility: Removing a deprecated, recalled, or confidential document from the network's weights requires either complex machine unlearning techniques or a complete retrain from a clean checkpoint.
• Specialized Engineering Overhead: Setting up training clusters, managing learning rates, avoiding gradient explosions, and mitigating catastrophic forgetting requires specialized machine learning infrastructure engineers.

4. In-Context Learning: The Emergence of Non-Parametric Reasoning

To bypass the costs of gradient descent at runtime, modern generative architectures leverage In-Context Learning (ICL).

In-Context Learning (ICL): An emergent capability of modern foundational LLMs wherein the model learns to execute a novel task purely by processing context, instructions, and demonstration pairs supplied within the inference prompt—without undergoing any parameter updates or weight modifications.

The Emergent Property of Scale

In neural networks, an emergent property is an ability or capability that appears spontaneously once a system crosses a threshold of parameter count and dataset scale, despite never being explicitly engineered into smaller iterations.

In smaller architectures, tasks such as Named Entity Recognition (NER) or sentiment analysis required task-specific classifier heads or explicit fine-tuning. In large models ($\ge 175\text{B}$ parameters), providing 3 to 5 labeled exemplars within the context window enables the network's self-attention heads to infer the task mapping dynamically:

However, few-shot prompting teaches the model how to solve a task through stylistic mimicry; it does not reliably feed the model what facts to reference when dealing with deep proprietary data.

5. From Examples to Grounding: The Conceptual Birth of RAG

Rather than filling the model's finite context window with input-output demonstrations, an alternative strategy emerged: Dynamic Context Injection.

Concrete Scenario: The Video Course Assistant

Consider an online education platform hosting a two-hour technical lecture on Linear Regression. A student submits the query: "How do we perform the optimization step in gradient descent?"

Instead of fine-tuning an LLM on the video transcription, or sending an empty prompt that might trigger hallucinations, the system orchestrates a targeted prompt assembly:

Student Query
   | "How do we perform the optimization step in gradient descent?"
   v
Search Subsystem
   | Identifies the precise timestamp transcript: Minute 05:00 to Minute 25:00
   v
Prompt Synthesis (Augmentation)
   | Injects the 20-minute segment directly into the LLM prompt as reference material
   v
Final LLM Generation
   | Generates answer grounded strictly in the provided transcript text

This paradigm shift marks the boundary where simple in-context learning evolves into Retrieval Augmented Generation (RAG): enhancing a model's generation quality by dynamically retrieving non-parametric external knowledge at execution time.

6. High-Level RAG Architecture: Information Retrieval Meets Text Generation

RAG represents the structural marriage of two distinct branches of Computer Science:

1. Information Retrieval (IR): Classical search and semantic matching algorithms designed to fetch relevant document fragments from large corpora.
2. Text Generation: Autoregressive neural models capable of synthesizing human-grade technical language from raw inputs.

The standard production RAG pipeline executes across four sequential stages:

• Stage 1: Indexing (Offline preparation of external data into searchable vector representations)
• Stage 2: Retrieval (Runtime semantic search to isolate the best candidate chunks)
• Stage 3: Augmentation (Assembling the grounded prompt structure)
• Stage 4: Generation (Inference execution through the foundational model)

7. Step 1: Indexing — Transforming Unstructured Data into Vector Stores

Indexing is the offline batch preparation process that converts disparate, unstructured files into an optimized, search-ready internal knowledge base.

Source Documents (PDFs, Transcripts, Cloud S3, Web HTML)
   |
   | 1. Document Ingestion (via DocumentLoaders)
   v
Raw In-Memory Strings
   |
   | 2. Text Chunking (Chunk Size = N tokens, Overlap = M tokens)
   v
Uniform Text Chunks
   |
   | 3. Dense Vector Embeddings (via Embedding Model)
   v
High-Dimensional Vectors: [e_1, e_2, ..., e_d]
   |
   | 4. Vector Database Storage (Embedding + Raw Text + Metadata)
   v
Vector Database (Chroma, FAISS, Pinecone, Weaviate)

Detailed Pipeline Mechanics

1. Document Ingestion:

   • Extraction scripts (e.g., LangChain's PyPDFLoader, YoutubeAudioLoader, WebBaseLoader) connect to data silos (AWS S3, local storage, private databases, Google Drive).
   • Parses arbitrary binary file formats and extracts clean, unified text strings into working memory.

2. Text Chunking:

   • The Problem with Monolithic Ingestion: Passing an entire textbook or 2-hour transcript into a prompt will exceed practical context limits or introduce high inference latency. Furthermore, semantic retrieval degrades if the searched text units are too broad.
   • The Solution: Split the text into bite-sized, semantically cohesive chunks using tools like RecursiveCharacterTextSplitter or semantic chunkers. Chunks preserve bounded topics (e.g., 500–1000 tokens) with sliding overlaps (e.g., 100 tokens) to prevent abrupt semantic cutoff along paragraph boundaries.

3. Dense Vector Embeddings:

   • Each isolated text chunk is passed through a deep embedding model (e.g., text-embedding-3-small, HuggingFace SentenceTransformers).
   • The text is projected into a $d$-dimensional continuous vector space:

• In this vector space, geometric distance directly mirrors semantic similarity. Sentences discussing identical concepts with different vocabulary reside in close proximity.

4. Vector Store Persistence:
   • The vector $\mathbf{v}$, its source raw text string, and associated metadata (e.g., timestamp, URL, chapter ID) are indexed inside a specialized vector database.
   • Local / In-Memory Stores: FAISS, ChromaDB.
   • Cloud / Distributed Stores: Pinecone, Weaviate, Milvus, Qdrant.

8. Step 2: Retrieval — Query Embedding and Semantic Search

Once the knowledge index is built, the system remains idle until a user enters a query at runtime. The retrieval phase then executes a multi-step semantic lookup:

User Query: "How do we perform the optimization step in gradient descent?"
   |
   | Run Query through the EXACT SAME Embedding Model
   v
Query Vector (q in R^d)
   |
   | Compute Vector Proximity against Stored Database Vectors
   v
Similarity Scoring: Cosine Similarity / Dot Product
   |
   | Identify Top-k Closest Matches (e.g., k = 3)
   v
Top-k Text Chunks Fetched + Re-ranked

Retrieval Mathematics & Algorithms

1. Query Embedding: The incoming runtime question is transformed into a dense vector using the exact embedding model used during indexing:

2. Similarity Metrics: The vector engine calculates the distance between the query vector $\mathbf{q}$ and every indexed document vector $\mathbf{d}_i$. The most common metric is Cosine Similarity:

3. Advanced Retrieval Strategies:
   • Maximal Marginal Relevance (MMR): Balances relevance to the query with diversity among the retrieved chunks to eliminate repetitive, redundant context.
   • Contextual Compression: Trims non-relevant sentences from the extracted chunks before passing them to the prompt builder.
   • Cross-Encoder Re-Ranking: Takes the top-$N$ candidates and applies a deep cross-encoder to produce a high-precision, re-ordered top-$k$ set.

9. Steps 3 & 4: Augmentation and Generation — Assembling Grounded Prompts

Once the retrieval engine extracts the top-$k$ document fragments, execution moves to the Augmentation and Generation stages.

Complete Code Representation (LangChain Implementation Pattern)

The following Python snippet illustrates how Document Ingestion, Splitting, Vector Indexing, Retrieval, and Generation chains combine using LangChain:

import os
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# 1. INDEXING STAGE: Ingestion & Splitting
loader = TextLoader("linear_regression_lecture_transcript.txt")
raw_documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=100,
    separators=["\n\n", "\n", " ", ""]
)
chunked_docs = text_splitter.split_documents(raw_documents)

# 1. INDEXING STAGE: Embedding & Vector Storage
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = FAISS.from_documents(chunked_docs, embedding_model)

# 2. RETRIEVAL STAGE: Convert Store to Retriever
retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 2}
)

# 3. AUGMENTATION STAGE: Defining the Grounded Prompt Template
prompt_template = """You are an authoritative course assistant. 
Answer the question strictly using the provided context below.
If the information is missing from the context, state clearly: "I do not know based on the provided material."

Context:
{context}

Question:
{question}

Answer:"""

prompt = ChatPromptTemplate.from_template(prompt_template)
llm = ChatOpenAI(model="gpt-4o", temperature=0.0)

# Helper function to format retrieved documents
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# 4. GENERATION STAGE: LCEL (LangChain Expression Language) Pipeline
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Execution
query = "How do we perform the optimization step in gradient descent?"
response = rag_chain.invoke(query)
print(response)

10. Architectural Comparison: Naive Inference vs. Fine-Tuning vs. RAG

Understanding which architecture to deploy depends on the technical requirements of the use case:

11. Summary and Key Takeaways

• The Limitation of Parametric Memory: Foundational LLMs store knowledge entirely within their weights and biases. Consequently, they fail when queried about private corporate data, post-cutoff events, or niche facts, often generating hallucinations instead.
• The Fine-Tuning Bottleneck: Fine-tuning alters model parameters using task-specific datasets. While it excels at teaching structure, style, and domain jargon, it is computationally expensive, struggles with real-time updates, and cannot cleanly purge outdated facts.
• In-Context Learning as a Foundation: Discovered at scale in models like GPT-3 (175B), In-Context Learning (ICL) allows models to adapt using dynamic prompt context alone, without any gradient updates.
• The Core Concept of RAG: RAG builds on ICL by swapping generic prompt demonstrations for precise, dynamically retrieved domain context fetched from an external source.
• Four Core Pipeline Steps:
  1. Indexing: Parse, split, embed, and store raw text documents in a high-dimensional vector database.
  2. Retrieval: Embed incoming user queries to fetch the top-$k$ most semantically relevant text fragments using similarity metrics such as Cosine Similarity.
  3. Augmentation: Merge the retrieved text fragments and the original user query into a unified, strictly bounded prompt.
  4. Generation: Pass the prompt to an LLM to synthesize a grounded, accurate response without relying on internal parametric guesses.
• Grounded Verification: RAG mitigates hallucinations by explicitly instructing the model to answer using only the supplied context and to output "I do not know" if the context is insufficient.
• Operational Agility: Updating a RAG system requires only adding or removing documents from the vector database—completely avoiding the expensive retraining cycles inherent to fine-tuning.