Chapter 12: What is RAG (Retrieval-Augmented Generation)?

Large Language Models (LLMs) like ChatGPT, Claude, and Gemini are trained on vast amounts of publicly available internet data. However, they lack access to private enterprise codebases, personal records, and confidential business documents. Retrieval-Augmented Generation (RAG) bridges this gap by dynamically retrieving relevant internal information and injecting it into the LLM's prompt context at runtime, allowing the model to answer proprietary questions accurately without hallucinating.

1. The Fundamental Limitation of Base LLMs

Base LLMs are static snapshots of the public internet. While they possess vast general knowledge, they fail when queried about non-public, proprietary, or highly specific personal information.

Public vs. Private Knowledge

• Public Knowledge: Questions like "How many subscribers does a major YouTube channel have?" can be answered immediately by an LLM because that data was scraped from public web pages during pre-training.
• Private/Personal Knowledge: Questions regarding private personal details, undisclosed financial data, or internal company workflows cannot be answered because that data was never part of the training set.

The Hallucination Danger

When queried about public facts, LLMs generate accurate responses. However, when asked about private information, an LLM rarely admits that it does not know. Instead, it predicts plausible-sounding words and guesses the answer—a behavior known as Hallucination.

Note: Base LLMs are non-deterministic autocomplete engines optimized for plausibility, not truth. Without explicit internal data provided at runtime, an LLM will confidently invent facts rather than return a blank answer.

2. The Enterprise Dilemma: Why LLMs Failed in Production (2022)

When ChatGPT launched in late 2022, software enterprises attempted to integrate LLMs into their developer workflows and internal support systems. They quickly ran into severe operational roadblocks.

Why Standard LLMs Fail Enterprise Workflows

1. Public Code vs. Proprietary Code: An LLM can write a perfect Binary Search algorithm in C++ or Python because millions of implementations exist on public GitHub repositories. However, it knows nothing about how proprietary software architectures (e.g., internal services at Akamai, Amazon, or Google) are built.
2. Confidentiality Safeguards: Companies do not publish their internal code, API specifications, software design docs, or bug tickets on the public internet.
3. Dangerous Hallucinations in Production: When developers asked LLMs to debug company-specific errors, the models invented plausible-sounding architectural components that did not exist, wasting hundreds of engineering hours.

3. What is RAG? (Retrieval-Augmented Generation)

RAG stands for Retrieval-Augmented Generation. Despite its complex name, the core mental model is remarkably straightforward: Give the LLM an open-book test.

Deconstructing the Acronym

• R - Retrieval: Fetching relevant bits of factual information from a target private knowledge source (PDFs, Databases, JSON, Markdown).
• A - Augmented: Adding the retrieved private information into the user's prompt as explicit context.
• G - Generation: Allowing the LLM to process the context and generate a natural language response strictly grounded in that retrieved knowledge.

Real-Life Analogy: The Job Recruiter Test

Imagine asking an LLM: "Will I get a job next month?"

The LLM has no idea who you are, so it cannot answer accurately. However, if you attach your Resume PDF to the prompt, the LLM reads your specific experience, skills, and education, and gives an accurate assessment.

Your Resume is the retrieved context that augments the LLM's generation.

4. Evolution of Context Ingestion: Naive Approaches vs. Reality

Before modern RAG architectures were standardized, engineers experimented with simpler methods to feed internal knowledge into LLMs.

Solution Comparison Matrix

The Token Cost Pitfall of Full Document Ingestion

Enterprise knowledge bases accumulate over decades—encompassing software design documents, architectural decisions, testing guidelines, and API manuals across millions of lines of text.

If an LLM reads all enterprise documentation on every user query, token usage skyrockets. A company's monthly LLM API bill would rapidly exceed its operating revenue.

5. First-Iteration RAG Architecture Breakdown

The first generation of RAG relied on structured knowledge sources and explicit keyword-matching retrieval scripts.

The 3-Step Execution Pipeline

1. Knowledge Base Setup: Information is organized into a searchable storage format (e.g., Python dictionary, JSON file, database).
2. Retrieval Step: An external Python script intercepts the user's question, scans the text for specific keywords, and extracts matching factual strings.
3. Augmented Prompt Generation: The extracted text is injected into the LLM's System Prompt alongside a strict directive: "Answer ONLY based on the provided context. Do not hallucinate."

6. Hands-On Implementation: First-Iteration RAG in Python

Below is a complete implementation demonstrating the foundational mechanics of a First-Iteration RAG pipeline using Python and the groq SDK.

import os
from groq import Groq

# Initialize Groq Client
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
MODEL_NAME = "llama-3.3-70b-versatile"

# ---------------------------------------------------------
# STEP 1: Define Private Knowledge Base
# ---------------------------------------------------------
knowledge_base = {
    "age": "The age of Pratyush is 25 years.",
    "networth": "The networth of Pratyush is 2000 INR."
}

# ---------------------------------------------------------
# STEP 2: Custom Keyword Retrieval Engine
# ---------------------------------------------------------
def retrieve_info(question: str) -> str:
    """
    Parses user question and retrieves relevant string from Knowledge Base.
    """
    cleaned_question = question.lower()
    
    if "age" in cleaned_question:
        return knowledge_base["age"]
    elif "networth" in cleaned_question:
        return knowledge_base["networth"]
    else:
        return "No relevant context found."

# ---------------------------------------------------------
# STEP 3: Augmented LLM Query Function
# ---------------------------------------------------------
def ask_rag_llm(question: str) -> str:
    # 1. Retrieve relevant context
    context = retrieve_info(question)
    
    # 2. Build constrained system prompt
    system_prompt = (
        f"Answer the user query in one line only. "
        f"Answer ONLY based on this context: [{context}]. "
        f"Do not use outside knowledge. Do not hallucinate."
    )
    
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": question}
    ]
    
    # 3. Call LLM
    response = client.chat.completions.create(
        model=MODEL_NAME,
        messages=messages
    )
    
    return response.choices[0].message.content

# ---------------------------------------------------------
# STEP 4: Execution & Testing
# ---------------------------------------------------------
if __name__ == "__main__":
    # Test Case 1: Query with Exact Keyword
    q1 = "What is Pratyush's age?"
    print(f"Query: {q1}")
    print(f"Response: {ask_rag_llm(q1)}\n")
    
    # Test Case 2: Query for Private Net Worth
    q2 = "What is Pratyush's networth?"
    print(f"Query: {q2}")
    print(f"Response: {ask_rag_llm(q2)}")

7. Fatal Pitfalls of Keyword-Based Retrieval

While First-Iteration RAG demonstrates the core concept, exact keyword matching is far too brittle for real-world enterprise applications.

Core Failure Modes of Naive Keyword RAG

1. Synonyms & Semantic Variations Fail

• Query: "How old is Pratyush?"
• Knowledge Base Key: "age"
• Result: The retriever looks for the substring "age". Because the user said "old", the software fails to retrieve the facts, and the LLM reports no information available.

2. Sensitivity to Typos

• Query: "What is Pratyush's agge?"
• Result: Simple string matching fails completely on minor spelling mistakes.

3. False Positive Retrieval

• Query: "What is age?"
• Result: The word "age" is detected, so the retriever blindly injects "The age of Pratyush is 25 years." into the prompt, returning a personal answer when a generic definition was requested.

4. Unmanageable Combinatorial Complexity

Attempting to patch keyword matching using if/else logic, Regex, or hardcoded synonym dictionaries requires infinite conditional rules:

# The Naive Patching Trap (Anti-Pattern)
if "age" in question or "old" in question or "years" in question or "dob" in question:
    return knowledge_base["age"]

Writing manual checks for every variation of human phrasing is impossible in production.

8. The Path to Modern RAG: Vector DBs and Semantic Search

To overcome the rigidity of exact string matching, modern AI engineering replaces keyword lookup with Semantic Vector Search.

What Changes in Modern Enterprise RAG?

1. Vector Embeddings: Text is converted into mathematical vectors representing high-dimensional semantic meaning. "How old are you?" and "What is your age?" produce nearly identical vector coordinates.
2. Chunking Strategies: Large documents are sliced into bite-sized semantic blocks rather than passed as raw multi-megabyte files.
3. Vector Databases (Vector DBs): Specialized stores (such as Qdrant, Pinecone, or Milvus) store embeddings and perform sub-millisecond similarity searches across billions of text chunks.

Summary and Key Takeaways

• RAG Solves LLM Blindspots: Base LLMs excel at general internet facts but know nothing about private enterprise codebases or confidential records. RAG injects proprietary context at query time.
• Prevention of Hallucinations: Constraining the system prompt with retrieved context prevents the model from inventing false answers to enterprise queries.
• System Prompts & Full Docs Don't Scale: Hardcoding facts in system prompts works only for tiny snippets; uploading entire document libraries causes massive token cost overruns and context window limits.
• The RAG Mental Model: Treat RAG as an open-book exam where a lightweight retrieval software finds the exact page needed and hands it to the LLM to write the response.
• Keyword Retrieval is Too Rigid: First-iteration RAG relying on string matching (if "age" in question) breaks on typos, synonyms, and phrasing variations (e.g., failing on "How old is Pratyush?").
• Modern RAG Relies on Semantics: Production-grade RAG uses Text Embeddings, Chunking, and Vector Databases (e.g., Qdrant) to match queries based on semantic meaning rather than exact word overlaps.