Building a Corrective RAG | Integrating Web Search as a Fallback Knowledge Base

Corrective Retrieval-Augmented Generation (CRAG) introduces a self-correcting evaluation mechanism to traditional RAG architectures. By evaluating retrieved documents with a lightweight grader and dynamically switching between internal knowledge refinement and external web search fallback, CRAG prevents LLM hallucinations and gracefully handles out-of-domain queries.

1. Why Corrective RAG? The Limits of Static Knowledge Bases

Traditional Retrieval-Augmented Generation (RAG) pipelines operate under a fragile assumption: that the static vector store contains the necessary, context-accurate information to answer any incoming user query. When this assumption fails, standard architectures exhibit severe failure modes:

• Context Window Overburden & Noise: Retriever nodes pull top-$K$ chunks based solely on raw vector similarity. If the vector store contains poor or noisy matches, irrelevant text floods the LLM context window.
• LLM Hallucination on Irrelevant Context: When forced to synthesize an answer from irrelevant or out-of-domain documents, Large Language Models frequently hallucinate plausible-sounding but completely fabricated facts.
• Static Corpus Limitations: Private or embedded knowledge bases $C$ are static. They cannot answer queries regarding dynamic, external, or non-indexed domains.

Corrective RAG mitigates these failure modes by inserting a retrieval evaluator between the retrieval and generation phases. If internal retrieval fails to yield relevant information, CRAG pivots its strategy, rewrites the search query, and fetches real-time external data via web search.

2. Mathematical and Structural Foundations

Formally, a CRAG system accepts an input query $X$ and operates over an accessible knowledge corpus $C = \{D_1, D_2, \dots, D_n\}$. The overall framework decomposes into three core components: a Retriever ($R$), an Evaluator ($E$), and a Generator ($G$).

Input Query: X
Corpus: C = {D_1, D_2, ..., D_n}

Retrieval Phase

The retriever $R$ fetches the top-$K$ relevant documents from $C$ based on semantic similarity:

Evaluation Phase

To prevent retrieval bias, each document $D_{Ri} \in R(X, C)$ is evaluated independently by the evaluator $E$. The evaluator computes a scalar confidence or relevance score $S_i$:

Based on aggregate document scores, the system assigns a global confidence status $\mathcal{C}$:

Generation Phase

The generator $G$ receives query $X$ alongside a dynamically constructed knowledge base $K$:

Where $K$ is defined conditionally as:

3. The CRAG Tripartite Process

The CRAG architecture processes documents across three strategic routes based on evaluation confidence:

1. The Correct Route

• Condition: The retrieved documents $D_{Ri}$ score high relevance against query $X$.
• Action: Trigger Internal Knowledge Refinement. The documents are partitioned into sentence-level strips, filtered for fine-grained key facts, and compiled into internal knowledge base $K_{in}$.
• Generation: $Y = G(X, K_{in})$.

2. The Incorrect Route

• Condition: All retrieved documents fall below the minimum relevance threshold ($\mathcal{C} = \text{Incorrect}$).
• Action: Trigger External Knowledge Search. The system discards internal retrieval, transforms query $X$ into an optimized web-search query $X'$, executes a web search, and builds external knowledge base $K_{ex}$.
• Generation: $Y = G(X, K_{ex})$.

3. The Ambiguous Route

• Condition: The evaluator returns intermediate or mixed confidence scores across documents.
• Action: Trigger a Hybrid Strategy. The system retains positive internal strips ($K_{in}$) while simultaneously triggering query transformation and web search ($K_{ex}$).
• Generation: $Y = G(X, K_{in} \cup K_{ex})$.

4. Internal Knowledge Refinement: Decomposing Documents into Strips

A core contribution of the CRAG research paper is replacing coarse document-level context injection with granular sentence-level Knowledge Refinement.

Creator's Intuition: Providing an entire 1,000-word retrieved document to an LLM introduces background noise. If only two sentences inside that document contain the actual answer, we must decompose the text into small "strips" (1–2 sentences) and evaluate each strip independently to eliminate context bias.

1. Decomposition: Every retrieved document $D_{Ri}$ is segmented into fine-grained units $S = \{s_1, s_2, \dots, s_m\}$, where each strip $s_j$ consists of 1 to 2 sentences.
2. Strip-Level Scoring: The evaluator calculates relevance scores $E(X, s_j)$ for every strip independently.
3. Recomposition: Non-relevant strips are purged. Highly relevant strips are reassembled in sequence to form $K_{in}$.

5. Web Search Fallback and Query Transformation Mechanics

When internal retrieval fails, submitting the raw user query directly to a search engine often yields poor results because conversational user queries differ fundamentally from search engine retrieval syntax.

Query Rewriting Logic

The system passes query $X$ through a dedicated Question Rewriter LLM Node running a specialized prompt to extract the underlying semantic intent:

Search Execution & External Base Construction

1. The rewritten query $X'$ is dispatched to a Search API (e.g., Tavly Search).
2. The top web pages are scraped, cleaned, and converted into unstructured text chunks.
3. These web chunks replace the internal retrieval context in the shared system state, forming $K_{ex}$.

6. Graph Flow Architecture & Decision Routing Logic

Using LangGraph, CRAG is structured as a stateful directed acyclic graph (DAG). State parameters flow across stateful nodes while conditional edges govern workflow transitions.

7. Python Implementation: State Definition and Setup

Below is the foundational Python setup, including imports, system state definitions using TypedDict, and vector store initialization using LangChain components.

import os
from typing import List, TypedDict, Literal
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field

# 1. Define Shared Graph State
class GraphState(TypedDict):
    question: str
    response: str
    vector_store: Chroma
    relevant_documents: List[Document]
    model: ChatOpenAI

# 2. Knowledge Base Source URLs
URLS = [
    "https://newsletter.alexmerced.com/p/what-is-word-embedding-in-deep",
    "https://newsletter.alexmerced.com/p/what-is-backpropagation-in-deep"
]

def build_vector_store() -> Chroma:
    """Loads web content, generates embeddings, and constructs Chroma VectorStore."""
    docs = []
    for url in URLS:
        loader = WebBaseLoader(url)
        docs.extend(loader.load())
    
    embeddings = OpenAIEmbeddings()
    vector_store = Chroma.from_documents(documents=docs, embedding=embeddings)
    return vector_store

8. Python Implementation: Document Grading Node

The evaluator node enforces structured output from the LLM via Pydantic schema validation. Each document chunk is evaluated independently against the user query.

# Pydantic Schema for Structured LLM Output
class GradeDocument(BaseModel):
    """Binary score for document relevance check."""
    binary_score: Literal["yes", "no"] = Field(
        description="Document is relevant to the question ('yes' or 'no')"
    )

def filter_documents(state: GraphState) -> GraphState:
    """
    Grades each retrieved document independently.
    Filters out non-relevant documents to formulate K_in.
    """
    question = state["question"]
    documents = state["relevant_documents"]
    model = state["model"]

    # Enforce structured output on the model
    structured_grader = model.with_structured_output(GradeDocument)

    system_prompt = (
        "You are a grader assessing relevance of a retrieved document to a user question.\n"
        "If the document contains keywords or semantic meaning related to the question, "
        "grade it as relevant. Return 'yes' or 'no'."
    )
    
    grade_prompt = ChatPromptTemplate.from_messages([
        ("system", system_prompt),
        ("human", "Retrieved document:\n\n{document}\n\nUser question: {question}")
    ])

    evaluator_chain = grade_prompt | structured_grader
    filtered_docs = []

    for doc in documents:
        result = evaluator_chain.invoke({"document": doc.page_content, "question": question})
        if result.binary_score.lower() == "yes":
            print("---GRADE: DOCUMENT RELEVANT---")
            filtered_docs.append(doc)
        else:
            print("---GRADE: DOCUMENT NOT RELEVANT---")

    state["relevant_documents"] = filtered_docs
    return state

9. Python Implementation: Query Rewriter and Web Search Integration

If all documents are graded as non-relevant (binary_score == "no"), the system executes query transformation followed by a search search engine query via the TavlySearchResults tool.

from langchain_community.tools import TavlySearchResults

def transform_query(state: GraphState) -> GraphState:
    """
    Transforms the user query into an optimized web-search query.
    """
    print("---TRANSFORMING QUERY FOR WEB SEARCH---")
    question = state["question"]
    model = state["model"]

    rewrite_prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a question rewriter that converts an input question "
                   "into a better version optimized for web search. "
                   "Analyze the input and extract its semantic intent."),
        ("human", "Initial Question: {question}\nProvide only the rewritten query.")
    ])

    rewriter_chain = rewrite_prompt | model
    response = rewriter_chain.invoke({"question": question})
    better_question = response.content.strip()

    print(f"Transformed Query: '{better_question}'")
    state["question"] = better_question
    return state


def web_search(state: GraphState) -> GraphState:
    """
    Executes external web search using Tavly API and builds K_ex context.
    """
    print("---EXECUTING EXTERNAL WEB SEARCH---")
    question = state["question"]
    
    web_search_tool = TavlySearchResults(k=3)
    search_results = web_search_tool.invoke({"query": question})

    # Join top search result contents into unified Document context
    search_content = "\n\n".join([res["content"] for res in search_results])
    external_doc = Document(page_content=search_content)

    # Replace internal knowledge base with external search findings (K_ex)
    state["relevant_documents"] = [external_doc]
    return state

10. Constructing and Compiling the LangGraph Workflow

This section links all defined nodes and conditional routers into a executable StateGraph.

from langgraph.graph import StateGraph, START, END

def decide_to_generate(state: GraphState) -> Literal["generate", "transform_query"]:
    """
    Conditional edge router evaluating the presence of relevant documents.
    """
    if len(state["relevant_documents"]) > 0:
        print("---DECISION: GENERATE FROM INTERNAL KNOWLEDGE (K_in)---")
        return "generate"
    else:
        print("---DECISION: NO RELEVANT DOCS. FALLBACK TO WEB SEARCH (K_ex)---")
        return "transform_query"


def generate_answer(state: GraphState) -> GraphState:
    """
    Generates final output response using provided context (K_in or K_ex).
    """
    print("---GENERATING FINAL ANSWER---")
    question = state["question"]
    documents = state["relevant_documents"]
    model = state["model"]

    prompt = ChatPromptTemplate.from_messages([
        ("system", "Answer the question based strictly on the provided context:\n\n{context}"),
        ("human", "Question: {question}")
    ])

    context_str = "\n\n".join([d.page_content for d in documents])
    generator_chain = prompt | model
    response = generator_chain.invoke({"context": context_str, "question": question})

    state["response"] = response.content
    return state


# Assemble Graph
workflow = StateGraph(GraphState)

# Add Nodes
workflow.add_node("filter_documents", filter_documents)
workflow.add_node("transform_query", transform_query)
workflow.add_node("web_search", web_search)
workflow.add_node("generate_answer", generate_answer)

# Connect Edges
workflow.add_edge(START, "filter_documents")

# Add Conditional Route based on Evaluator Output
workflow.add_conditional_edges(
    "filter_documents",
    decide_to_generate,
    {
        "generate": "generate_answer",
        "transform_query": "transform_query"
    }
)

workflow.add_edge("transform_query", "web_search")
workflow.add_edge("web_search", "generate_answer")
workflow.add_edge("generate_answer", END)

# Compile Graph
app = workflow.compile()

11. Execution Walkthrough: In-Domain vs. Out-of-Domain Scenarios

To demonstrate system behavior, we trace two separate queries against a vector store pre-loaded with articles on Word Embeddings and Backpropagation.

Scenario A: In-Domain Query Execution

• User Query: "What are word embeddings and how do they work?"

Scenario B: Out-of-Domain Query Execution (Web Fallback)

• User Query: "Explain binary heap"

12. Comparative Synthesis: Standard RAG vs. Self-RAG vs. Corrective RAG

Summary and Key Takeaways

1. Self-Correction Architecture: Corrective RAG (CRAG) introduces a dedicated evaluator node that calculates document relevance prior to generation, preventing hallucination caused by bad retrievals.
2. Dual-Knowledge Pivot: CRAG dynamically switches between an internal knowledge base ($K_{in}$) and an external dynamic web knowledge base ($K_{ex}$) based on document grading results.
3. Independent Strip Refinement: Raw documents are decomposed into fine-grained 1-to-2 sentence strips and graded independently to remove irrelevant contextual noise.
4. Three Execution Branches: CRAG routes execution through Correct (use $K_{in}$), Incorrect (rewrite query and use $K_{ex}$), or Ambiguous (combine $K_{in} \cup K_{ex}$) paths based on threshold boundaries.
5. Semantic Query Rewriting: Raw queries that trigger the external fallback path are transformed by an LLM node into search-optimized queries before executing web search.
6. Integration with LangGraph: CRAG's stateful cyclic routing and conditional logic are naturally modeled using LangGraph state chains, enabling deterministic runtime control.
7. Production Fault Tolerance: By combining internal static vector search with dynamic Tavly web search, CRAG reliably resolves both domain-specific private queries and generic dynamic queries within a unified pipeline.