Evaluating RAG: Testing the Generator & Full Pipeline with the RAG Triad

Retrieval-Augmented Generation (RAG) applications must be evaluated using a decoupled, layered approach rather than treating the architecture as an opaque black box. By breaking testing down across Component Level (isolated Retriever and Generator), Pipeline Level (the interconnected RAG Triad), and Application Level, engineers can systematically isolate points of failure—distinguishing between context noise, retriever ranking errors, model hallucination, and semantic divergence.

1. The Multi-Level RAG Evaluation Framework

Evaluating an end-to-end RAG system all at once obscures where a failure originates. A faulty answer can stem from an incomplete retrieval step, noisy context stuffing, or an unfaithful generator. To prevent diagnostic ambiguity, evaluation proceeds across three progressive abstraction layers:

2. The Generator Component: Mental Model and Implementation

The Core Role of the Generator

In a RAG architecture, the generator is fundamentally a conditioned mapping function. It receives two inputs:

1. The User Query ($Q$).
2. The Retrieved Context ($C$).

Its objective is to synthesize a natural-language response ($A$) using strictly the semantic evidence provided in $C$, suppressing the base model's internal parametric knowledge when that knowledge contradicts or exceeds $C$.

Naive Implementation

Below is the baseline implementation of the generator using LangChain with gpt-4o-mini at temperature = 0.0. A zero-temperature setting minimizes sampling randomness during deterministic evaluations.

# src/generator.py
import os
from dotenv import load_dotenv
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()

llm = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0.0
)

# Baseline naive prompt
prompt = ChatPromptTemplate.from_messages([
    ("system", (
        "You are a helpful teaching assistant for a course on LLM evaluations. "
        "Answer the student's question only from the context provided below.\n\n"
        "Rules:\n"
        "1. Use only information present in the context. Do not add outside knowledge.\n"
        "2. If the context does not contain enough information to answer, say: "
        "'I don't have enough information in the course material to answer that.'\n"
        "3. Keep the answer clear and concise.\n\n"
        "Context:\n{context}"
    )),
    ("human", "{question}")
])

chain = prompt | llm | StrOutputParser()

def generate_answer(query: str, context: str) -> str:
    return chain.invoke({"question": query, "context": context})

3. Generator Failure Modes

When diagnosing generator failures in isolation, systems break along two orthogonal axes: Faithfulness and Answer Relevance.

Failure Mode 1: Unfaithfulness (Extrinsic Hallucination)

The generator invents claims, falls back on its pre-training data, or extrapolates beyond what the context states.

Real-World Case Study: In the landmark Moffatt v. Air Canada incident, Air Canada's customer-facing chatbot hallucinated a bereavement fare refund policy that was not in its official rules, assuring the customer that claims could be filed retroactively. The tribunal ruled the company liable for the chatbot's invented statements. In RAG systems, such failures occur when the generator's attention drifts away from the provided context chunks.

• User Query: "Does the AI Engineering program include live classes?"
• Retrieved Context: "The AI Engineering program includes recorded lessons, coding assignments, projects, and weekly doubt-solving sessions."
• Faulty Output (Unfaithful): "Yes, the program includes two live classes every week along with weekly doubt-solving sessions."
• Root Cause: The context did not mention live classes; the LLM fabricated them out of general web-corpus priors.

Note: A response can be faithful without being factually true in reality. If the retrieved context incorrectly states "The moon is made of cheddar cheese," a faithful generator must answer based strictly on that claim. Correctness is evaluated separately at the application level.

Failure Mode 2: Answer Irrelevance

The generator stays strictly within the context boundaries, producing zero hallucinations, but fails to address what the user actually asked.

• User Query: "Does the AI Engineering program include live classes?"
• Retrieved Context: "The program includes coding assignments, projects, recorded lessons, and weekly doubt-solving sessions."
• Faulty Output (Irrelevant): "The program includes coding assignments, projects, recorded lessons, and weekly doubt-solving sessions."
• Analysis: The statement is 100% faithful to the text, yet it never directly answers the user's inquiry regarding live classes.
• Ideal Output: "The provided context does not confirm that the program includes live classes. It only confirms recorded lessons, coding assignments, projects, and weekly doubt-solving sessions."

4. Generator Metrics in Isolation

To assess the generator independently of retriever quality, we feed it Golden Context ($C_{gold}$) from a curated benchmark, ensuring that retrieval errors cannot contaminate the score.

Metric 1: Faithfulness

Faithfulness quantifies the degree to which every claim asserted in the generated answer ($A$) can be directly inferred from the reference context ($C_{gold}$).

Algorithmic Execution:

1. Deconstruction: An LLM-as-a-Judge extracts discrete atomic assertions (claims) from the generated answer $A$:

2. Verification: For each claim $c_i$, the judge verifies whether $c_i$ is logically entailed by the provided context $C$:

3. Scoring Formula:

Worked Trace:

• Query: "What does it mean for a benchmark to get saturated?"
• Golden Context ($C_{gold}$): "A benchmark becomes saturated when the evaluation scores of state-of-the-art models cluster near the theoretical maximum, making it impossible to discriminate performance differences."
• Generated Answer ($A$): "A benchmark is saturated when model scores cluster near the ceiling. In addition, benchmark contamination occurs when test sets leak into pre-training data."
• Claims Extracted:
  • $c_1$: "Scores cluster near the ceiling." $\rightarrow$ Supported ($v(c_1) = 1$)
  • $c_2$: "Contamination occurs when test sets leak into pre-training data." $\rightarrow$ Unsupported by $C_{gold}$ ($v(c_2) = 0$)
• Score:

Metric 2: Answer Relevance

Answer Relevance evaluates whether the answer directly addresses the intent of the query, penalizing off-topic tangents, evasions, or repetitive filler.

Answer Relevance is a reference-free evaluation; it evaluates the output solely against the input question without requiring ground-truth answer text.

Algorithmic Execution:

1. Extract atomic statements from the generated answer $A$:

2. For each statement $s_j$, evaluate whether it directly answers, clarifies, or qualifies the query $Q$:

3. Scoring Formula:

Worked Trace:

• Query: "What does it mean for a benchmark to get saturated?"
• Statements Extracted:
  • $s_1$: "A benchmark gets saturated when model scores cluster high and close together." $\rightarrow$ Relevant ($r(s_1) = 1$)
  • $s_2$: "It can no longer tell top models apart." $\rightarrow$ Relevant ($r(s_2) = 1$)
  • $s_3$: "Separately, benchmark contamination happens when test data leaks into training data." $\rightarrow$ Irrelevant ($r(s_3) = 0$)
• Score:

5. Curating the Golden Evaluation Dataset

Automating evaluation requires a structured golden dataset pairing queries with authoritative context excerpts.

Dataset Schema (faithfulness_dataset.json)

[
  {
    "question": "What is the primary trade-off when increasing chunk size in RAG?",
    "golden_context": [
      "Larger chunk sizes preserve broader contextual semantics and prevent fragmented ideas, but introduce extraneous noise into the prompt, reducing contextual relevancy and increasing prompt token costs."
    ]
  },
  {
    "question": "What is online evaluation?",
    "golden_context": [
      "Online evaluation measures an LLM application's performance on live production traffic after deployment, operating continuously without curated golden ground-truth labels."
    ]
  }
]

Dataset Construction Pipeline

Instead of manually drafting queries or delegating the generation entirely to ungrounded LLMs, use a hybrid extraction-and-review workflow:

Chroma Vector DB (862 raw chunks)
       |
       | Export script (export_chroma_chunks.py)
       v
chunks_export.json
       |
       | Step-by-step LLM synthesis (Claude / GPT-4)
       v
Candidate Query + Golden Context Pairs
       |
       | Human-in-the-Loop Verification (Filtering bad questions/drifts)
       v
Final Curated Benchmark (faithfulness_dataset.json)

6. Implementation: Generator Evaluation with DeepEval

We use the open-source evaluation framework DeepEval to run our isolated tests. DeepEval constructs test cases using LLMTestCase objects and applies LLM-as-a-Judge evaluators.

# evals/eval_generator.py
import json
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from src.generator import generate_answer

# 1. Load Ground Truth Benchmark
with open("data/faithfulness_dataset.json", "r") as f:
    eval_data = json.load(f)

# 2. Configure Evaluator Models and Thresholds
JUDGE_MODEL = "gpt-4o-mini"
PASSING_THRESHOLD = 0.70

faithfulness_metric = FaithfulnessMetric(
    threshold=PASSING_THRESHOLD,
    model=JUDGE_MODEL,
    include_reason=True
)

answer_relevancy_metric = AnswerRelevancyMetric(
    threshold=PASSING_THRESHOLD,
    model=JUDGE_MODEL,
    include_reason=True
)

test_cases = []

# 3. Build Test Cases in Isolation (Passing Golden Context directly to Generator)
for row in eval_data:
    query = row["question"]
    golden_context = row["golden_context"] # List of strings
    
    # Flatten context for generator input
    context_str = "\n\n".join(golden_context)
    
    # Synthesize answer using generator under test
    actual_output = generate_answer(query=query, context=context_str)
    
    # Construct DeepEval Test Case
    test_case = LLMTestCase(
        input=query,
        actual_output=actual_output,
        retrieval_context=golden_context  # The reference ground truth context
    )
    test_cases.append(test_case)

# 4. Execute Parallelized Evaluation Suite
if __name__ == "__main__":
    evaluate(
        test_cases=test_cases,
        metrics=[faithfulness_metric, answer_relevancy_metric]
    )

Initial Run Analysis

Running the naive generator on 15 benchmark questions yields:

• Faithfulness: $\approx 91\%$
• Answer Relevance: $\approx 73\%$

Why is baseline Faithfulness naturally high?

Modern instruction-tuned LLMs excel at standard instruction-following. When given the directive "Answer only using the context below," the model readily sticks to the supplied text.

Why does baseline Answer Relevance lag?

When the context does not explicitly provide an immediate, direct answer, the baseline prompt fails to guide the generator on how to handle partial information. As a result, the generator often echoes irrelevant excerpts from the context or provides unhelpful evasions, causing Answer Relevance to score significantly lower.

Generator Optimization: Prompt Engineering Iteration

To fix Answer Relevance without fine-tuning, the generator's system prompt must be refined through iterative error analysis.

# src/generator.py (Refined Production Prompt)
production_prompt = ChatPromptTemplate.from_messages([
    ("system", (
        "You are an expert teaching assistant evaluating technical course content. "
        "Your task is to provide an exact, highly targeted answer to the user's question "
        "relying strictly on the context provided below.\n\n"
        "Strict Operational Guidelines:\n"
        "1. Faithfulness Guarantee: Use ONLY facts explicitly declared in the context. "
        "Do not inject outside knowledge or logical extensions.\n"
        "2. Do Not Overstate: If the context identifies similarities or subtle contrasts, "
        "do not exaggerate them into distinct categories or methods.\n"
        "3. Address the Query Directly: Formulate your opening sentence to answer the core "
        "inquiry immediately. Do not simply summarize the context.\n"
        "4. Handling Partial Evidence: If the context lacks sufficient evidence to answer "
        "the query comprehensively, explicitly state what is known from the context and "
        "clearly clarify what information remains missing.\n"
        "5. Tone & Synthesis: Synthesize informal lecture transcript content into clean, "
        "concise technical language without requiring literal word-for-word string matches.\n\n"
        "Context:\n{context}"
    )),
    ("human", "{question}")
])

Post-Optimization Results

Re-running the evaluation suite with the refined system prompt yields a measurable improvement across both metrics:

Experiment Tracking with Confident AI

DeepEval integrates natively with Confident AI to track evaluation runs. By running:

deepeval view

Every prompt iteration, chunking strategy, evaluation score, and failed test-case reasoning is pushed to an experiment dashboard, serving as an automated regression tracker for prompt engineering.

7. Pipeline Integration: Gluing Retriever and Generator

With both the Retriever (reranking enabled) and the Generator independently validated, we link them into an end-to-end RAGPipeline.

# src/rag_pipeline.py
from typing import Dict, Any, List
from src.retriever import get_relevant_chunks  # Validated retriever
from src.generator import generate_answer      # Validated generator

class RAGPipeline:
    def __init__(self, top_k: int = 5):
        self.top_k = top_k

    def run(self, query: str) -> Dict[str, Any]:
        # 1. Retrieve raw chunks from Vector DB via reranker
        retrieved_docs: List[str] = get_relevant_chunks(query, k=self.top_k)
        
        # 2. Format context string
        context_str = "\n\n".join(retrieved_docs)
        
        # 3. Synthesize answer via generator
        answer = generate_answer(query=query, context=context_str)
        
        return {
            "query": query,
            "answer": answer,
            "retrieved_context": retrieved_docs
        }

rag_pipeline = RAGPipeline(top_k=5)

8. Pipeline-Level Evaluation: The RAG Triad

Once components are linked, evaluating the pipeline changes fundamentally: the generator now receives dynamic chunks from the retriever rather than static, clean golden contexts.

The RAG Triad establishes complete architectural observability by assessing three core relationships:

1. Contextual Relevance ($Q \longleftrightarrow C$): Does the retriever pull context that directly pertains to the question?
2. Faithfulness ($C \longleftrightarrow A$): Is the generator's response derived exclusively from the retrieved context?
3. Answer Relevance ($Q \longleftrightarrow A$): Does the generator's response directly answer what the user asked?

Calculating Contextual Relevance

Contextual Relevance measures the density of useful information in the retrieved chunks, ensuring the prompt is not cluttered with irrelevant noise.

9. End-to-End Pipeline Evaluation Script

# evals/eval_rag_pipeline.py
import json
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualRelevancyMetric
)
from src.rag_pipeline import rag_pipeline

with open("data/faithfulness_dataset.json", "r") as f:
    eval_data = json.load(f)

JUDGE_MODEL = "gpt-4o-mini"
PASSING_THRESHOLD = 0.70

# Define all 3 RAG Triad Metrics
faithfulness_metric = FaithfulnessMetric(threshold=PASSING_THRESHOLD, model=JUDGE_MODEL)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=PASSING_THRESHOLD, model=JUDGE_MODEL)
contextual_relevancy_metric = ContextualRelevancyMetric(threshold=PASSING_THRESHOLD, model=JUDGE_MODEL)

test_cases = []

for row in eval_data:
    query = row["question"]
    
    # Execute full RAG Pipeline
    result = rag_pipeline.run(query)
    
    # In pipeline evaluation, context comes dynamically from the retriever
    test_case = LLMTestCase(
        input=query,
        actual_output=result["answer"],
        retrieval_context=result["retrieved_context"]
    )
    test_cases.append(test_case)

if __name__ == "__main__":
    evaluate(
        test_cases=test_cases,
        metrics=[
            faithfulness_metric,
            answer_relevancy_metric,
            contextual_relevancy_metric
        ]
    )

10. The Curious Case of the Retriever: Root Cause Analysis

Running the complete pipeline evaluation reveals an unexpected performance divergence across the metrics:

This presents an apparent paradox when compared against the earlier isolated retriever benchmarks:

• Isolated Retriever Recall: $99\%$
• Isolated Retriever Precision: $89\%$
• Pipeline Contextual Relevance: $42\%$

Resolving the Paradox

How can a retriever demonstrate exceptional recall and precision in component testing, yet drop to a failing $42\%$ Contextual Relevance during pipeline testing?

1. Context Recall ($99\%$) asks: "Did we capture the necessary answers somewhere in our retrieved text?" $\rightarrow$ Yes. The required needles are present in the haystack.
2. Contextual Precision ($89\%$) asks: "Are the chunks containing the relevant information ranked near the top of the context window?" $\rightarrow$ Yes. The reranker successfully promotes relevant chunks to the top ranks.
3. Contextual Relevance ($42\%$) asks: "What proportion of the total text inside those chunks is useful information rather than filler?" $\rightarrow$ No. The individual chunks are laden with irrelevant content.

Because raw lecture transcripts contain conversational transitions, banter, and tangential remarks, an entire 1,000-token chunk may be retrieved simply because one sentence matched the query. While the chunk as a whole is technically "relevant" (satisfying precision), the remaining sentences dilute the context window, causing Contextual Relevance to plunge to $42\%$.

Engineering Remediation Strategies

To increase Contextual Relevance without hurting recall:

• Reduce Chunk Size: Decrease chunk sizes from large blocks down to smaller segments (e.g., 200–300 tokens) to filter out surrounding conversational noise.
• Sentence-Window Retrieval: Index single sentences for vector similarity searching, but return a modest context window (e.g., $\pm 2$ sentences) to the generator during synthesis.
• Contextual Compression / Extractive Summarization: Introduce a lightweight post-retrieval compression step that strips away sentences showing low semantic similarity to the query before prompt construction.

11. Component-Level vs. Pipeline-Level Comparison

12. Summary and Key Takeaways

• Decoupled Isolation Is Essential: Always evaluate the generator independently using golden reference contexts before connecting it to a live retriever. This ensures generation errors are not misattributed to retrieval flaws.
• Two Core Failure Modes Govern the Generator: A generator can fail by hallucinating ungrounded claims (Unfaithfulness) or by producing grounded responses that fail to address the query (Answer Irrelevance).
• The RAG Triad Measures the Complete Pipeline: Pipeline-level observability requires tracking three interconnected dimensions: Contextual Relevance ($Q \longleftrightarrow C$), Faithfulness ($C \longleftrightarrow A$), and Answer Relevance ($Q \longleftrightarrow A$).
• Golden Datasets Require Human Oversight: High-quality evaluation benchmarks are best created using a hybrid approach: export vector database chunks, run step-by-step LLM question generation, and apply human-in-the-loop review.
• Prompt Engineering Significantly Boosts Generator Performance: Explicitly instructing the model on how to handle partial evidence and direct query answering raised Answer Relevance from $73\%$ to $92\%$.
• The "Retriever Paradox" Explains Context Noise: A retriever can achieve $99\%$ Recall and $89\%$ Precision while still falling to $42\%$ Contextual Relevance. High-level chunk precision does not guarantee claim-level information density.
• Context Dilution Demands Architectural Tuning: When Contextual Relevance drops, the fix lies in retrieval mechanics—such as reducing chunk sizes, adding semantic compressors, or using sentence-window techniques—rather than altering the generation prompt.