Evaluation Pipeline Implementation with Gemini and Ragas

This chapter covers the systematic evaluation of Retrieval-Augmented Generation (RAG) pipelines using the Ragas framework and Google Gemini as the evaluation judge. It explores the mechanics of advanced evaluation metrics—specifically Context Recall and Answer Correctness—detailing their mathematical foundations, failure modes, and practical implementation within an observability dashboard.

1. Understanding Context Recall

Context Recall measures the extent to which the retrieved context aligns with the ground truth (the ideal, golden reference answer). It evaluates whether the retrieval pipeline successfully fetched all the necessary information required to construct a complete and accurate response.

The Core Mental Model

To calculate Context Recall, the evaluation pipeline uses an LLM as an external judge to perform the following steps:

1. Analyze the Ground Truth: The LLM analyzes the ground truth (expected answer) and breaks it down into individual, distinct factual statements or "claims."
2. Cross-Reference with Retrieved Chunks: The LLM checks whether each extracted claim can be directly attributed to any of the retrieved context chunks.
3. Calculate the Ratio: The final score is the ratio of ground truth claims supported by the retrieved context to the total number of ground truth claims.

Real-World Mathematical Example

Consider a user query about a company's return policy.

• Ground Truth (Expected Answer): "Tech Nest allows returns within 30 days. The item must be in its original packaging, and a receipt is required. Returns are free of charge."
• Extracted Claims ($N = 4$):
  1. Tech Nest allows returns within 30 days.
  2. The item must be in its original packaging.
  3. A receipt is required.
  4. Returns are free of charge.

Now, suppose the retrieval pipeline fetches three chunks:

• Chunk 1: "Our return window is 30 days for all Tech Nest purchases."
• Chunk 2: "Customers must present a valid receipt to process any return."
• Chunk 3: "We do not charge restocking or shipping fees for returned items; they are free."

The LLM judge maps the claims to the retrieved chunks:

2. Context Recall vs. Hallucination (The Ground Truth Distinction)

A common point of confusion is whether an unsupported claim in Context Recall implies that the generator is hallucinating.

Note: No. An unsupported claim in Context Recall does not mean the RAG pipeline is hallucinating.

Why This Distinction Matters

• The Ground Truth is Golden: The ground truth is our static, pre-verified reference answer. It is assumed to be 100% correct and factual.
• We are Evaluating Retrieval, Not Generation: In Context Recall, we are not validating the LLM's generated response. Instead, we are validating the vector store retrieval pipeline.
• The Diagnostic Signal: A low Context Recall score indicates that the retrieved context is incomplete. It tells us that the vector database failed to return the chunks necessary to construct the ideal answer.

Root Causes of Low Context Recall

When your evaluation pipeline flags a low Context Recall score, it is typically caused by one of three architectural issues:

1. Missing Chunks (Data Ingestion Failure): The information required to answer the query does not exist in the vector database. It was either never ingested, or it was lost during document parsing.
2. $K$ Value is Too Small: The relevant chunks exist in the database, but your retrieval parameter $K$ (the number of retrieved documents) is set too low. For example, if $K=3$, but the critical piece of information is contained in the 4th most similar chunk, it will be excluded.
3. Embedding Semantic Gap: The embedding model failed to capture the semantic relationship between the user query and the target chunk, causing the chunk to rank poorly. This issue can be mitigated by using a higher-quality embedding model or implementing a Re-ranking step (e.g., using Cohere or BGE re-rankers).

3. Answer Correctness: The Dual-Component Metric

While metrics like Faithfulness and Relevancy evaluate the structural integrity of the RAG pipeline, Answer Correctness evaluates the actual quality and accuracy of the final generated answer.

Answer Correctness measures the alignment between the generated response and the ground truth. It is a composite metric consisting of two distinct components:

Component 1: The Factual Component (F1-Score)

The factual component evaluates the overlap of hard facts between the generated response and the ground truth. The LLM judge extracts individual claims from both the generated response and the ground truth, categorizing them into three buckets:

• True Positives ($TP$): Claims present in both the generated response and the ground truth (factually correct and aligned).
• False Positives ($FP$): Claims present in the generated response but not in the ground truth (potential hallucinations or irrelevant additions).
• False Negatives ($FN$): Claims present in the ground truth but missing from the generated response (omissions).

Using these buckets, we calculate the Factual $F_1$ score:

Component 2: The Semantic Component (Cosine Similarity)

The semantic component measures the conceptual similarity between the generated response and the ground truth. Even if the exact wording or specific factual claims differ slightly, this component checks if the overall meaning is preserved.

• The evaluation pipeline generates vector embeddings for both the generated response and the ground truth.
• It calculates the Cosine Similarity between these two vectors:

4. Mathematical Formulation of Answer Correctness

The final Answer Correctness score is a weighted average of the Factual $F_1$ score and the Semantic Similarity score.

The Formula

Where:

• $W_1$ is the weight assigned to the factual component.
• $W_2$ is the weight assigned to the semantic component.
• The weights are constrained such that:

Hyperparameter Tuning ($W_1$ and $W_2$)

These weights act as hyperparameters that you can tune based on your specific application requirements:

• Factual-Heavy ($W_1 > 0.5$): Critical for high-precision domains (e.g., financial reporting, medical advice, legal compliance) where exact numbers, dates, and claims are paramount.
• Semantic-Heavy ($W_2 > 0.5$): Useful for creative writing, conversational agents, or summarization tasks where the overall tone and meaning are more important than rigid factual matching.

5. Step-by-Step Workflow of Answer Correctness Evaluation

The following vertical workflow illustrates how the RAG response, user input, and reference ground truth are processed to compute the final Answer Correctness score.

6. Synthesis of Core Ragas Metrics

To build a robust evaluation pipeline, you must understand how these different metrics target different components of the RAG lifecycle.

7. Practical Implementation & Observability Dashboard

Evaluating these metrics manually is impractical at scale. The industry standard is to automate this process using Ragas combined with an LLM judge (such as Gemini) and visualize the results in an observability dashboard.

Python Implementation: Evaluating with Gemini and Ragas

Below is a complete, production-ready script demonstrating how to configure Ragas to use Google Gemini as the evaluation LLM to run a pipeline evaluation.

import os
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
    answer_correctness
)
from langchain_google_genai import ChatGoogleGenerativeAI, GoogleGenerativeAIEmbeddings

# 1. Configure API Keys
os.environ["GOOGLE_API_KEY"] = "your-gemini-api-key"

# 2. Initialize Gemini LLM and Embeddings for Evaluation
evaluator_llm = ChatGoogleGenerativeAI(
    model="gemini-1.5-pro",
    temperature=0.0
)

evaluator_embeddings = GoogleGenerativeAIEmbeddings(
    model="models/text-embedding-004"
)

# 3. Override Ragas Default Models
# This ensures Ragas uses Gemini instead of defaulting to OpenAI
context_precision.llm = evaluator_llm
context_recall.llm = evaluator_llm
faithfulness.llm = evaluator_llm
answer_relevancy.llm = evaluator_llm
answer_relevancy.embeddings = evaluator_embeddings
answer_correctness.llm = evaluator_llm
answer_correctness.embeddings = evaluator_embeddings

# 4. Prepare the Evaluation Dataset (The "Golden Dataset")
# This dataset represents queries, retrieved contexts, generated responses, and ground truths.
eval_data = {
    "question": [
        "What is the Tech Nest return policy?"
    ],
    "contexts": [[
        "Our return window is 30 days for all Tech Nest purchases.",
        "Customers must present a valid receipt to process any return.",
        "We do not charge restocking or shipping fees for returned items; they are free."
    ]],
    "answer": [
        "Tech Nest allows returns within 30 days. You must provide a receipt, and returns are completely free."
    ],
    "ground_truth": [
        "Tech Nest allows returns within 30 days. The item must be in its original packaging, and a receipt is required. Returns are free of charge."
    ]
}

dataset = Dataset.from_dict(eval_data)

# 5. Run the Evaluation Pipeline
print("Running Ragas evaluation with Gemini...")
results = evaluate(
    dataset=dataset,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
        answer_correctness
    ]
)

# 6. Export and Analyze Results
df_results = results.to_pandas()
print("\nEvaluation Results:")
print(df_results[[
    "question", 
    "context_precision", 
    "context_recall", 
    "faithfulness", 
    "answer_relevancy", 
    "answer_correctness"
]])

Analyzing the Observability Dashboard

When running evaluations across a "Golden Dataset" (a curated set of test queries and ground truths), results are aggregated into an observability dashboard (often built with Streamlit or Phoenix).

Diagnostic Case Study: Analyzing "Golden_01" (Tech Nest Return Policy)

• Faithfulness (0.86): The generated response is mostly grounded, but contains a minor claim not explicitly supported by the retrieved context (e.g., claiming returns can be done "in-store" when the context only mentions shipping).
• Context Recall (0.75): The retrieval system missed one key aspect of the ground truth (the requirement for "original packaging").
• Actionable Fix:
  1. Check if the "original packaging" rule is documented in the source files.
  2. If it is, increase the retrieval parameter $K$ from 3 to 5, or implement a re-ranking step to ensure the packaging policy chunk is retrieved.

Summary and Key Takeaways

• Context Recall measures the completeness of the retrieval step by checking what percentage of the ground truth claims are present in the retrieved context chunks.
• Retrieval Failures (such as a small $K$ value, missing database chunks, or embedding semantic gaps) are the primary drivers of low Context Recall scores.
• Answer Correctness is an end-to-end metric that combines factual accuracy (via an $F_1$ score on extracted claims) and semantic similarity (via cosine similarity of embeddings).
• Factual vs. Semantic Weights ($W_1$ and $W_2$) are adjustable hyperparameters that allow developers to tune the Answer Correctness metric for strict factual compliance or general semantic alignment.
• LLM-as-a-Judge frameworks like Ragas automate complex evaluations by using advanced models like Gemini to extract claims, check attributions, and calculate mathematical scores.
• Golden Datasets are essential for continuous integration and deployment (CI/CD) of LLM applications, allowing teams to catch regressions in retrieval or generation quality before they hit production.