Automated RAG Benchmarking and Metric Calculation

This chapter explores the systematic evaluation of Retrieval-Augmented Generation (RAG) pipelines. Rather than relying on manual, ad-hoc inspection, we implement an automated benchmarking framework using an LLM-as-a-Judge (specifically, Llama 3.1 8B). We will dissect the core metrics of the RAG Triad—Answer Relevancy, Context Precision, and Context Recall—and analyze their mathematical formulations, failure modes, and optimization strategies.

1. The RAG Triad Evaluation Framework

To comprehensively evaluate a RAG application, we must analyze the interactions between three core components: the User Query, the Retrieved Context, and the Generated Response. This relationship is formalized as the RAG Triad.

Structure & Concept Flow

[User Query]

/ \

Context / \ Answer

Precision / \ Relevancy

/ \

v v

[Retrieved] ------> [Generated

Context] Grounded- Response]

ness

The triad breaks down into three distinct evaluation flows:

Query to Context (Context Precision / Relevancy): Evaluates whether the retrieval engine fetches high-quality, relevant information and ranks it appropriately.
Context to Response (Groundedness / Faithfulness): Verifies that the LLM's generated response is strictly derived from the retrieved context, eliminating hallucinations.
Query to Response (Answer Relevancy): Measures how well the generated response directly addresses the user's original query.

2. Answer Relevancy: Mechanics and LLM-as-a-Judge Workflow

Answer Relevancy quantifies how pertinent the generated response is to the user's query. A high score indicates the model avoided filler text, stayed on topic, and directly answered the prompt.

The LLM-as-a-Judge Execution Flow

To calculate Answer Relevancy without relying on exact keyword matching, the LLM Judge (Llama 3.1 8B) executes a two-step reverse-engineering process:

Structure & Concept Flow

User Query (Ground Truth)

| (Comparison Step)

[Similarity Check] <------------------+

| |

| (Dense Vector Embedding) |

v |

Average Similarity Score [0 to 1] |

| (Step 2: Generate Questions)

Actual Output (RAG Response) --------> LLM Judge (Llama 3.1 8B)

Step 1: Question Generation: The LLM Judge reads only the generated output and reverse-engineers $N$ hypothetical questions that this output could successfully answer.
Step 2: Semantic Similarity Check: The framework performs a semantic similarity check between the LLM-generated questions and the original user query.

Dense Search vs. Sparse Search

The similarity check does not rely on simple keyword matching (sparse search). Instead, it uses a dense search based on vector embeddings. Depending on your performance and latency requirements, you can implement this using two embedding strategies:

Embedding Strategy	Architecture	Latency	Accuracy / Depth
Bi-Encoders	Embeds queries and documents independently into a shared vector space; compares them using cosine similarity.	Low (Fast execution)	Moderate (Good for quick, scalable searches)
Cross-Encoders	Feeds both texts simultaneously into the transformer network to calculate a unified attention-based similarity score.	High (Slower execution)	High (In-depth, highly accurate comparison)

3. Troubleshooting Low Answer Relevancy Scores

When your RAG pipeline yields a low Answer Relevancy score, it typically points to one of three systemic issues:

Off-Topic Answers: The LLM misunderstood the prompt or drifted during generation, producing an answer to a query that was never asked.
Padded Responses (Fillers): The response contains excessive boilerplate text, disclaimers, or conversational noise. The LLM Judge penalizes these because the generated questions from the padded text do not align with the core user query.
Incomplete Answers: The user query asked for multiple details (e.g., "What are the fees and the minimum balance?"), but the response only addressed one aspect. When the LLM Judge generates questions from this incomplete response, the missing aspects cause a drop in the average similarity score.

4. Context Precision: Evaluating Chunk Ranking and Relevancy

Context Precision measures whether the retrieval system successfully ranks the most relevant information chunks at the top of the retrieved results (Top- $K$ ).

The Importance of Ranking

In RAG pipelines, retrieval engines return a list of ranked chunks. We want the most relevant chunks to appear first, while less relevant chunks or noise should be pushed to the bottom.

Structure & Concept Flow

Retrieval Results (Top-K)

+-- Chunk 1: Highly Relevant (Rank 1) --> Ideal Case

+-- Chunk 2: Partially Relevant (Rank 2)

+-- Chunk 3: Irrelevant Noise (Rank 3)

+-- Chunk 4: Highly Relevant (Rank 4) --> Penalized due to poor ranking

If irrelevant noise is placed at Rank 1 and the highly relevant chunk is pushed to Rank 4, the LLM may suffer from "lost in the middle" phenomena or prioritize the noise during generation.

The Role of Re-ranking

If your evaluation reveals low Context Precision, the primary engineering lever to resolve this is Re-ranking. A Re-ranker acts as a secondary, highly accurate scoring layer that re-orders the initial retrieve-set before passing it to the LLM.

5. Mathematical Formulation of Context Precision

To calculate Context Precision, the LLM Judge evaluates each retrieved chunk for relevancy against the query. It then applies a penalty based on the rank of the relevant chunks.

Real-World Example

Query: "What is the minimum balance for my savings account?"
Retrieved Chunks:
1. Chunk 1: "The minimum balance for savings accounts is $10,000." (Relevant)
2. Chunk 2: "Non-maintenance fees are $25 per month." (Relevant)
3. Chunk 3: "KYC updates are required every 8 years." (Irrelevant Noise)
4. Chunk 4: "An initial deposit of $5,000 is required for account opening." (Relevant)

Penalty and Precision Calculation

The framework calculates precision at each rank ( $P@i$ ) for the relevant chunks. If a relevant chunk is positioned after an irrelevant chunk, its precision score is discounted.

At Rank 1 (Relevant):

P@1 = \frac{1}{1} = 1.0

At Rank 2 (Relevant):

P@2 = \frac{2}{2} = 1.0

At Rank 3 (Irrelevant): Skipped for scoring, but acts as a penalty for subsequent ranks.
At Rank 4 (Relevant):

P@4 = \frac{3}{4} = 0.75 \quad \text{(Penalized because of Chunk 3)}

To find the final Context Precision score, we calculate the average of the precision values at each rank where a relevant chunk was found:

\begin{aligned} \text{Context Precision} &= \frac{P@1 + P@2 + P@4}{\text{Total Relevant Chunks}} \\ \text{Context Precision} &= \frac{1.0 + 1.0 + 0.75}{3} \approx 0.917 \end{aligned}

Note: If we do not use an evaluation framework (such as Ragas or TruLens), we must manually define these scoring rules, penalty thresholds, and LLM prompts to parse the retrieved chunks.

6. Context Recall: Ground Truth Alignment via Atomic Claims

While Context Precision focuses on ranking and relevancy, Context Recall measures whether the retrieval system successfully retrieved all the necessary information required to construct the ground truth answer.

The Atomic Claims Workflow

Context Recall is calculated by comparing the Reference Answer (the gold-standard ground truth) against the Retrieved Context.

Structure & Concept Flow

Reference Answer (Ground Truth)

LLM Judge (Llama 3.1 8B)

| (Decomposes into)

[Atomic Claim 1] ---> Is this claim present in the retrieved chunks? ---> Yes/No

[Atomic Claim 2] ---> Is this claim present in the retrieved chunks? ---> Yes/No

[Atomic Claim 3] ---> Is this claim present in the retrieved chunks? ---> Yes/No

Decomposition: The LLM Judge reads the reference answer and breaks it down into independent, atomic statements (claims).
Attribution: The Judge analyzes the retrieved chunks to see if each atomic claim can be attributed to (found within) the retrieved context.
Scoring: The final score is the ratio of successfully attributed claims to the total number of claims in the reference answer.

\begin{aligned} \text{Context Recall} &= \frac{\text{Number of Attributed Claims}}{\text{Total Number of Atomic Claims in Reference}} \end{aligned}

Example

Reference Answer: "Urban branches require a $10,000 minimum balance, whereas rural branches require $5,000."
Extracted Atomic Claims:
- Claim 1: "Urban branches require a minimum balance."
- Claim 2: "The minimum balance for urban branches is $10,000."
- Claim 3: "Rural branches require a minimum balance."
- Claim 4: "The minimum balance for rural branches is $5,000."
Evaluation: If the retrieved chunks only mention urban branch requirements, only Claims 1 and 2 are attributed. The Context Recall score would be:

\frac{2}{4} = 0.50

7. Comparative Synthesis of RAG Metrics

Metric	Primary Inputs Required	Core Evaluation Question	Typical Failure Mode	Primary Optimization Lever
Answer Relevancy	User Query, Generated Response	Does the generated response directly and cleanly answer the user's question?	Off-topic answers, padded responses, incomplete details.	Prompt engineering, system instructions, output parsing.
Context Precision	User Query, Retrieved Chunks	Are the retrieved chunks ranked correctly with the most relevant information at the top?	Irrelevant noise appearing at Rank 1; relevant chunks pushed down.	Re-ranking models (e.g., Cohere Rerank, BGE-Reranker).
Context Recall	Reference Answer, Retrieved Chunks	Did the retrieval engine fetch all the necessary information to construct the ground truth?	Missing critical facts or context chunks during vector search.	Chunk size tuning, hybrid search (dense + sparse), metadata filtering.

8. Summary and Key Takeaways

The RAG Triad is the industry-standard framework for evaluating RAG applications, isolating performance issues into retrieval (Context Precision/Recall), generation (Answer Relevancy), and alignment (Groundedness).
Answer Relevancy is calculated by having an LLM Judge reverse-engineer questions from the generated response and comparing them semantically to the original user query.
Dense embeddings (using Bi-encoders for speed or Cross-encoders for accuracy) are required to perform semantic similarity checks for relevancy scoring.
Context Precision penalizes your system if relevant chunks are positioned below irrelevant noise in the retrieved results.
Re-ranking is the most effective engineering solution to improve low Context Precision scores.
Context Recall relies on decomposing the gold-standard reference answer into atomic claims and checking if those claims exist within the retrieved context chunks.