Mastering G-Eval: The Deterministic LLM-as-a-Judge Framework Explained

This chapter explores application-level quality evaluation for Retrieval-Augmented Generation (RAG) systems. It details the shift from granular, count-based metrics to holistic, judgment-based metrics, uncovers the failure modes of naive LLM judges, and breaks down the inner workings of the G-Eval framework—specifically its Chain-of-Thought (CoT) evaluation steps and probability-weighted scoring mechanism.

1. The Offline RAG Evaluation Hierarchy

Comprehensive evaluation of a production RAG application requires testing across three distinct architectural tiers:

At the Application Level, the evaluation suite expands into three specialized operational pillars:

Application Quality focuses on three foundational metrics:

• Correctness: Determining whether the generated answer is factually true and aligned with accepted real-world ground truth.
• Completeness: Verifying whether the system fully addresses all sub-questions within a user query or provides a partial response.
• Style: Assessing whether the output conforms to brand-specific pedagogical guidelines (e.g., intuitive framing, conversational prose, explaining concepts before introducing jargon).

2. Count-Based vs. Judgment-Based Metrics

Understanding why classical evaluation methods fall short requires distinguishing between count-based metrics and judgment-based metrics.

The Count-Based Evaluation Pattern

Standard RAG metrics—such as Context Recall, Context Precision, Faithfulness, Answer Relevance, and Context Relevance—rely on claim extraction and discrete counting:

Why Count-Based Approaches Fail on Complex Tasks

Count-based metrics fail when applied to holistic attributes such as Style, Completeness, or Semantic Correctness:

1. The Pedagogical Structure Trap (Style)

Consider assessing whether an answer matches an intuitive "Why-What-How" teaching style. This structural progression exists across the entire response. Breaking the response into isolated sentences destroys the pedagogical narrative; a single sentence cannot be evaluated for "Why-What-How" in isolation.

2. The Analogy Problem (Correctness)

Suppose a model clarifies an abstract database concept using an analogy:

"A database index is like the index at the back of a textbook: it allows the reader to jump directly to a page without scanning every chapter."

If an evaluator isolates this analogy as an individual claim and compares it to a factual ground-truth text about B-Trees, the claim appears factually unrelated or contradictory. The isolated claim is penalized, even though the analogy improves the overall explanation.

Evaluating these properties requires an evaluator capable of synthesizing information across the entire response: an LLM-as-a-Judge.

3. The Naive LLM-as-a-Judge Approach and Its Failure Modes

In a naive LLM-as-a-judge setup, an evaluator prompt instructs an advanced model (such as GPT-4) to evaluate an answer on a scale from 0 to 10:

While simple to implement, this naive approach produces high score variance across identical test runs. Running the exact same evaluation suite multiple times can yield unstable macro scores (e.g., swinging from $60\%$ to $75\%$).

The Two Root Causes of Variance

1. The Vague Constitution Problem

A brief prompt like "decide how factually correct it is" lacks a definitive set of rules. In one API call, the LLM might penalize the response for omitting details present in the golden dataset. In the next call, it might prioritize semantic accuracy and ignore omitted points. Without an explicit rulebook, the judge's internal criteria shift between runs.

2. The Discrete Integer Sampling Problem

LLMs sample tokens based on a probability distribution. When grading an answer between a 7 and an 8, the internal logit distribution might look like this:

• Token "8": $P = 0.51$
• Token "7": $P = 0.40$
• Token "6": $P = 0.09$

Because "8" has the highest probability, standard decoding outputs 8. In a subsequent run, slight variations in context handling or sampling may shift probabilities marginally:

• Token "7": $P = 0.51$
• Token "8": $P = 0.40$
• Token "6": $P = 0.09$

The model now outputs 7. Sampling an integer directly forces a continuous underlying assessment into coarse discrete steps, causing significant variance when scores are aggregated across test suites.

4. The G-Eval Framework: Core Innovations

Introduced in 2023, the G-Eval framework addresses the instability of naive LLM judges through two core mechanisms:

G-Eval Pipeline
   |
   | Innovation 1: Chain-of-Thought (CoT) Task Decomposition
   v
Evaluation Criteria converted into explicit Evaluation Steps & Scoring Rubrics
   |
   | Innovation 2: Probability-Weighted Scoring
   v
Continuous Expected Score derived from Top-k Output Token Log Probabilities

Innovation 1: Detailed Rubrics and Evaluation Steps

Rather than relying on high-level instructions, G-Eval decomposes evaluation criteria into a step-by-step assessment protocol. This can be generated using Chain-of-Thought (CoT) prompting or supplied directly as an explicit rubric:

High-Level Criterion: "Evaluate factual correctness."
   |
   | (Chain-of-Thought Breakdown)
   v
Formal Evaluation Steps:
1. Isolate the factual assertions made in the actual output.
2. Cross-reference assertions exclusively against the golden answer.
3. Penalize only direct contradictions or demonstrably false statements.
4. Do not penalize for brevity or omitted details if existing claims are accurate.
5. Provide a deterministic score using the specified range rubric.

Defining these steps constrains the LLM judge's reasoning, ensuring the same evaluative standard is applied across all test cases.

Innovation 2: Probability-Weighted Scoring

Rather than parsing a single discrete integer token (e.g., "8"), G-Eval inspects the model's output logits for the scoring step. It extracts the top-$k$ numerical score tokens, normalizes their probabilities, and calculates a continuous mathematical expectation.

Raw Output Logits:
Token "8": P = 0.70
Token "7": P = 0.20
Token "9": P = 0.05
Token "the": P = 0.01 (Discarded)
Token ":": P = 0.04 (Discarded)
   |
   | Filter & Normalize
   v
Normalized Probabilities:
P~("8") = 0.737
P~("7") = 0.210
P~("9") = 0.053
   |
   | Compute Mathematical Expectation
   v
Continuous Expected Score = (8 * 0.737) + (7 * 0.210) + (9 * 0.053) = 7.84

Calculating an expected value smooths out minor token probability shifts, preventing erratic score swings between evaluations.

5. Mathematical Foundations of Probability-Weighted Scoring

During generation, the final linear layer of a Transformer projects hidden states across vocabulary size $V$, generating raw unnormalized log-odds (logits) $z$. The probability distribution over all tokens is calculated via the softmax function:

When instructed to score an answer from $0$ to $10$, the model assigns the vast majority of its probability mass to numeric tokens $S = \{0, 1, 2, \dots, 10\}$.

Derivation of the Expected Continuous Score

Step 1: Top-$k$ Extraction and Token Filtering

We query the model to return the top-$k$ most probable tokens at the score generation position. Non-numeric tokens are removed, leaving a set of valid score tokens $S_{valid} \subseteq \{0, 1, \dots, 10\}$ with raw probabilities $p_i = P(y = s_i)$.

Step 2: Probability Mass Normalization

Because non-score tokens were removed and only the top-$k$ candidates were retained, the sum of remaining probabilities $\sum_{s_i \in S_{valid}} p_i < 1.0$. The probabilities are normalized so their sum equals $1$:

Step 3: Expected Value Calculation

The continuous score is the expectation of the score variable over the normalized distribution:

Step 4: Metric Scale Normalization

To scale the score to standard unit intervals $[0, 1]$ for automated pipeline checks:

Numerical Walkthrough

Suppose an LLM judge evaluates an actual output against a reference and generates the following top-5 tokens:

1. Calculate the sum of valid probabilities:

2. Normalize valid probabilities:

3. Compute expected value:

4. Scale to $[0, 1]$:

If a downstream threshold requires $\tau \ge 0.70$, the test case passes ($0.7841 \ge 0.70$). In a standard setup, a shift from token 8 to 7 drops the score by a full point ($8 \to 7$). With probability weighting, the score shifts smoothly (e.g., $7.84 \to 7.42$), preserving stability across evaluation runs.

6. End-to-End Implementation Architecture with DeepEval

The implementation below uses DeepEval, which provides an abstraction for building G-Eval pipelines with custom parameters.

Golden Dataset Schema (goldens/correct_n_s_goldens.json)

[
  {
    "input": "What is an offline evaluation in a RAG pipeline?",
    "expected_output": "Offline evaluation is the process of assessing a RAG system using a predefined, curated golden dataset before deploying the model to production. It isolates system performance across retrieval and generation components without exposing end users to bugs or inaccuracies."
  },
  {
    "input": "Explain the difference between dense and sparse retrieval.",
    "expected_output": "Sparse retrieval uses lexical matching based on token frequencies and term overlaps (e.g., BM25). Dense retrieval matches queries and documents using continuous vector embeddings produced by encoders, capturing semantic meaning and context beyond exact keyword overlap."
  }
]

Evaluation Script (eval_application.py)

import json
from deepeval import evaluate
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval, Rubric
from my_rag_app import rag_pipeline  # The target RAG application under test

# 1. Load Golden Dataset
with open("goldens/correct_n_s_goldens.json", "r") as f:
    golden_data = json.load(f)

# 2. Define Deterministic G-Eval Metrics

# Metric A: Correctness Metric with Explicit Rubric
correctness_metric = GEval(
    name="Correctness",
    model="gpt-4o-mini",
    evaluation_params=[
        LLMTestCaseParams.INPUT, 
        LLMTestCaseParams.ACTUAL_OUTPUT, 
        LLMTestCaseParams.EXPECTED_OUTPUT
    ],
    evaluation_steps=[
        "Isolate all factual assertions made within the actual output.",
        "Compare only these factual claims against the expected output.",
        "A claim is wrong ONLY if it directly contradicts the expected output or asserts a verifiable untruth.",
        "A factually accurate answer must score at least 9 even if it is brief, omits secondary context, or covers fewer points than the expected output.",
        "Do not deduct points for brevity or missing context; only explicit errors count as failures."
    ],
    rubric=[
        Rubric(score_range=(0, 4), expected_output="The response contains clear factual inaccuracies or contradicts the expected output."),
        Rubric(score_range=(5, 8), expected_output="The response is largely accurate but contains minor factual ambiguities or mild inaccuracies."),
        Rubric(score_range=(9, 10), expected_output="All factual claims are completely true and consistent with the reference answer.")
    ],
    threshold=0.7,
    strict_mode=False  # Required: Enables logit probability weighting
)

# 3. Build Test Cases
test_cases = []
for item in golden_data:
    query = item["input"]
    expected = item["expected_output"]
    
    # Execute RAG pipeline to generate the live answer
    actual = rag_pipeline.generate(query)
    
    test_case = LLMTestCase(
        input=query,
        actual_output=actual,
        expected_output=expected
    )
    test_cases.append(test_case)

# 4. Execute the Deterministic Evaluation Suite
if __name__ == "__main__":
    evaluate(test_cases=test_cases, metrics=[correctness_metric])

7. Metric 1 Deep-Dive: Correctness and Iterative Rubric Evolution

Correctness vs. Faithfulness

A common mistake in RAG evaluation is conflating Faithfulness with Correctness. They measure fundamentally different properties:

• Faithfulness: Measures whether the generated answer is derived exclusively from the retrieved context chunks (groundedness).
• Correctness: Measures whether the answer is factually accurate against an accepted real-world standard, regardless of whether the retrieved context contained it.

The Problem: When Thorough Golden Datasets Penalize Conciseness

In our baseline run over 15 test questions, the correctness score dropped to $66\%$ (8 passed, 7 failed):

Initial Run Results:
Pass: 8 / 15
Fail: 7 / 15
Correctness Macro Average: 0.66 (Target Threshold: >= 0.70)

Analyzing the failure logs revealed that the golden reference answers written by domain experts were comprehensive essays. In contrast, the RAG generator produced concise, accurate answers that covered only $\sim 70\%$ of the points mentioned in the golden dataset.

Because the initial instructions did not distinguish between missing context and factual errors, the LLM judge penalized the concise responses as "incorrect."

The Solution: Refining the Evaluation Steps and Rubric

The evaluation steps were adjusted to separate brevity from incorrectness:

# Updated Correctness Evaluation Steps
evaluation_steps = [
    "Compare only factual assertions in the actual output against the expected output.",
    "A claim is wrong ONLY if it directly contradicts the expected output or is factually false.",
    "A factually accurate answer must score at least 9, even if it is shorter, less detailed, or covers fewer points than the reference answer.",
    "Do NOT penalize for brevity, missing examples, or omitted non-critical details.",
    "Additional correct factual details must never lower the score."
]

Post-Adjustment Evaluation Results

Rerunning the evaluation with the updated rubric yielded a score of $84\%$:

Updated Run 1:
Macro Correctness Score: 0.84 | Passed: 14 | Failed: 1 (Test Case 7 failed)

Updated Run 2 (Immediately following):
Macro Correctness Score: 0.83 | Passed: 14 | Failed: 1 (Identical Test Case 7 failed)

The system stabilized at $\approx 83-84\%$, and consistently flagged the exact same failing test case (Test Case 7), demonstrating that the metric had become reliable and deterministic.

8. Metric 2 Deep-Dive: Completeness and Generator Optimization

Defining Completeness

While Correctness penalizes false statements, Completeness evaluates whether the output covers all distinct sub-questions or requirements posed in the user prompt:

An answer can be $100\%$ factually correct while still failing completeness by omitting Sub-part 2 or Sub-part 3.

Diagnosing Baseline Completeness Failures

Running a 15-question evaluation suite using a baseline Completeness metric yielded poor results:

Completeness Macro Score: 0.68
Passed: 5 / 15
Failed: 10 / 15

Inspecting the system prompt of the RAG generator revealed the source of the issue:

"Be concise. Answer the user's question directly from the context. Do not elaborate or offer unsolicited details."

The generator had been over-constrained toward brevity, causing it to answer only the first part of compound questions and drop remaining sub-questions.

Resolving Completeness via Prompt Engineering

The RAG generator prompt was modified to encourage addressing multi-part questions while remaining grounded in the retrieved context:

Previous Generator Prompt:
"Answer concisely based on the context. Say as little as possible."
   |
   | (Prompt Engineering Refinement)
   v
Optimized Generator Prompt:
"Answer thoroughly based on the provided context. Identify every distinct 
part of the user query and address each one. If a concept has multiple components 
or the question contains multiple sub-questions, systematically cover all of them 
rather than stopping at the first. Do not extrapolate beyond the context."

Post-Optimization Results

Re-evaluating the system with the updated generator prompt produced a significant improvement:

Baseline Run:   Macro Score = 0.68 | Passed: 5  | Failed: 10
Optimized Run:  Macro Score = 0.75 | Passed: 14 | Failed: 1

By refining the generator prompt, completeness pass rates jumped from $33\%$ to $93\%$, resolving the issue without retraining or fine-tuning the underlying model.

9. Metric 3 Deep-Dive: Brand & Pedagogical Style (Reference-Free Evaluation)

Reference-Free Evaluation

Unlike Correctness and Completeness, which evaluate outputs against golden answers, evaluating Brand and Pedagogical Style does not require a reference answer. The judge evaluates the output directly against a rubric defining the organization's voice and pedagogical approach.

Formalizing Brand Voice

The target pedagogical style was defined using an explicit rubric:

• Use an intuitive, conversational tone.
• Explain the core concept and intuition in plain language before introducing formal notation or jargon.
• Unpack technical terms immediately when introduced.
• Use analogies or real-world examples to clarify abstract ideas.

Handling Over-Correction in Evaluator Rubrics

During initial style evaluation, the model scored poorly:

Initial Scores: Correctness: 0.83 | Completeness: 0.75 | Style: 0.54

Debugging the failure logs revealed an over-correction in the evaluation rubric. The rubric contained the following rule:

"Reward the use of a concrete example or analogy."

The LLM judge interpreted this rule as a mandatory requirement, failing any response that did not include an explicit analogy—even for straightforward factual queries (e.g., "What does API stand for?") where an analogy is unnecessary.

Evaluator Prompt Adjustment:
"An analogy or concrete example is a bonus when explaining abstract concepts, 
but a clear, direct, and well-explained answer without an analogy is fully acceptable."

Simultaneously, the generator's prompt was updated to explain the core intuition before introducing technical jargon:

Generator Style Directive:
"Write in flowing, conversational prose as an instructor would explain something out loud. 
Avoid unprompted bulleted lists. Explain the intuition first in plain language, 
and unpack technical terms when used."

Final Multi-Metric Results

Rerunning the evaluation suite with both adjustments yielded:

All three metrics reached production-ready levels, providing a stable, automated evaluation pipeline.

10. Engineering Insights, Best Practices, and Trade-Offs

When to Auto-Generate Steps vs. When to Hardcode

• Use Auto-Generated CoT Steps (Exploratory): Useful when establishing initial baselines for unfamiliar metrics. Supplying high-level criteria allows the LLM to propose reasonable evaluation steps.
• Use Hardcoded Steps & Rubrics (Production): In continuous integration and deployment pipelines, allowing the judge to generate its own steps introduces variance across builds. Hardcoding the steps into the test suite guarantees that the same evaluative criteria are used across every run.

The Inter-Metric Balancing Act

Optimizing evaluation metrics requires balancing trade-offs across components:

Evaluating a RAG application requires measuring metrics concurrently. Optimizing for a single metric in isolation can degrade other aspects of the system.

11. Summary and Key Takeaways

• Count-Based vs. Judgment-Based Split: Classical RAG evaluation relies on counting extracted claims (e.g., Faithfulness, Precision). Holistic properties—such as factual correctness, answer completeness, brand style, and safety—require judgment-based evaluation.
• Naive LLM-as-a-Judge Failure Modes: Asking an LLM to assign an integer score from 0 to 10 based on a brief prompt results in significant score variance. This instability stems from vague evaluative criteria and discrete integer token sampling.
• G-Eval Innovation 1 (Explicit Evaluation Steps): G-Eval replaces loose system prompts with detailed, step-by-step instructions and defined scoring rubrics, constraining the judge's reasoning and minimizing subjective drift.
• G-Eval Innovation 2 (Probability-Weighted Scoring): Rather than sampling discrete tokens, G-Eval inspects the top-$k$ output logits, normalizes their probabilities, and computes an expected value $\mathbb{E}[\text{Score}] = \sum s_i \cdot \tilde{p}_i$. This turns discrete token jumps into stable, continuous measurements.
• Correctness vs. Faithfulness: Faithfulness assesses whether an answer is grounded in the retrieved context. Correctness assesses whether the answer is factually true in the real world. A response can be faithful while remaining factually incorrect, and vice versa.
• Iterative Rubric Refinement: Discrepancies between comprehensive golden answers and concise generated outputs can cause judges to penalize brevity. Evaluation rubrics should clearly distinguish omitted non-critical context from actual factual errors.
• Reference-Free Capabilities: Metrics like Brand Style do not require reference answers. They evaluate outputs directly against clear qualitative rubrics (e.g., assessing conversational tone, plain-language explanations, or unpacking technical jargon).
• Impact of Prompt Engineering: When completeness or style scores fall short, refining generator prompts (e.g., instructing the model to cover all sub-questions or adjust tone) often resolves the issue without requiring model fine-tuning.