How to Test AI Applications: GenAI Evals, LLM-as-a-Judge & RAG Testing

Building Generative AI applications involves moving past deterministic programming into probabilistic systems. Because Large Language Models (LLMs) predict the next token based on probabilities, testing these applications requires a fundamental shift from traditional unit testing to AI-assisted evaluations (Evals). This guide covers the core methodologies, metrics, and architectural patterns required to robustly test and evaluate LLMs, Retrieval-Augmented Generation (RAG) pipelines, and AI agents.

1. The Paradigm Shift: Deterministic vs. Non-Deterministic Code

In traditional software engineering, code is deterministic. For any given input, there is exactly one correct, predictable output.

Real-Life Analogy: The Addition Function

Consider a simple function written to add two integers:

int sum(int a, int b) {
    return a + b;
}

This code is entirely deterministic. We can easily write unit tests with assertions:

• If input is $(4, 5)$, the expected output is $9$.
• If input is $(1, 4)$, the expected output is $5$.

If the function returns anything other than the expected value, the test fails. The assertion is binary: Actual == Expected.

The GenAI Reality: Non-Deterministic Outputs

Now, consider an LLM-powered e-commerce chatbot. We ask it: "What is the return policy?"

Our company's official policy is: "Products may be returned within 30 days of delivery."

Because LLMs are probabilistic, they do not return the exact same string every time. Depending on the model's temperature, system prompts, and random seed, it might return any of the following semantically identical variations:

1. "You can return your product within 30 days of delivery."
2. "Our return window is 30 days after the product is delivered."
3. "Customers have up to 30 days from delivery to initiate a return."

All three responses are correct. However, a traditional assertion using direct string matching (actual_string == expected_string) would fail all three tests. To evaluate GenAI applications, we must assess semantic meaning rather than exact syntax.

2. Why Traditional Unit Testing Fails for GenAI

Relying on manual human verification ("looks good to me") is a common but dangerous pitfall in production-grade AI development.

While manual testing works for 5 to 10 sample queries during development, it fails in production for several reasons:

• Scale: A production system might handle hundreds of queries per minute. Manual verification cannot scale to this volume.
• Regression Risks: Any minor change in the system requires re-testing the entire query space. These changes include:
  • Modifying the system prompt.
  • Upgrading or changing the underlying LLM (e.g., GPT-3.5 to GPT-4o).
  • Adjusting RAG parameters (e.g., changing chunk size or reducing retrieval from Top-5 to Top-3 to decrease latency).
  • Adding, removing, or modifying external tools/APIs.
  • Adjusting model temperature.

Without automated evaluations, a change meant to reduce latency (like reducing RAG retrieval from Top-5 to Top-3) can silently break the model's ability to answer complex queries, leading to silent failures and hallucinations in production.

3. What are AI Evaluations (Evals)?

An AI Evaluation (Eval) is a systematic, repeatable method used to measure whether an AI system behaves as expected. Think of Evals as unit tests for non-deterministic systems.

Instead of asserting exact string equality, an Eval measures whether the output aligns with an expected behavior.

4. The Three Core Pillars of AI Evaluation

To set up a robust evaluation framework, you must define three core components:

1. Test Inputs

These are the queries or prompts fed into your AI application. For an e-commerce bot, this includes questions like "What is your return policy?" or "Can I return shoes after 20 days?"

2. Expected Behavior

This defines what a successful response looks like. It is highly dependent on your application's architecture:

• For Chatbots: The presence of specific facts (e.g., "must mention 30 days").
• For AI Agents: Correct tool invocation (e.g., "must call the calculator tool when math is required").
• For RAG Pipelines: Retrieval of the correct context documents.
• For Structured Output: Adherence to a specific JSON schema.

3. Evaluation Mechanism

The method used to compare the actual response against the expected behavior. The three primary evaluation mechanisms are:

5. Evaluation Datasets vs. Golden Datasets

To run evaluations, you need structured datasets. Two key terms are commonly used in the industry:

Golden Dataset Structures by Application Type

The structure of your Golden Dataset changes depending on what your AI application does:

A. Standard Chatbot

• Query: "What is the return policy?"
• Expected Output: "Products can be returned within 30 days of delivery."

B. AI Agent (Tool Calling)

• Query: "What is $737 + 849$?"
• Expected Tool: calculator
• Expected Arguments: {"num1": 737, "num2": 849}
• Expected Output: 1586

C. RAG (Retrieval-Augmented Generation)

• Query: "How do I return a damaged item?"
• Expected Context/Source Document: doc_id_982_damaged_goods_policy
• Expected Output: "Damaged items must be reported within 48 hours for a full refund."

D. Structured Output (JSON)

• Query: "Extract user info: John Doe is 29 years old."
• Expected Schema:
  JSON

  Copy

  {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "age": { "type": "integer" }
    },
    "required": ["name", "age"]
  }
  

6. Designing a Robust Test Dataset

A high-quality evaluation dataset should not just contain happy-path queries. It must be diverse and include four distinct categories of questions:

Where Do These Datasets Come From?

1. Cold Start (Developer-Curated): Initially, developers, product managers, and domain experts manually write the first 50–100 test cases.
2. Production-Driven (User Queries): Once the application is live, real user queries are captured, anonymized, and added to the evaluation dataset.
3. Synthetic Generation: You can use a powerful LLM (like GPT-4) to generate synthetic questions and edge cases based on your source documents.

7. Deterministic (Code-Based) Evaluation

Whenever possible, use deterministic code to evaluate your AI application. Code-based evaluations are fast, cost $0$ in API tokens, and provide binary, reliable results.

When to Use Code-Based Evaluations

• JSON Validation: If your application must return JSON, use standard code to verify it is valid JSON and matches your schema.
• Tool Calling: Verify if the correct tool was called with the correct arguments.
• HTTP Status Codes: Ensure your API returns a 200 OK status.
• RAG Empty Fallbacks: If no documents are retrieved from the vector database, verify that your code intercepts this and returns a pre-defined fallback message instead of letting the LLM hallucinate.

# Example: Deterministic RAG Fallback Evaluation
def test_rag_fallback_behavior(retrieved_chunks):
    if len(retrieved_chunks) == 0:
        # Deterministic fallback bypasses the LLM entirely
        return "I do not have access to this information in my memory."
    else:
        return call_llm_with_context(retrieved_chunks)

8. Non-Deterministic Evaluation: LLM-as-a-Judge

When evaluating semantic meaning, tone, or open-ended text, deterministic code is insufficient. In these cases, we use a second, highly capable LLM (e.g., GPT-4o or Claude 3.5 Sonnet) to act as an evaluator or "judge."

Why Use a Separate Judge LLM?

• System Prompt Separation: The primary LLM is optimized to act as a helpful assistant. The Judge LLM is given a system prompt optimized purely for critical evaluation, grading, and scoring.
• Structured Grading: The Judge LLM can output its evaluation in a structured JSON format, providing scores and reasoning.

9. The Big Five AI Quality Metrics

When using LLM-as-a-Judge, we evaluate the generator's output against five core metrics:

Deep Dive into the Metrics

1. Relevance

Relevance measures if the model stayed on topic and addressed the user's intent.

• High Relevance:
  • Query: "What is the return policy?"
  • Response: "Our return policy allows returns within 30 days."
• Low Relevance:
  • Query: "What is the return policy?"
  • Response: "Our company was founded in 2019 and we sell high-quality shoes." (The statement is true, but it does not answer the user's question).

2. Correctness

Correctness measures factual accuracy against a known ground truth.

• Query: "What is the return policy?"
• Ground Truth: 30 days.
• Response: "You can return items within 15 days." (This is highly relevant, but factually incorrect).

3. Faithfulness (Critical for RAG)

Faithfulness measures whether the LLM generated its answer strictly using the retrieved context documents, without making outside assumptions.

• Retrieved Context: "Users can return items within 30 days."
• Response: "You can return items within 30 days. Your refund will be processed within 24 hours."
• Evaluation: Low Faithfulness. Even if the 24-hour refund claim is true in reality, it was not in the retrieved context. The LLM hallucinated or brought in outside knowledge.

4. Helpfulness

Helpfulness measures if the response is complete and actionable.

• Query: "Can I return a damaged shoe?"
• Response: "Yes."
• Evaluation: Low Helpfulness. While correct and relevant, a helpful response would explain how to return it, the time limit, and if fees apply.

5. Hallucination

Hallucination measures if the model generated false or unsupported information. This is the inverse of faithfulness but focuses on the injection of incorrect facts.

10. Scoring Methodologies: Rubrics, Pointwise, and Pairwise Evaluations

The Rationale for a 1-to-5 Scoring Scale

When instructing a Judge LLM to score a metric, a $1$ to $5$ Likert scale is highly recommended over $1$ to $10$ or $1$ to $100$.

• On a $1$ to $100$ scale, the difference between $79$ and $81$ is too granular and subjective for an LLM to score consistently.
• A $1$ to $5$ scale provides clear, distinct boundaries:

• 5: Excellent (Perfect alignment, no issues).
• 4: Good (Minor issues, but mostly correct).
• 3: Average (Passable, but missing key details).
• 2: Below Average (Significant errors or omissions).
• 1: Poor (Completely incorrect, irrelevant, or harmful).

Defining Evaluation Rubrics

To get consistent scores from a Judge LLM, you must provide a strict Rubric in its system prompt.

Example: Faithfulness Rubric Prompt

You are an expert judge evaluating LLM responses for Faithfulness.
Rate the response from 1 to 5 based on the following criteria:

- 5: Every factual claim in the response is directly supported by the provided context.
- 4: Almost all claims are supported; any unsupported claim is minor and non-factual.
- 3: The main answer is supported, but some meaningful claims are not found in the context.
- 2: A large portion of the response contains claims not supported by the context.
- 1: The response directly contradicts the context or is entirely fabricated.

Pointwise vs. Pairwise Evaluation

There are two primary ways to structure LLM-as-a-Judge evaluations:

1. Pointwise Evaluation

The Judge LLM evaluates a single response in isolation and assigns a score based on a rubric.

• Pros: Simple, fast, and easy to track historical performance trends.
• Cons: Can suffer from scoring drift if the model's internal calibration shifts.

2. Pairwise Evaluation

The Judge LLM is given a single query along with two different responses (e.g., Response A from GPT-3.5 and Response B from GPT-4o) and must choose which response is better.

• Pros: LLMs are highly accurate at comparative ranking (A vs. B testing).
• Cons: More expensive (requires generating two responses and sending both to the judge) and does not provide an absolute score.

11. Human Evaluation: Handling Subjectivity

While automated Evals are highly scalable, human evaluation remains the gold standard for final validation.

Managing Subjectivity with Averaging

Because human judges are subjective, we use multiple human evaluators and average their scores to find the consensus.

If $N$ human judges score a response, the final score is calculated as:

Where $S_i$ is the score given by the $i$-th judge.

• Pros: Highly reliable, captures subtle nuances in language, tone, and brand alignment.
• Cons: Extremely expensive, slow, and cannot run continuously in a CI/CD pipeline.

12. Offline vs. Online Evaluation Workflows

A complete AI testing strategy splits evaluations into two distinct phases: Offline and Online.

Offline Evaluation (Pre-Production)

• When: Run during development, before code is merged or deployed.
• How: Run your Golden Dataset through the pipeline. Use deterministic code and LLM-as-a-Judge to score the changes.
• Goal: Ensure no regressions in accuracy, relevance, or faithfulness before going live.

Online Evaluation (Post-Production)

• When: Run continuously on live production traffic.
• How:
  • Capture explicit user feedback (e.g., thumbs up/down, star ratings).
  • Run implicit evaluations (e.g., tracking if a user copies a response or closes the chat window).
  • Run real-time guardrails to detect hallucinations or toxic outputs before they reach the user.
• Goal: Monitor real-world performance, detect drift, and collect new edge cases for your offline Golden Dataset.

13. Summary and Key Takeaways

• GenAI is Non-Deterministic: Traditional exact-string assertions fail because LLMs generate semantically identical but syntactically diverse responses.
• Automate to Scale: Manual "looks good to me" testing cannot scale and fails to catch regressions caused by prompt, model, or RAG configuration changes.
• Prioritize Code-Based Evals: Use deterministic code for structured outputs (JSON validation, tool calling, status codes) to save cost and latency.
• Leverage LLM-as-a-Judge: For semantic evaluation, use a separate, highly capable LLM with strict rubrics to grade outputs.
• The Big Five Metrics: Evaluate your system using Relevance, Correctness, Faithfulness, Helpfulness, and Hallucination.
• Use a 1-to-5 Scale: Keep scoring ranges tight ($1$ to $5$) to ensure the Judge LLM provides consistent, meaningful scores.
• Combine Offline and Online Evals: Use offline Golden Datasets in your CI/CD pipeline to prevent regressions, and online monitoring to capture real-world user behavior and edge cases.