Building & Testing an E-Commerce AI Chatbot: RAG, Tools, and AI Evaluations

In modern AI engineering, building a functional Large Language Model (LLM) application is only the first half of the equation; ensuring its continuous accuracy, context-faithfulness, and reliability in production is the critical second half. This chapter covers the end-to-end design, implementation, and multi-dimensional evaluation of an e-commerce customer support chatbot using Retrieval-Augmented Generation (RAG), dynamic function calling (tools), and an evaluation harness that combines deterministic structural checks with qualitative "LLM-as-a-Judge" evaluation pipelines.

In simple generative applications, a client issues an HTTP request to a backend server, which forwards the prompt to an upstream LLM API and directly returns the text response back to the user.

Structure & Concept Flow

Naive Workflow:

Client -> Backend Server -> LLM API -> Backend Server -> Client

In this traditional model, developers operate on "blind faith"—assuming that if the LLM produces a coherent string, the answer is inherently correct. However, enterprise AI applications cannot operate under this assumption. A response might sound articulate while containing hallucinated policies, wrong delivery dates, or improper tool invocations.

To transition from naive prompt wrapping to production-grade engineering, we must treat our AI system as a multi-component pipeline requiring comprehensive evaluation. Testing an enterprise chatbot requires evaluating:

Retrieval Accuracy: Did the vector store retrieve the exact document required to answer the query?
Tool Selection and Execution: Did the LLM invoke the correct function, pass valid arguments, and incorporate the tool's execution results?
Response Quality: Is the generated response faithful to the retrieved context, accurate according to ground truth, helpful, and free of hallucinations?

2. Architectural Paradigms: Knowledge-Based, Tool-Driven, and Hybrid Workflows

Real-world customer support queries fall into three distinct architectural categories based on the nature of the information required to fulfill them.

Structure & Concept Flow

User Query Types

+-----------------------+-----------------------+

| | |

v v v

Type A: Knowledge-Based Type B: Tool-Driven Type C: Hybrid

(Static Policy RAG) (Dynamic Action Tools) (Policy + Live Data)

Type A: Knowledge-Based Questions (Static Policy RAG)

Knowledge-based queries depend on static enterprise documentation (e.g., corporate refund policies, standard return windows, shipping rules). These queries do not change per user or per order.

Mechanism: Vector database similarity search (Retrieval-Augmented Generation).
Evaluation Focus: Document retrieval precision ( $Top-K$ accuracy) and context-faithfulness.

Type B: Tool-Driven Questions (Dynamic Execution)

Dynamic queries depend on live transactional states (e.g., "Where is my order ORD1001?", "What is my expected delivery date?"). Static policy documents cannot answer these questions because the answer lives inside transactional databases or third-party APIs.

Mechanism: Function calling (LLM Tools).
Evaluation Focus: Deterministic tool choice matching, input parameter extraction (e.g., correct parsing of order IDs), and execution output handling.

Type C: Hybrid Questions (Combined Knowledge + Tools)

Complex queries require synthesizing static enterprise rules with real-time transactional status.

Real-Life Analogy: Imagine a customer asking: "I received my order ORD102 20 days ago. Can I still return it?"

To answer this, the chatbot must perform a two-step operation:

Invoke a dynamic OrderStatus Tool to verify that ORD102 was indeed delivered 20 days ago.

Query the vector store for the Return Policy Document to retrieve the static rule (e.g., "Returns are accepted within 30 days of delivery").

Perform temporal reasoning: Since $20 \le 30$ , generate the response confirming return eligibility.

3. System Architecture & Execution Flow

The e-commerce chatbot architecture integrates an embedded Vector Database, dynamic Java-annotated Tooling, a unified execution Tracker, and an LLM orchestration layer (Spring AI / Java engine).

Structure & Concept Flow

User / Postman / Evaluation Pipeline

| 1. POST /chat { message: "When will ORD1001 arrive?" }

SupportController

| 2. Delegates query

SupportService

+---> 3. VectorStore.similaritySearch(query)

| |

| v 4. Returns top-matching policy document (e.g., shipping-policy.txt)

+---> 5. ChatClient.prompt() with User Query + Retrieved Context + Available Tools

+---> 6. LLM analyzes query and determines tool requirement

| |

| v 7. LLM requests invocation: DeliveryEstimateTool(orderId="ORD1001")

+---> 8. SupportService executes DeliveryEstimateTool

+---> 9. Records execution event in ToolTracker

v 10. Returns raw tool result to LLM

v 11. LLM synthesizes context, tool result, and query into final text

AgentResponse (Constructed with Answer + Full Trace Metadata)

Caller / Evaluation Harness

4. The Role of Observability: Capturing Execution Traces

Evaluating an AI application requires more than just inspecting the final text string returned to the user. If an LLM returns "Yes, your order can be returned," we cannot verify whether it made a lucky guess, retrieved the correct document, or invoked the correct function without accessing internal system states.

These internal operational details are called Execution Traces.

Execution Trace Schema (`AgentResponse`)

To enable deterministic and automated qualitative testing, the backend service must encapsulate both the generated answer and the full trace of execution inside a structured response schema:

Trace Parameter	Type	Purpose / Description
`answer`	`String`	The final natural language response generated by the LLM for the end user.
`retrievedDocuments`	`List<String>`	The list of source metadata identifiers (e.g., `return-policy.txt`) fetched from the vector store.
`toolsUsed`	`List<String>`	The exact function/tool identifiers executed during prompt fulfillment.
`toolArguments`	`Map<String, String>`	The parsed parameters passed by the LLM to each tool (e.g., `{"getDeliveryEstimate": "ORD1001"}`).
`toolResults`	`Map<String, String>`	The exact string responses returned by the backend tool execution.
`context`	`String`	The full prompt context injected into the system prompt from retrieved vector chunks.

5. Building the E-Commerce Agent: RAG & Tool Integration

The implementation consists of static Knowledge Base files, isolated Java Function Tools, a central ToolTracker for trace collection, and the core SupportService.

5.1 Static Knowledge Base Files

The enterprise knowledge base consists of plain-text documents placed in the application resources (/resources/knowledge/):

refund-policy.txt: Defines refund processing windows (e.g., refunds credited within 5–7 business days).
return-policy.txt: Specifies return eligibility windows (e.g., products can be returned within 30 days of delivery if unused and in original packaging).
shipping-policy.txt: Outlines standard transit times (e.g., standard shipping takes 3–5 business days).

5.2 Dynamic Tools with Trace Tracking

To capture execution traces without altering production business logic, each tool receives a shared ToolTracker instance that records call metadata upon execution.

Tool Tracker Implementation

JAVA

package com.coderarmy.support.tools;

import java.util.*;

public class ToolTracker {
    private final List<String> toolsUsed = new ArrayList<>();
    private final Map<String, String> toolArguments = new HashMap<>();
    private final Map<String, String> toolResults = new HashMap<>();

    public synchronized void record(String toolName, String argument, String result) {
        if (!toolsUsed.contains(toolName)) {
            toolsUsed.add(toolName);
        }
        toolArguments.put(toolName, argument);
        toolResults.put(toolName, result);
    }

    public synchronized void clear() {
        toolsUsed.clear();
        toolArguments.clear();
        toolResults.clear();
    }

    public List<String> getToolsUsed() { return new ArrayList<>(toolsUsed); }
    public Map<String, String> getToolArguments() { return new HashMap<>(toolArguments); }
    public Map<String, String> getToolResults() { return new HashMap<>(toolResults); }
}

Delivery Estimate Tool Implementation

JAVA

package com.coderarmy.support.tools;

import org.springframework.ai.tool.annotation.Tool;
import java.util.Locale;

public class DeliveryEstimateTool {
    private final ToolTracker toolTracker;

    public DeliveryEstimateTool(ToolTracker toolTracker) {
        this.toolTracker = toolTracker;
    }

    @Tool(description = "Get the expected delivery date for an order given its order ID")
    public String getDeliveryEstimate(String orderId) {
        String normalizedId = orderId != null ? orderId.trim().toUpperCase(Locale.ROOT) : "";
        String result;

        if ("ORD1001".equals(normalizedId)) {
            result = "Order ORD1001 is expected to be delivered on October 15, 2026.";
        } else if ("ORD102".equals(normalizedId)) {
            result = "Order ORD102 was delivered on September 20, 2026.";
        } else if ("ORD103".equals(normalizedId)) {
            result = "Order ORD103 is scheduled for delivery tomorrow.";
        } else {
            result = "Delivery information not found for order ID: " + orderId;
        }

        toolTracker.record("getDeliveryEstimate", normalizedId, result);
        return result;
    }
}

5.3 Agent Support Service

The SupportService class manages vector storage, document indexing, context construction, tool registration, and trace aggregation into an AgentResponse.

JAVA

package com.coderarmy.support.service;

import com.coderarmy.support.model.AgentResponse;
import com.coderarmy.support.tools.ToolTracker;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.*;

@Service
public class SupportService {

    private final ChatClient chatClient;
    private final VectorStore vectorStore;
    private final ToolTracker toolTracker;
    private final Object deliveryEstimateTool;
    private final Object orderStatusTool;
    private final Object refundAmountTool;
    private final Resource[] knowledgeResources;

    public SupportService(ChatClient.Builder chatClientBuilder,
                          VectorStore vectorStore,
                          ToolTracker toolTracker,
                          Object deliveryEstimateTool,
                          Object orderStatusTool,
                          Object refundAmountTool,
                          Resource[] knowledgeResources) {
        this.chatClient = chatClientBuilder.build();
        this.vectorStore = vectorStore;
        this.toolTracker = toolTracker;
        this.deliveryEstimateTool = deliveryEstimateTool;
        this.orderStatusTool = orderStatusTool;
        this.refundAmountTool = refundAmountTool;
        this.knowledgeResources = knowledgeResources;
    }

    public void loadKnowledgeBase() throws Exception {
        List<Document> documents = new ArrayList<>();
        for (Resource resource : knowledgeResources) {
            String filename = resource.getFilename();
            try (InputStream is = resource.getInputStream()) {
                String content = new String(is.readAllBytes(), StandardCharsets.UTF_8);
                Map<String, Object> metadata = new HashMap<>();
                metadata.put("source", filename);
                documents.add(new Document(content, metadata));
            }
        }
        vectorStore.add(documents);
        System.out.println("Knowledge Base Successfully Loaded into Vector Store.");
    }

    public AgentResponse chat(String userMessage) {
        toolTracker.clear();

        // 1. Similarity Search in Vector Store
        List<Document> similarDocs = vectorStore.similaritySearch(userMessage);
        List<String> retrievedDocs = new ArrayList<>();
        StringBuilder contextBuilder = new StringBuilder();

        for (Document doc : similarDocs) {
            String source = (String) doc.getMetadata().getOrDefault("source", "unknown");
            retrievedDocs.add(source);
            contextBuilder.append("Source: ").append(source).append("\n")
                          .append("Content: ").append(doc.getContent()).append("\n\n");
        }

        String retrievedContext = contextBuilder.toString();

        // 2. System Prompt Engineering
        String systemPrompt = """
            You are an official E-Commerce Customer Support Agent.
            Use the provided policy context and available tools to answer customer questions accurately.
            
            Policy Context:
            %s
            """.formatted(retrievedContext);

        // 3. Execution via Chat Client with Tools
        String rawAnswer = chatClient.prompt()
                .system(systemPrompt)
                .user(userMessage)
                .tools(deliveryEstimateTool, orderStatusTool, refundAmountTool)
                .call()
                .content();

        // 4. Construct Structured Response with Traces
        AgentResponse response = new AgentResponse();
        response.setAnswer(rawAnswer);
        response.setRetrievedDocuments(retrievedDocs);
        response.setToolsUsed(toolTracker.getToolsUsed());
        response.setToolArguments(toolTracker.getToolArguments());
        response.setToolResults(toolTracker.getToolResults());
        response.setContext(retrievedContext);

        return response;
    }
}

6. Structuring the Golden Dataset

To evaluate an AI application, we use a Golden Dataset: a curated test set containing domain queries, ground-truth answers, expected behaviors, expected tool choices, and expected document sources.

Golden Dataset Schema (`golden-dataset.json`)

JSON

[
  {
    "id": "RAG-RET-01",
    "question": "How many days do I have to return a product?",
    "expectedAnswer": "Products can be returned within 30 days of delivery.",
    "expectedBehavior": "Explain that products must be unused and in original packaging, returnable within 30 days.",
    "expectedTool": null,
    "expectedArgument": null,
    "expectedSource": "return-policy.txt"
  },
  {
    "id": "TOOL-STAT-01",
    "question": "Where is my order ORD1001?",
    "expectedAnswer": "Order ORD1001 is expected to be delivered on October 15, 2026.",
    "expectedBehavior": "Invoke the delivery estimate tool and report the exact delivery date.",
    "expectedTool": "getDeliveryEstimate",
    "expectedArgument": "ORD1001",
    "expectedSource": null
  },
  {
    "id": "COMBINED-01",
    "question": "I received order ORD102 three weeks ago. Can I still return it?",
    "expectedAnswer": "Yes, order ORD102 was delivered 21 days ago, which is within the 30-day return window.",
    "expectedBehavior": "Fetch order status using tool and cross-reference return policy document to verify eligibility.",
    "expectedTool": "getOrderStatus",
    "expectedArgument": "ORD102",
    "expectedSource": "return-policy.txt"
  }
]

7. Evaluation Strategies: Deterministic Rules vs. LLM-as-a-Judge

Testing AI applications requires a dual-layer evaluation harness combining deterministic code checks with non-deterministic model checks.

Structure & Concept Flow

Test Execution

+-------------------------+-------------------------+

| |

v v

Deterministic Evaluation Qualitative Evaluation

(Code-Based Structural Verification) (LLM-as-a-Judge Pipeline)

- Response Existence - Relevance Score [1-5]

- Exact Tool Matching - Correctness Score [1-5]

- Argument Exactness - Faithfulness Score [1-5]

- Source File Matching - Helpfulness Score [1-5]

- Hallucination-Free Score [1-5]

7.1 Deterministic Evaluation Layer

Deterministic evaluations run programmatic assertions against execution traces. They are fast, reproducible, cost-free, and objective.

\begin{aligned} Pass_{deterministic} &= Pass_{response} \land Pass_{tool} \land Pass_{arg} \land Pass_{source} \end{aligned}

Response Existence Check ( $Pass_{response}$ ): Asserts that the response string is non-null and non-empty.
Tool Selection Matching ( $Pass_{tool}$ ):
- If expectedTool == null, asserts that toolsUsed is empty.
- If expectedTool != null, asserts that toolsUsed.contains(expectedTool).
Argument Exactness Check ( $Pass_{arg}$ ): Asserts that toolArguments.get(expectedTool) matches expectedArgument.
Source Retrieval Matching ( $Pass_{source}$ ):
- If expectedSource != null, asserts that retrievedDocuments.contains(expectedSource).

7.2 Qualitative Evaluation Layer (LLM-as-a-Judge)

Deterministic checks verify operational mechanics (e.g., whether a tool was called), but they cannot evaluate natural language properties such as tone, completeness, or factual accuracy. To evaluate these properties, we pass the execution trace to an isolated Judge LLM.

Evaluation Dimensions

The Judge LLM evaluates the system response across 5 dimensions on a discrete scale of $[1, 5]$ :

Relevance: Does the generated answer directly address the user's specific query?
Correctness: Is the answer factually aligned with the ground-truth expectedAnswer?
Faithfulness: Is the response strictly derived from the provided context without inventing facts?
Helpfulness: Is the answer clear, professional, and actionable for a customer?
Hallucination-Free: Is the response completely devoid of ungrounded or speculative statements?

Mathematical Formulation & Quality Thresholds

Let $M$ be the set of qualitative evaluation metrics:

M = \{\text{Relevance}, \text{Correctness}, \text{Faithfulness}, \text{Helpfulness}, \text{HallucinationFree}\}

For a given test case $i$ , the qualitative evaluation passes if and only if every dimension meets or exceeds a predefined quality threshold $\tau$ (typically $\tau = 4$ ):

\begin{aligned} Pass_{quality}^{(i)} &= \bigwedge_{m \in M} \left( Score_{m}^{(i)} \ge \tau \right) \end{aligned}

The overall evaluation condition for test case $i$ requires passing both evaluation layers:

\begin{aligned} Pass_{final}^{(i)} &= Pass_{deterministic}^{(i)} \land Pass_{quality}^{(i)} \end{aligned}

For a test suite containing $N$ golden dataset evaluation cases, aggregate domain metrics are calculated as follows:

\begin{aligned} Score_{avg}(m) &= \frac{1}{N} \sum_{i=1}^{N} Score_{m}^{(i)} \quad \forall m \in M \end{aligned}

8. Implementation: The Evaluation Pipeline in Code

The automated test harness loads the Golden Dataset, executes queries against the backend service, runs deterministic trace assertions, calls the Judge LLM for qualitative scoring, and aggregates test pass rates.

8.1 Judge Response Schema (`JudgeResult`)

JAVA

package com.coderarmy.support.eval;

public class JudgeResult {
    private int relevanceScore;
    private int correctnessScore;
    private int faithfulnessScore;
    private int helpfulnessScore;
    private int hallucinationFreeScore;
    private String reasoning;

    // Getters and Setters
    public int getRelevanceScore() { return relevanceScore; }
    public void setRelevanceScore(int relevanceScore) { this.relevanceScore = relevanceScore; }
    public int getCorrectnessScore() { return correctnessScore; }
    public void setCorrectnessScore(int correctnessScore) { this.correctnessScore = correctnessScore; }
    public int getFaithfulnessScore() { return faithfulnessScore; }
    public void setFaithfulnessScore(int faithfulnessScore) { this.faithfulnessScore = faithfulnessScore; }
    public int getHelpfulnessScore() { return helpfulnessScore; }
    public void setHelpfulnessScore(int helpfulnessScore) { this.helpfulnessScore = helpfulnessScore; }
    public int getHallucinationFreeScore() { return hallucinationFreeScore; }
    public void setHallucinationFreeScore(int hallucinationFreeScore) { this.hallucinationFreeScore = hallucinationFreeScore; }
    public String getReasoning() { return reasoning; }
    public void setReasoning(String reasoning) { this.reasoning = reasoning; }
}

8.2 Full Automated Test Harness

JAVA

package com.coderarmy.support.eval;

import com.coderarmy.support.model.AgentResponse;
import com.coderarmy.support.model.EvalCase;
import com.coderarmy.support.service.SupportService;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import java.io.InputStream;
import java.util.*;

import static org.junit.jupiter.api.Assertions.*;

@SpringBootTest
public class SupportAgentEvaluationTest {

    @Autowired
    private SupportService supportService;

    @Autowired
    private ChatClient.Builder chatClientBuilder;

    private final ObjectMapper objectMapper = new ObjectMapper();

    @Test
    public void runGoldenDatasetEvaluations() throws Exception {
        // 1. Load Golden Dataset
        InputStream is = getClass().getResourceAsStream("/golden-dataset.json");
        List<EvalCase> evalCases = objectMapper.readValue(is, new TypeReference<List<EvalCase>>() {});

        ChatClient judgeClient = chatClientBuilder.build();
        List<String> failures = new ArrayList<>();

        double totalRelevance = 0;
        double totalCorrectness = 0;
        double totalFaithfulness = 0;
        double totalHelpfulness = 0;
        double totalHallucinationFree = 0;

        // 2. Iterate through Test Cases
        for (EvalCase evalCase : evalCases) {
            System.out.println("Executing Test Case: " + evalCase.getId());

            // Execute System
            AgentResponse response = supportService.chat(evalCase.getQuestion());

            // Deterministic Assertions
            boolean responsePass = response.getAnswer() != null && !response.getAnswer().isBlank();
            
            boolean toolPass;
            if (evalCase.getExpectedTool() == null) {
                toolPass = response.getToolsUsed().isEmpty();
            } else {
                toolPass = response.getToolsUsed().contains(evalCase.getExpectedTool());
            }

            boolean argPass = true;
            if (evalCase.getExpectedArgument() != null) {
                String actualArg = response.getToolArguments().get(evalCase.getExpectedTool());
                argPass = evalCase.getExpectedArgument().equalsIgnoreCase(actualArg);
            }

            boolean sourcePass = true;
            if (evalCase.getExpectedSource() != null) {
                sourcePass = response.getRetrievedDocuments().contains(evalCase.getExpectedSource());
            }

            boolean deterministicPass = responsePass && toolPass && argPass && sourcePass;

            // Qualitative LLM-as-a-Judge Evaluation
            JudgeResult judgeResult = evaluateWithJudge(judgeClient, evalCase, response);

            totalRelevance += judgeResult.getRelevanceScore();
            totalCorrectness += judgeResult.getCorrectnessScore();
            totalFaithfulness += judgeResult.getFaithfulnessScore();
            totalHelpfulness += judgeResult.getHelpfulnessScore();
            totalHallucinationFree += judgeResult.getHallucinationFreeScore();

            boolean qualityPass = judgeResult.getRelevanceScore() >= 4 &&
                                  judgeResult.getCorrectnessScore() >= 4 &&
                                  judgeResult.getFaithfulnessScore() >= 4 &&
                                  judgeResult.getHelpfulnessScore() >= 4 &&
                                  judgeResult.getHallucinationFreeScore() >= 4;

            boolean finalPass = deterministicPass && qualityPass;

            if (!finalPass) {
                failures.add(evalCase.getId() + " - DeterministicPass: " + deterministicPass + ", QualityPass: " + qualityPass);
            }
        }

        // 3. Compute and Output Aggregate Statistics
        int totalCases = evalCases.size();
        System.out.println("=== EVALUATION SUMMARY ===");
        System.out.println("Total Test Cases: " + totalCases);
        System.out.println("Failed Cases: " + failures.size());
        System.out.printf("Average Relevance Score: %.2f\n", totalRelevance / totalCases);
        System.out.printf("Average Correctness Score: %.2f\n", totalCorrectness / totalCases);
        System.out.printf("Average Faithfulness Score: %.2f\n", totalFaithfulness / totalCases);
        System.out.printf("Average Helpfulness Score: %.2f\n", totalHelpfulness / totalCases);
        System.out.printf("Average Hallucination-Free Score: %.2f\n", totalHallucinationFree / totalCases);

        assertTrue(failures.isEmpty(), "Evaluation suite failed for cases: " + failures);
    }

    private JudgeResult evaluateWithJudge(ChatClient judgeClient, EvalCase evalCase, AgentResponse response) {
        String judgeUserPrompt = """
            Evaluate the following AI application response based on ground truth and execution traces.
            
            User Question: %s
            Expected Answer: %s
            Expected Behavior: %s
            Retrieved Context: %s
            Tools Executed: %s
            Actual AI Response: %s
            
            Score each dimension from 1 (poor) to 5 (excellent):
            1. Relevance: Directness to user query.
            2. Correctness: Factual alignment with expected answer.
            3. Faithfulness: Strict adherence to provided context.
            4. Helpfulness: Clarity and operational utility.
            5. HallucinationFree: Absence of ungrounded or invented claims.
            
            Provide structured output matching the JudgeResult schema along with reasoning.
            """.formatted(
                evalCase.getQuestion(),
                evalCase.getExpectedAnswer(),
                evalCase.getExpectedBehavior(),
                response.getContext(),
                response.getToolsUsed(),
                response.getAnswer()
            );

        return judgeClient.prompt()
                .system("You are an impartial, strict evaluator of AI application performance.")
                .user(judgeUserPrompt)
                .call()
                .entity(JudgeResult.class);
    }
}

9. Real-World Pitfalls, Debugging, and Failure Modes

During testing, evaluations can fail due to several distinct operational edge cases.

1. Hallucination in Tool Invocations

Failure Mode: The LLM bypasses an available dynamic tool and guesses a parameter value or delivery status based on statistical priors.
Root Cause: Poor system prompt instructions or weak tool description annotations.
Remediation: Explicitly instruct the system prompt: "Never guess order details. If an order ID is present, you MUST call the appropriate tracking tool before formulating a response."

2. Retriever Misalignment

Failure Mode: The vector database fails to return the required policy file due to semantic distance issues or poor vector representations.
Root Cause: Oversized chunks or lack of document metadata.
Remediation: Use small, focused document chunks, add document metadata (e.g., source: return-policy.txt), and optimize similarity search thresholds or $Top-K$ parameters.

3. Judge LLM Non-Determinism & Flakiness

Failure Mode: A test case fails on one run but succeeds on a re-run due to variance in the Judge LLM's scoring.
Root Cause: High temperature settings or ambiguous scoring rubrics in the judge's prompt.
Remediation: Set the Judge LLM temperature to $0.0$ , enforce structured JSON response schemas (.entity(JudgeResult.class)), and define clear scoring guidelines in the prompt.

10. Summary and Key Takeaways

Beyond Text-Only Evaluation: Evaluating production LLM systems requires inspecting operational execution traces (toolsUsed, toolArguments, retrievedDocuments, context), not just the raw response string.
Trace Extraction Pattern: Capture internal tool execution metrics by injecting state trackers (e.g., ToolTracker) into custom dynamic tools.
Dual-Layer Evaluation Harness: Combine fast, cost-free deterministic checks (structural existence, exact tool matching, argument equality, source matching) with qualitative LLM-as-a-Judge evaluation pipelines.
Three Functional Query Types: E-commerce customer support systems must handle static policy queries via RAG (Type A), dynamic live transactional queries via tools (Type B), and combined hybrid queries requiring policy reasoning over live tool data (Type C).
The Golden Dataset: A well-structured test set containing ground-truth answers, expected behaviors, required tool choices, arguments, and source documents is essential for automated testing.
Five Qualitative Evaluation Dimensions: Measure response quality across Relevance, Correctness, Faithfulness, Helpfulness, and Hallucination-Free scores using structured output schemas (JudgeResult).
Deterministic Judgments First: Perform code-based deterministic assertions before running LLM-based qualitative evaluations to reduce evaluation latency and API costs.

Building & Testing an E-Commerce AI Chatbot | RAG, Tools & AI Evals | FDE Full Course #18

Building & Testing an E-Commerce AI Chatbot: RAG, Tools, and AI Evaluations

1. The Evolution of AI Chatbot Testing: From Blind Faith to Observability

2. Architectural Paradigms: Knowledge-Based, Tool-Driven, and Hybrid Workflows

Type A: Knowledge-Based Questions (Static Policy RAG)

Type B: Tool-Driven Questions (Dynamic Execution)

Type C: Hybrid Questions (Combined Knowledge + Tools)

3. System Architecture & Execution Flow

4. The Role of Observability: Capturing Execution Traces

Execution Trace Schema (`AgentResponse`)

5. Building the E-Commerce Agent: RAG & Tool Integration

5.1 Static Knowledge Base Files

5.2 Dynamic Tools with Trace Tracking

Tool Tracker Implementation

Delivery Estimate Tool Implementation

5.3 Agent Support Service

6. Structuring the Golden Dataset

Golden Dataset Schema (`golden-dataset.json`)

7. Evaluation Strategies: Deterministic Rules vs. LLM-as-a-Judge

7.1 Deterministic Evaluation Layer

7.2 Qualitative Evaluation Layer (LLM-as-a-Judge)

Evaluation Dimensions

Mathematical Formulation & Quality Thresholds

8. Implementation: The Evaluation Pipeline in Code

8.1 Judge Response Schema (`JudgeResult`)

8.2 Full Automated Test Harness

9. Real-World Pitfalls, Debugging, and Failure Modes

1. Hallucination in Tool Invocations

2. Retriever Misalignment

3. Judge LLM Non-Determinism & Flakiness

10. Summary and Key Takeaways

Building & Testing an E-Commerce AI Chatbot | RAG, Tools & AI Evals | FDE Full Course #18

Building & Testing an E-Commerce AI Chatbot: RAG, Tools, and AI Evaluations

1. The Evolution of AI Chatbot Testing: From Blind Faith to Observability

2. Architectural Paradigms: Knowledge-Based, Tool-Driven, and Hybrid Workflows

Type A: Knowledge-Based Questions (Static Policy RAG)

Type B: Tool-Driven Questions (Dynamic Execution)

Type C: Hybrid Questions (Combined Knowledge + Tools)

3. System Architecture & Execution Flow

4. The Role of Observability: Capturing Execution Traces

Execution Trace Schema (AgentResponse)

5. Building the E-Commerce Agent: RAG & Tool Integration

5.1 Static Knowledge Base Files

5.2 Dynamic Tools with Trace Tracking

Tool Tracker Implementation

Delivery Estimate Tool Implementation

5.3 Agent Support Service

6. Structuring the Golden Dataset

Golden Dataset Schema (golden-dataset.json)

7. Evaluation Strategies: Deterministic Rules vs. LLM-as-a-Judge

7.1 Deterministic Evaluation Layer

7.2 Qualitative Evaluation Layer (LLM-as-a-Judge)

Evaluation Dimensions

Mathematical Formulation & Quality Thresholds

8. Implementation: The Evaluation Pipeline in Code

8.1 Judge Response Schema (JudgeResult)

8.2 Full Automated Test Harness

9. Real-World Pitfalls, Debugging, and Failure Modes

1. Hallucination in Tool Invocations

2. Retriever Misalignment

3. Judge LLM Non-Determinism & Flakiness

10. Summary and Key Takeaways

Execution Trace Schema (`AgentResponse`)

Golden Dataset Schema (`golden-dataset.json`)

8.1 Judge Response Schema (`JudgeResult`)