RAG Explained with a Real Project: Build an AI Customer Support Chatbot

Large Language Models (LLMs) possess vast general knowledge but remain fundamentally blind to private corporate data, evolving product catalogs, and proprietary business policies. Retrieval-Augmented Generation (RAG) bridges this enterprise gap by coupling external knowledge retrieval systems with generative models, enabling accurate, context-grounded, and auditable responses without costly model retraining.

1. The Enterprise LLM Dilemma: Why Raw LLMs Fail in Business

When deploying general-purpose foundational models (e.g., GPT-4, Claude, Llama) inside enterprise customer-facing environments—such as an Amazon-scale customer support workflow—three critical limitations arise:

1. Knowledge Cutoffs: Foundational models are frozen in time at the end of their pre-training phase. They have no knowledge of events, inventory updates, or policy changes that occurred post-cutoff.
2. Lack of Private/Proprietary Context: A base LLM does not have access to an enterprise's internal refund documentation, internal ticketing records, user order histories, or private shipping SLAs.
3. Hallucination Under Uncertainty: LLMs are probabilistic token-prediction engines, not factual databases. When asked a hyper-specific question regarding a private return policy (e.g., "What is the return window for an opened consumer electronics item purchased during Black Friday?"), an ungrounded LLM may hallucinate a plausible-sounding but legally inaccurate answer.

To make an LLM an effective customer support agent, the model must be provided with the exact, authoritative company documents relevant to the user's specific inquiry at inference time.

2. Real-Life Analogy: The Closed-Book vs. Open-Book Exam

To understand why RAG is the enterprise standard for contextual AI, consider how humans take examinations:

Real-Life Analogy: The Open-Book Exam

• Standard Base LLM = Closed-Book Exam: The student must rely purely on their biological memory. If they never studied your internal business handbook, or if the rule changed yesterday, they will either fail the question or fabricate an answer based on general assumptions.
• Fine-Tuning = Intensive Cramming/Retraining: The student spends weeks memorizing every page of your corporate manuals. However, if your return policy changes tomorrow morning, their memorized knowledge is outdated unless you re-enroll them in training school.
• RAG = Open-Book Exam with an Expert Research Assistant: The student sits at the desk with their strong analytical and reasoning capabilities intact. Before answering a question, a research assistant immediately fetches the exact two paragraphs from the company policy binder that cover the topic. The student reads those two paragraphs and synthesizes a direct, factually accurate answer.

3. Fine-Tuning vs. Retrieval-Augmented Generation (RAG)

Engineering teams frequently debate whether to fine-tune an open-source model on internal documentation or implement a RAG pipeline. In production customer support environments, RAG is overwhelmingly preferred for factual knowledge injection.

Note: A common enterprise pattern is a hybrid approach: use RAG to supply factual data, and use fine-tuning if you need the model to adhere strictly to proprietary JSON schemas or a distinct corporate voice.

4. The Two Core Phases of RAG

Every RAG architecture consists of two distinct, decoupled pipelines operating at different cadences: the Ingestion Pipeline (offline or asynchronous) and the Query/Generation Pipeline (real-time, synchronous).

5. Phase 1 Deep-Dive: The Ingestion Pipeline

The ingestion pipeline converts unstructured organizational documentation (PDFs, Markdown files, Confluence pages, Word documents) into a searchable vector index.

1. Document Loading

Raw files are parsed to extract clean string content while preserving document-level metadata (source path, document version, department owner, date updated).

2. Chunking Strategies

LLMs and embedding models have finite context windows. Furthermore, embedding a 100-page document into a single vector dilutes specific factual details. Therefore, documents must be split into manageable pieces called chunks.

• Chunk Size: The number of tokens or characters per chunk (e.g., 500 characters or 256 tokens).
• Chunk Overlap: The sliding window margin shared between consecutive chunks (typically 10%–20%, e.g., 50 characters). Overlap ensures that a sentence or semantic concept split across an arbitrary boundary retains its context in at least one chunk.

3. Vector Embeddings

An embedding model (such as text-embedding-3-small, text-embedding-ada-002, or an open-source model like all-MiniLM-L6-v2) converts a text chunk into a dense vector of floating-point numbers:

where $d$ represents the dimensionality of the embedding space (e.g., $d = 1536$ for OpenAI embeddings, or $d = 384$ for MiniLM). Chunks with similar semantic meaning are mapped to spatial locations close to one another within this $d$-dimensional space, regardless of whether they share the exact same vocabulary.

6. Mathematical Foundations of Semantic Search

Traditional databases rely on lexical keyword matching (such as SQL LIKE queries or BM25/Elasticsearch full-text search). Keyword search fails when a user asks:

"How do I get my cash back for a damaged package?"

If the document states:

"Reimbursements for defective items are processed within 5 business days,"

a strict lexical query will miss the match because the words cash back and reimbursement, or damaged and defective, do not match lexically. Vector embeddings solve this by encoding semantic relationships.

Cosine Similarity

The standard metric used to measure semantic closeness between a query vector $\vec{q}$ and a document chunk vector $\vec{d}$ is Cosine Similarity, which computes the cosine of the angle $\theta$ between them:

• A score of $1.0$ indicates identical semantic direction.
• A score of $0.0$ indicates orthogonality (no semantic relationship).
• A score of $-1.0$ indicates opposite semantic orientation.

If vectors are normalized to unit length ($\|\vec{q}\| = 1$ and $\|\vec{d}\| = 1$), the cosine similarity simplifies directly to the dot product:

Computational Complexity

For a collection of $N$ document chunks in a vector store, a brute-force exact search compares the query against every chunk, yielding a time complexity of:

To handle millions of chunks at interactive latencies ($\le 50\text{ ms}$), vector databases implement Approximate Nearest Neighbor (ANN) indexing algorithms, such as:

• HNSW (Hierarchical Navigable Small World): Graph-based indexing offering $\mathcal{O}(\log N)$ search complexity.
• IVF (Inverted File Index): Clustering-based indexing that partitions the vector space into Voronoi cells.

7. Vector Databases: Architecture and Options

A vector database is purpose-built to persist high-dimensional vectors, execute low-latency similarity queries, and manage associated payload metadata (e.g., raw text, source URL, timestamps).

Vector Store Selection Matrix

8. Phase 2 Deep-Dive: Query and Retrieval-Augmented Generation Flow

When an end customer submits a query to the support chatbot, the system executes the following deterministic sequence:

9. System Architecture: Building an AI Support Chatbot with Spring AI

Spring AI brings familiar enterprise-grade software engineering patterns (dependency injection, portable interfaces, templated configuration) to AI application development, mirroring Spring's portable abstraction model across various LLM and Vector Store providers.

10. End-to-End Implementation Walkthrough

The following enterprise implementation demonstrates how to build an Amazon-style customer support chatbot using Spring Boot 3 and Spring AI.

Step 1: Project Dependencies (pom.xml)

Add the core Spring AI dependencies to your Maven configuration:

<dependencies>
    <!-- Spring Boot Web Starter for REST Endpoints -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <!-- Spring AI OpenAI Starter (Handles Chat & Embeddings) -->
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
        <version>1.0.0-M1</version>
    </dependency>

    <!-- Test dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Step 2: Application Configuration (application.yml)

Configure the model properties and your API credentials:

spring:
  application:
    name: ai-customer-support-service
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o-mini
          temperature: 0.1 # Low temperature ensures deterministic, factual output
      embedding:
        options:
          model: text-embedding-3-small

Step 3: Document Ingestion Service

This service loads raw support policy documents, chunks them into uniform blocks, generates vector embeddings, and stores them in the vector database on application startup.

package com.fde.support.service;

import jakarta.annotation.PostConstruct;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.document.Document;
import org.springframework.ai.reader.TextReader;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class DocumentIngestionService {

    private static final Logger log = LoggerFactory.getLogger(DocumentIngestionService.class);

    private final VectorStore vectorStore;

    @Value("classpath:data/amazon_support_policy.txt")
    private Resource supportPolicyResource;

    public DocumentIngestionService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    @PostConstruct
    public void ingestSupportDocuments() {
        log.info("Starting ingestion of customer support documentation...");

        // 1. Load document text
        TextReader textReader = new TextReader(supportPolicyResource);
        textReader.getCustomMetadata().put("source", "amazon_support_policy.txt");
        List<Document> rawDocuments = textReader.get();

        // 2. Split documents into chunks (e.g., 400 tokens per chunk with 50-token overlap)
        TokenTextSplitter splitter = new TokenTextSplitter(400, 50, 5, 10000, true);
        List<Document> chunkedDocuments = splitter.apply(rawDocuments);

        log.info("Document segmented into {} chunks. Generating embeddings...", chunkedDocuments.size());

        // 3. Write chunks to the VectorStore (Embedding is automatically invoked)
        vectorStore.accept(chunkedDocuments);

        log.info("Ingestion complete. Documents indexed in vector store.");
    }
}

Step 4: Customer Support RAG Query Service

This component handles user inquiries by querying the vector store for the most relevant context and assembling the prompt for the LLM.

package com.fde.support.service;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.chat.prompt.PromptTemplate;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

@Service
public class CustomerSupportService {

    private final VectorStore vectorStore;
    private final ChatClient chatClient;

    private static final String SYSTEM_PROMPT_TEMPLATE = """
        You are a helpful, professional customer support agent for our e-commerce platform.
        Answer the customer's question using ONLY the provided context below.
        If the answer cannot be found in the context, politely state:
        "I am sorry, but I do not have access to that information in our current policy documentation. Please contact a human agent."
        
        Strict Rules:
        - Do not speculate or extrapolate beyond the provided text.
        - Do not mention that you received "context" or "documents". Speak naturally as the company representative.
        
        CONTEXT:
        {context}
        
        CUSTOMER QUESTION:
        {question}
        """;

    public CustomerSupportService(VectorStore vectorStore, ChatClient.Builder chatClientBuilder) {
        this.vectorStore = vectorStore;
        this.chatClient = chatClientBuilder.build();
    }

    public String handleCustomerQuery(String customerQuestion) {
        // 1. Retrieve the top 3 most relevant chunks using similarity search
        SearchRequest searchRequest = SearchRequest.query(customerQuestion)
                .withTopK(3)
                .withSimilarityThreshold(0.70); // Discard low-confidence matches

        List<Document> matchedDocuments = vectorStore.similaritySearch(searchRequest);

        // 2. Concatenate retrieved chunks into a single context payload
        String aggregatedContext = matchedDocuments.stream()
                .map(Document::getContent)
                .collect(Collectors.joining("\n---\n"));

        // 3. Populate prompt template variables
        PromptTemplate promptTemplate = new PromptTemplate(SYSTEM_PROMPT_TEMPLATE);
        Prompt prompt = promptTemplate.create(Map.of(
                "context", aggregatedContext.isEmpty() ? "No relevant policy documents found." : aggregatedContext,
                "question", customerQuestion
        ));

        // 4. Request grounded completion from LLM
        return chatClient.prompt(prompt)
                .call()
                .content();
    }
}

Step 5: Web Layer REST Controller

Expose a simple, clean REST interface to client applications:

package com.fde.support.controller;

import com.fde.support.service.CustomerSupportService;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api/v1/support")
public class SupportChatController {

    private final CustomerSupportService supportService;

    public SupportChatController(CustomerSupportService supportService) {
        this.supportService = supportService;
    }

    public record ChatRequest(String question) {}
    public record ChatResponse(String answer) {}

    @PostMapping("/chat")
    public ResponseEntity<ChatResponse> chat(@RequestBody ChatRequest request) {
        if (request.question() == null || request.question().trim().isEmpty()) {
            return ResponseEntity.badRequest().body(new ChatResponse("Question must not be empty."));
        }

        String answer = supportService.handleCustomerQuery(request.question());
        return ResponseEntity.ok(new ChatResponse(answer));
    }
}

11. Prompt Engineering for Grounding & Eliminating Hallucinations

An AI customer support system cannot afford open-ended creativity. Hallucination suppression requires three defensive layers:

Prompt Construction Best Practices

• Provide an Explicit Out: Always supply the model with an approved fallback sentence when information is missing. Otherwise, it will attempt to extrapolate an answer.
• Delimit the Context: Surround raw context with distinctive tokens (such as ### CONTEXT ### or triple backticks) to defend against basic prompt injection attacks hidden within user input.
• Anti-Sycophancy Instructions: Direct the model to avoid validating erroneous premises made by the user (e.g., "Customer: 'Since your return window is 90 days, can I return this?'" $\rightarrow$ "Agent: 'Actually, our return window is 30 days...'").

12. Production Engineering Pitfalls, Edge Cases, and Mitigations

1. The "Lost in the Middle" Phenomenon

Research demonstrates that LLMs pay higher attention to information located at the extreme beginning and end of long contexts, often overlooking details placed in the middle.

• Mitigation: Keep your Top-K small (3 to 5 chunks). If more context is required, place the most relevant chunks at the very top or bottom of the injected context window.

2. Bad Chunk Boundaries

If a critical table or multi-step instruction is split across two chunks without overlap, half the context becomes orphaned.

• Mitigation: Employ semantic chunking or ensure a generous overlap (15%–20%). For tabular data, preserve JSON or Markdown table formats within a single chunk.

3. Outdated Embeddings

When a company policy changes, stale vectors left in the database can cause the system to return conflicting answers.

• Mitigation: Include metadata tags such as version, effective_date, and doc_id. Before writing new chunks, execute a purge transaction:

DELETE FROM vector_store WHERE metadata->>'doc_id' = 'refund_policy_v1';

4. Cold-Start and Embedding Latency

Generating embeddings over millions of tokens during service startup can block application availability.

• Mitigation: Decouple ingestion from the API server. Ingest documents via offline background workers (e.g., Spring Batch, Apache Kafka, or scheduled jobs) rather than executing @PostConstruct hooks in production.

13. Summary and Key Takeaways

• RAG solves enterprise LLM limitations: It eliminates knowledge cutoffs, incorporates private business documents, and prevents hallucinations without requiring costly model retraining.
• The "Open-Book" Paradigm: Foundational LLMs act as intelligent reasoning engines; RAG acts as the dynamic, real-time research assistant supplying verified facts.
• Fine-Tuning vs. RAG: Fine-tuning teaches style, formatting, and tone; RAG provides facts, real-time updates, and verifiable citations.
• Two decoupled pipelines: Ingestion operates asynchronously (Load $\rightarrow$ Chunk $\rightarrow$ Embed $\rightarrow$ Store), while Query runs in real time (Embed Query $\rightarrow$ Similarity Search $\rightarrow$ Augment Prompt $\rightarrow$ Generate Answer).
• Chunking requires overlap: Use a 10%–20% token overlap to ensure critical context is not severed at arbitrary boundary cutoffs.
• Mathematical foundation: Semantic similarity is calculated using Cosine Similarity ($\cos \theta = \frac{\vec{q} \cdot \vec{d}}{\|\vec{q}\|\|\vec{d}\|}$), allowing systems to connect queries with relevant documents even when vocabulary differs.
• Defensive prompt design: Always configure an explicit fallback phrase, set a low temperature ($\le 0.2$), and apply a similarity search threshold to eliminate hallucinations in production support systems.