Ep 14 | Building an End-to-End Semantic RAG System

Retrieval-Augmented Generation (RAG) combines custom enterprise knowledge bases with the generative capabilities of Large Language Models (LLMs). While traditional Keyword-based RAG fails when users ask questions using synonyms or alternative phrasing, Semantic RAG leverages dense vector embeddings and cosine similarity to retrieve relevant information based on meaning rather than exact string matches. This module walks through building a complete, production-ready Semantic RAG pipeline from scratch and analyzes the scaling limits that necessitate dedicated Vector Databases.

1. The Limitations of Naive RAG vs. The Semantic Solution

In basic keyword-based RAG systems, retrieval relies on exact string or substring matching. If a user's question contains words that do not appear verbatim in the document corpus, the retrieval step fails entirely.

Semantic RAG solves this failure mode by mapping both the document corpus and the user query into a shared continuous vector space using embedding models. Even if terms differ, semantically similar concepts map to vectors that sit close to one another in higher-dimensional space.

2. Environment Setup & Dependency Management

To construct the full semantic RAG pipeline, set up an isolated Python virtual environment and install the required dependencies:

• groq: Provides high-speed inference for language models.
• sentence-transformers: Generates dense numerical embeddings locally.
• numpy: Performs vector math (dot products and vector norms).
• python-dotenv: Manages environment variables and API keys securely.

Environment Initialization Commands

# Create and activate a clean virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install core dependencies
pip install groq python-dotenv sentence-transformers numpy

3. System Architecture & Initializing Models

The system architecture consists of two main models:

1. Embedding Model: SentenceTransformer('all-MiniLM-L6-v2') converts raw text into 384-dimensional dense floating-point arrays.
2. LLM Generator Model: Groq(llama-3.3-70b-versatile) takes the retrieved text context alongside the prompt to formulate a natural language answer.

Python Initializer Module

import os
import sys
import numpy as np
from dotenv import load_dotenv
from sentence_transformers import SentenceTransformer
from groq import Groq

# Load environment configuration (.env containing GROQ_API_KEY)
load_dotenv()

# Initialize Embedding Model (Transforms text to 384-dim vectors)
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# Initialize Groq LLM Client
groq_client = Groq(api_key=os.getenv("GROQ_API_KEY"))
GROQ_MODEL_NAME = "llama-3.3-70b-versatile"

4. Building the Knowledge Base & Generating Document Embeddings

A knowledge base contains domain-specific knowledge (e.g., enterprise HR policies). During startup, every document (or sentence chunk) in the knowledge base is encoded into a dense vector array.

# Knowledge Base Corpus
documents = [
    "Employees receive 24 days of paid leave per year.",
    "Remote work is permitted up to 2 days per week with manager approval.",
    "Health insurance covers medical, dental, and vision care for full-time employees.",
    "The annual performance review occurs every year in December.",
    "Employees receive a internet allowance of $3000 per month for WFH setups.",
    "Working hours are flexible between 8 AM and 6 PM."
]

# Vectorize the complete corpus into a single 2D NumPy array
document_embeddings = embedding_model.encode(documents)

5. Implementing Cosine Similarity Math

Cosine similarity measures the cosine of the angle between two multi-dimensional vectors in vector space. It evaluates how aligned two texts are in terms of context and intent, yielding a score between -1.0 (opposite) and 1.0 (identical direction).

def calculate_cosine_similarity(vec_a: np.ndarray, vec_b: np.ndarray) -> float:
    """
    Computes the cosine similarity score between two 1D NumPy arrays.
    """
    dot_product = np.dot(vec_a, vec_b)
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    
    if norm_a == 0 or norm_b == 0:
        return 0.0
        
    return float(dot_product / (norm_a * norm_b))

6. Constructing the Semantic Retriever Module

The retriever function takes the vector representation of a query, compares it against every stored document vector using calculate_cosine_similarity, sorts the similarity scores in descending order, and returns the highest-scoring text block.

def retrieve_top_context(query_vector: np.ndarray, doc_embeddings: np.ndarray, docs: list):
    """
    Scans document vector embeddings and returns the highest scoring context item.
    """
    scores = []
    
    for idx, doc_vec in enumerate(doc_embeddings):
        score = calculate_cosine_similarity(query_vector, doc_vec)
        # Store metadata tuple: (Score, Document Text, Index)
        scores.append((score, docs[idx], idx))
    
    # Sort descending based on similarity score
    scores.sort(key=lambda x: x[0], reverse=True)
    
    # Return the highest matching entry: (top_score, top_doc_text)
    top_score, top_text, _ = scores[0]
    return top_score, top_text

7. Context Injection & Augmented LLM Generation

Once the semantic retriever extracts the top matching text chunk, this string is injected directly into the LLM system prompt as ground-truth context. The LLM is instructed to strictly restrict its response generation to this provided context.

def generate_rag_response(user_query: str, retrieved_context: str) -> str:
    """
    Augments the user query with semantic context and generates an LLM response.
    """
    system_prompt = (
        "You are an enterprise AI assistant. Answer the user's question using ONLY "
        "the provided context block. If the context does not contain the answer, "
        "state that the information is unavailable.\n\n"
        f"CONTEXT:\n{retrieved_context}"
    )
    
    response = groq_client.chat.completions.create(
        model=GROQ_MODEL_NAME,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_query}
        ],
        temperature=0.2 # Low temperature for accurate context adherence
    )
    
    return response.choices[0].message.content

8. Complete End-to-End Flow Implementation

Below is the complete, self-contained Python pipeline demonstrating semantic query vectorization, score matching, context extraction, and LLM answer generation:

# full_rag.py

import os
import sys
import numpy as np
from dotenv import load_dotenv
from sentence_transformers import SentenceTransformer
from groq import Groq

# 1. Environment & Model Initialization
load_dotenv()
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
groq_client = Groq(api_key=os.getenv("GROQ_API_KEY"))

# 2. Knowledge Base Corpus
documents = [
    "Employees receive 24 days of paid leave per year.",
    "Remote work is permitted up to 2 days per week with manager approval.",
    "Health insurance covers medical, dental, and vision care for full-time employees.",
    "The annual performance review occurs every year in December.",
    "Employees receive a internet allowance of $3000 per month for WFH setups.",
    "Working hours are flexible between 8 AM and 6 PM."
]

# 3. Batch Vector Encoding
doc_embeddings = embedding_model.encode(documents)

# 4. Mathematical Cosine Similarity Function
def cosine_similarity(vec1, vec2):
    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))

# 5. Semantic Vector Retrieval
def retrieve(query_embedding):
    scores = []
    for idx, doc_vec in enumerate(doc_embeddings):
        score = cosine_similarity(query_embedding, doc_vec)
        scores.append((score, documents[idx]))
    
    # Sort descending by score
    scores.sort(key=lambda x: x[0], reverse=True)
    return scores[0]  # Returns (top_score, top_document_text)

# 6. LLM Inference Execution
def ask_llm(query: str, context: str):
    system_prompt = (
        f"Answer the question using only the context provided below.\n\n"
        f"Context: {context}"
    )
    completion = groq_client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": query}
        ]
    )
    return completion.choices[0].message.content

# 7. Pipeline Execution Walkthrough
if __name__ == "__main__":
    # Test Query containing non-matching vocabulary ("vacation" vs "paid leave")
    user_query = "How much vacation do I get?"
    
    # Step A: Vectorize User Query
    query_vector = embedding_model.encode(user_query)
    
    # Step B: Perform Semantic Retrieval
    match_score, top_context = retrieve(query_vector)
    
    print(f"User Query: '{user_query}'")
    print(f"Top Semantic Match Score: {match_score:.4f}")
    print(f"Retrieved Context: '{top_context}'\n")
    
    # Step C: Generate Final Answer
    final_answer = ask_llm(user_query, top_context)
    print("LLM Response:")
    print(final_answer)

Output Trace

User Query: 'How much vacation do I get?'
Top Semantic Match Score: 0.4321
Retrieved Context: 'Employees receive 24 days of paid leave per year.'

LLM Response:
You receive 24 days of paid leave per year.

9. Production Realities: Memory Overhead & $O(N)$ Compute Bottlenecks

While storing vector arrays directly in system RAM using NumPy works for small datasets, it encounters critical architectural bottlenecks at enterprise scale.

1. In-Memory RAM Scaling Analysis

Using Python's sys.getsizeof(), we can measure the memory footprint of storing embedding arrays in RAM:

import sys
memory_bytes = sys.getsizeof(doc_embeddings)
print(f"Memory size for 6 vectors: {memory_bytes} bytes") 
# Output: ~7808 bytes (~7.8 KB)

Scaling this mathematical footprint demonstrates how quick in-memory arrays become unusable:

2. Linear Compute Time Bottleneck ($O(N)$)

In our custom retrieve() function, calculating cosine similarity requires scanning every single vector sequentially:

• For 6 documents: 6 vector math calculations ($< 1 \text{ ms}$).
• For 6,000,000 documents: 6 million continuous vector dot-product computations per incoming query.

Executing a linear scan across millions of floating-point arrays introduces query latencies ranging from several seconds to minutes, rendering real-time applications impractical.

10. Why Dedicated Vector Databases Are Mandatory

To scale Semantic RAG to enterprise datasets containing millions of chunks, native RAM arrays must be replaced with a Vector Database (e.g., Qdrant, Pinecone, Milvus, Weaviate).

Core Problems Vector Databases Solve

1. Sub-linear Search ($O(\log N)$): Instead of scanning every vector sequentially, Vector DBs construct graph structures (like HNSW - Hierarchical Navigable Small World) to perform Approximate Nearest Neighbor (ANN) search within milliseconds.
2. Persistent On-Disk Storage: Embeddings are stored efficiently on disk or high-speed NVMe drives with vector caching, eliminating the need to fit the entire dataset into volatile system RAM.
3. Dynamic Filtering: Allows combining unstructured vector search with structured SQL-like metadata filtering (e.g., department == 'HR' AND similarity > 0.70).
4. Similarity Threshold Guardrails: Real-world RAG implementations enforce minimum cosine similarity score cutoffs (e.g., score >= 0.35). Queries falling below the threshold return a fallback response, preventing context pollution and LLM hallucinations.

Summary and Key Takeaways

• Keyword Matching Limits: Naive keyword search fails when user queries use synonyms or phrasing different from the source text (e.g., matching "vacation" against "paid leave").
• Semantic Embeddings: Models like all-MiniLM-L6-v2 convert arbitrary text lengths into fixed-size dense numerical vectors, capturing core semantic meaning.
• Cosine Similarity: Mathematical measurement calculating vector alignment, providing a metric for how well document contexts match user intent.
• Context-Injected Generation: Semantic RAG passes high-scoring context directly into the system prompt of models like Groq's Llama-3.3-70B, restricting generation to verified facts.
• In-Memory Storage Bottlenecks: Storing vector arrays in raw NumPy memory scales linearly ($O(N)$) in both RAM consumption and compute time, leading to memory failure at scale.
• Vector DB Necessity: Production RAG systems require specialized databases like Qdrant to enable disk persistence, metadata filtering, and fast $O(\log N)$ approximate nearest neighbor retrieval.