Multi-Tenant RAG with Qdrant: Metadata Filtering & HNSW Vector Indexing

This chapter explores enterprise-grade Retrieval-Augmented Generation (RAG) using Qdrant vector database. It details the Hierarchical Navigable Small World (HNSW) algorithm for scalable vector search, demonstrates Metadata Payload Filtering for multi-tenant isolation, and outlines a practical YouTube Revision Assistant application.

1. Practical Application: The YouTube DSA Revision Assistant

When students study large video series (such as a 100-video Data Structures and Algorithms course), a common problem arises: The Needle in a Haystack Search. A student remembers learning a specific concept (e.g., "Sliding Window Pattern" or a specific "Binary Search" edge case), but cannot identify which video or exact timestamp contains that explanation.

Manually opening 6–10 videos and scrubbing through them frame-by-frame wastes significant time and effort.

Architecture Overview

1. Extraction: Convert YouTube video audio into text transcripts using automated transcription tools.
2. Metadata Enrichment: Attach metadata payloads to each transcript chunk, including:
   • video_id: Unique YouTube video ID.
   • video_url: Full link to the video.
   • timestamp: Exact point in the video (e.g., 30m 07s).
   • topic_category: Relevant DSA topic tag (e.g., binary_search).
3. Storage: Vectorize text chunks and store them in a Qdrant collection alongside their metadata payloads.
4. Retrieval: When a user queries "Where was the sliding window optimization discussed?", Qdrant performs similarity search and returns:
   • The exact text context.
   • The specific video link with an embedded time marker (https://youtu.be/...&t=1807s).

2. The Scalability Wall of Flat Vector Search

In basic vector search implementations, a query vector is compared against every single vector stored in the database using Cosine Similarity.

For a small database, this brute-force approach is feasible:

Where $N$ is the number of vectors and $d$ is the vector dimension size (e.g., 384 dimensions).

Why Naive Linear Search Fails in Production

• Small Collections ($N \le 1,000$): Search completes in milliseconds ($O(N)$ brute-force linear scan).
• Enterprise Collections ($N = 100,000,000$): Performing $100$ million high-dimensional dot products per query degrades performance. A single query can take 20 minutes or longer, making real-time applications unusable.

To scale vector retrieval to millions of data points without linear slowdowns, specialized graph-based indexing structures like HNSW are required.

3. Core Mental Model: Hierarchical Navigable Small World (HNSW)

HNSW (Hierarchical Navigable Small World) is an approximate nearest neighbor (ANN) graph algorithm that enables sub-linear ($O(\log N)$) search speeds across vast vector collections.

Real-Life Analogy: Navigating from Bangalore to Delhi

Imagine sitting in a small room in BTM Layout, Bangalore, with the goal of reaching Kalyan Jewellers in Connaught Place, New Delhi.

The Linear Search Approach (Brute-Force)

You walk out of your house into the nearest street in BTM Layout. You walk into every single shop, office, and room asking: "Are you Kalyan Jewellers in Connaught Place, Delhi?" They say no. You check the next street, then the next neighborhood, moving systematically across Bangalore, then state by state, city by city, until you eventually find the target shop in Delhi.

• Result: You will eventually find it, but it requires checking millions of irrelevant shops along the way.

The HNSW Hierarchical Approach (Multi-Layer Skip Traversal)

Instead of searching locally at the lowest level, you navigate by jumping across abstraction layers:

How HNSW Applies to Vector Spaces

1. Top Layers (Highways): Sparse graph networks with long-distance links. The algorithm quickly traverses vast regions of vector space to reach the proximity of the target region.
2. Bottom Layers (Local Streets): Dense graph networks with short-distance links. Once the search enters the target sub-region, it transitions down layers to perform fine-grained comparisons among local vector neighbors.

Note: HNSW does not calculate cosine similarity against every vector in the database. It navigates vector connections hierarchically, skipping irrelevant clusters and isolating the nearest neighbors in $O(\log N)$ operations.

4. Naive Vector Search vs. HNSW Traversal

5. Enterprise Metadata Payloads & Multi-Tenancy

In production environments, vector databases rarely store isolated text chunks. Vector data is typically paired with structured JSON metadata payloads.

Raw Data vs. Structured JSON Payloads

{
  "id": "point_001",
  "vector": [0.023, -0.114, 0.891, "... total 384 floats ..."],
  "payload": {
    "text": "Employees are eligible for 24 days of paid vacation per year.",
    "category": "vacation",
    "department": "HR",
    "company": "AcmeCorp",
    "is_active": true
  }
}

The Waste of Unfiltered Search

If a user asks: "How many vacation days do I get?", running an unfiltered vector search compares the query against policy documents across all categories (salary, work_life_balance, reimbursement, deprecated_policies).

Executing HNSW on a dataset filtered to 10 relevant vectors is significantly faster than executing HNSW across 10,000,000 unfiltered vectors.

6. Qdrant Payload Indexing (create_payload_index)

By default, Qdrant does not build lookup indexes for every field inside a payload. To filter queries efficiently by metadata fields (such as category or is_active), an explicit Payload Index must be created prior to executing filtered searches.

When you instruct Qdrant to create a payload index on a field (e.g., category), Qdrant pre-computes an inverted mapping between values and point IDs:

Python API Usage

from qdrant_client import QdrantClient
from qdrant_client.models import PayloadSchemaType

client = QdrantClient(url="http://localhost:6333")

# Pre-index the 'category' field in the collection
client.create_payload_index(
    collection_name="enterprise_knowledge",
    field_name="category",
    field_schema=PayloadSchemaType.KEYWORD,
)

7. Qdrant Filtering Mechanics (Must, Must Not, Should)

Qdrant provides logical boolean operators to construct precise search filters.

8. Full Implementation: Production-Grade Qdrant RAG Pipeline

Below is a complete script demonstrating JSON document loading, payload index creation, upserting structured points, executing filtered search, and querying an LLM.

import os
import json
from qdrant_client import QdrantClient
from qdrant_client.models import (
    VectorParams,
    Distance,
    PointStruct,
    Filter,
    FieldCondition,
    MatchValue,
    PayloadSchemaType
)
from fastembed import TextEmbedding
from groq import Groq

# 1. Initialize Clients
qdrant = QdrantClient(url="http://localhost:6333")
encoder = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")
groq_client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

COLLECTION_NAME = "corporate_policies"

# 2. Reset / Re-create Collection
if qdrant.collection_exists(COLLECTION_NAME):
    qdrant.delete_collection(COLLECTION_NAME)

qdrant.create_collection(
    collection_name=COLLECTION_NAME,
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)

# 3. Create Payload Index for Metadata Filtering
qdrant.create_payload_index(
    collection_name=COLLECTION_NAME,
    field_name="category",
    field_schema=PayloadSchemaType.KEYWORD,
)

# 4. Load JSON Data with Metadata
raw_json_data = [
    {
        "id": 1,
        "text": "Full-time employees receive 24 days of paid vacation per year.",
        "category": "leave",
        "is_active": True
    },
    {
        "id": 2,
        "text": "Standard working hours are 8 hours per day, Monday through Friday.",
        "category": "workplace",
        "is_active": True
    },
    {
        "id": 3,
        "text": "Travel expense claims must be submitted within 30 days.",
        "category": "reimbursement",
        "is_active": True
    },
    {
        "id": 4,
        "text": "Legacy Policy: Employees receive 12 days of paid leave per year.",
        "category": "leave",
        "is_active": False
    }
]

# 5. Extract Text, Generate Embeddings & Upsert Points
texts = [doc["text"] for doc in raw_json_data]
embeddings = list(encoder.embed(texts))

points = []
for idx, doc in enumerate(raw_json_data):
    points.append(
        PointStruct(
            id=doc["id"],
            vector=embeddings[idx].tolist(),
            payload=doc  # Complete payload storing text, category, and is_active flag
        )
    )

qdrant.upsert(collection_name=COLLECTION_NAME, points=points)
print(f"Successfully upserted {len(points)} points into '{COLLECTION_NAME}'.")

# 6. Execute Filtered Vector Search
query_text = "How many leave days am I granted annually?"
query_vector = list(encoder.embed([query_text]))[0].tolist()

# Define strict payload filter: category == 'leave' AND is_active == True
search_filter = Filter(
    must=[
        FieldCondition(
            key="category",
            match=MatchValue(value="leave")
        ),
        FieldCondition(
            key="is_active",
            match=MatchValue(value=True)
        )
    ]
)

search_results = qdrant.search(
    collection_name=COLLECTION_NAME,
    query_vector=query_vector,
    query_filter=search_filter,
    limit=2
)

# 7. Extract Context for RAG
retrieved_contexts = [hit.payload["text"] for hit in search_results]
context_str = "\n".join(retrieved_contexts)

# 8. Query LLM Generation Step
system_prompt = "You are an HR AI Assistant. Answer questions strictly using the provided context."
user_prompt = f"Context:\n{context_str}\n\nQuestion: {query_text}"

response = groq_client.chat.completions.create(
    model="llama-3.3-70b-versatile",  # Updated supported model tag
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ],
    temperature=0.0
)

print("\n--- LLM Final Response ---")
print(response.choices[0].message.content)

9. Production Pitfalls & Debugging Mindset

Missing Payload Indexes

• Symptom: Unindexed filtered vector queries work on small dev datasets, but suffer from high latencies or timeouts in production.
• Root Cause: Without calling create_payload_index, Qdrant evaluates payload metadata dynamically during query execution instead of using optimized inverted maps.
• Fix: Pre-index all payload fields used in must, should, or must_not clauses.

Model Name Deprecations

• Symptom: Endpoint calls fail with Model Deprecated or 404 Not Found errors (e.g., legacy llama-3-70b-versatile deprecation notices).
• Fix: Monitor LLM provider updates (such as Groq release notes) and update model configuration strings to active model endpoints (e.g., llama-3.3-70b-versatile or openai/gpt-os-120b).

Over-Filtering vs. Under-Filtering

• Under-filtering: Searching across unrelated categories wastes compute and can introduce irrelevant context into RAG prompts.
• Over-filtering: Applying overly restrictive must conditions can return zero matches, starving the LLM of necessary context. Use should clauses when fallback behavior is preferred.

10. Summary and Key Takeaways

• Linear Search Bottleneck: Brute-force $O(N)$ vector comparisons work well for small datasets, but degrade at scale ($100\text{M+}$ points).
• HNSW Acceleration: The Hierarchical Navigable Small World algorithm reduces vector search complexity to sub-linear time ($O(\log N)$) using multi-layer graph structures.
• Hierarchical Navigation: HNSW navigates through sparse top layers (long-distance hops) down to dense lower layers (local comparisons) to isolate relevant vector regions quickly.
• Metadata Payloads: Storing structured JSON payloads alongside vectors enables multi-tenant filtering, scoping searches to specific categories, companies, or user IDs.
• Explicit Payload Indexing: Qdrant requires explicit payload indexing via create_payload_index() to execute high-performance filtered searches.
• Filter Logical Operators: Qdrant supports structured filtering using must (AND), must_not (NOT), and should (OR) operators.
• Targeted RAG Context: Filtering vectors prior to top-$K$ extraction ensures the LLM prompt receives relevant, domain-specific context.