Full text search using Elasticsearch for blazingly fast search

This video explains the challenges of implementing product search in rapidly growing datasets using traditional relational databases and illustrates how full-text search engines like Elastic Search revolutionized search capabilities using inverted indexing. It highlights key concepts such as search speed, relevance, typo tolerance, and demonstrates a practical benchmark comparing PostgreSQL and Elastic Search.

1. Challenges of Search in Relational Databases (Circa 2005)

Initially, implementing search with relational databases involved straightforward SQL queries using pattern matching (LIKE clause with %), which was efficient on small datasets (thousands of products). However, as datasets grew to millions of entries, this approach became problematic.

• Searching millions of rows using pattern matching degrades performance drastically (from milliseconds to seconds or even minutes).
• Pattern matching requires scanning every row and field character-by-character, causing slow response times.
• Traditional databases treat all matches equally—no concept of relevance or ranking. For example, a search for "laptop" could return unrelated items like laptop bags before actual laptops.
• Typo handling is non-existent: misspelled queries return fewer or irrelevant results.
• Slow latency frustrates users, impacts conversion rates, and is unacceptable for high-traffic commercial sites.

Key Insight: Relational databases operate like a librarian who finds books by scanning every book sequentially, which becomes inefficient at scale and lacks relevance sorting.

2. The Information Explosion and Search Requirements

By 2005, the internet and e-commerce sectors were growing exponentially: Google indexing billions of pages, Amazon cataloging millions of products, LinkedIn indexing millions of profiles.

• Customers expect search results in milliseconds.
• Typo tolerance, relevant ranking, and smart query matching became critical features.
• Companies needed faster, scalable, and more intelligent search mechanisms.

Conclusion: Traditional relational search methods were no longer sufficient; the problem demanded new fundamental approaches.

3. The Inverted Index: A Paradigm Shift

The inverted index is the key revolutionary idea that transformed text-based search.

• Concept: Instead of scanning every document/book in response to a query, preprocess all documents to build an index mapping each term to the documents where it appears.
• For example, the term "machine" would link to multiple documents, showing where it appears (e.g., pages or positions).
• This index allows direct retrieval of documents containing the query terms, drastically speeding up search.
• This inverts the traditional search problem (term→document instead of document→term), hence "inverted index."

How It Works (Example):

Advantages:

• Faster search: Only relevant documents are accessed.
• Relevance scoring: Search engines estimate the importance of results based on term frequency and document features.
• Supports typo tolerance, phrase matching, and field-based boosting.

4. Elastic Search and Apache Lucene

Elastic Search is a high-performance, distributed full-text search engine built on top of Apache Lucene, a mature open-source search library implementing inverted indexing.

• Elastic Search supports complex queries, relevance scoring, and typo tolerance.
• It uses the BM25 algorithm to rank search results based on:
  • Term frequency (how often a term appears in a document)
  • Document frequency (how common the term is across all documents)
  • Document length
  • Field boosting (importance based on where terms appear — title > description > content)

Field Boosting Explained:

Users can customize field boosting to tailor search relevance according to their product/domain needs.

5. Features Enabled by Full Text Search Engines

• Relevance-based ranking: Results are sorted by significance, not just exact matches.
• Typo tolerance: Misspelled queries can still yield correct results by approximate matching.
• Autocomplete/type-ahead: Instant suggestions while typing, enhanced by relevance and typo correction.
• Fast searching on large-scale datasets: Millisecond latency even with millions of documents.

These features improve user experience significantly compared to simple SQL pattern matching.

6. Comparison: PostgreSQL Full Text Search vs Elastic Search

A practical demonstration compares PostgreSQL’s traditional LIKE query to Elastic Search’s inverted index querying:

• PostgreSQL search again scanned the entire table using case-insensitive pattern matching.
• Elastic Search queried its prebuilt index with optimized relevance and returned results faster.
• Both returned the same number of results for fairness, but Elastic Search was consistently faster.

7. Architectural and Implementation Highlights

• Data population involved inserting 50,000 product reviews with sentiment (positive/negative) into both PostgreSQL and Elastic Search.
• PostgreSQL used a typical table with fields: ID, review (text), sentiment (keyword).
• Elastic Search used an index with mappings specifying:
  • review as type text (enabling full-text search)
  • sentiment as type keyword (for exact matches)
• Search APIs stream results asynchronously to frontend to minimize waiting time and compare latencies in real-time.
• Elastic Search queries are JSON-based (DSL), supporting detailed filtering, scoring, and boosting parameters.

8. Practical Recommendations for Backend Engineers

• Master relational databases first: Understanding indexing, queries, and optimization is essential as it underpins most backend tasks.
• Use Elastic Search or PostgreSQL’s built-in full-text search features for search-heavy applications:
  • Elastic Search is preferable for large-scale applications requiring advanced relevance, typo tolerance, and fast query response.
  • PostgreSQL full text search is suitable for moderate-scale applications or where additional infrastructure is undesirable.
• Elastic Search knowledge can often be leveraged via existing documentation and code snippets; deep theory understanding is optional unless building custom search engines.
• The ELK stack (Elastic Search, Logstash, Kibana) is a powerful ecosystem also used heavily for log management and analytics, besides search.

Summary Table: Traditional vs Full-Text Search Characteristics

Key Takeaway: The inverted index technology implemented by Elastic Search and Lucene revolutionized search by prioritizing speed, relevance, and robustness, enabling modern, scalable, typo-tolerant search experiences that traditional relational databases cannot efficiently provide at scale.