How it works
Define the search behavior first: which fields are searchable, whether all query words must match, and whether users need phrases, filters, or typo tolerance. Searching post bodies for “electric bike” is different from looking up an exact product ID or completing a prefix as someone types.
Analyze documents into searchable terms. An analyzer might lowercase text, split it into tokens, and normalize word forms; language-specific rules matter. Apply compatible analysis to queries so that indexed text and search terms can match.
Build an inverted index: each term points to a postings list of documents containing it. For example, “electric” might map to documents 12 and 30, while “bike” maps to 12 and 45. Intersecting those lists finds document 12 for an AND query; stored token positions support phrase matching.
Retrieve candidates and rank them. A scoring method such as BM25 uses term frequency, term rarity, and document length; field boosts can make title matches worth more than body matches. Combine text relevance with explicit filters such as location, category, or visibility, rather than treating every constraint as a ranking signal.
At larger scale, partition documents across search shards. A coordinator sends the query to relevant shards, each returns its best candidates, and the coordinator merges them using comparable scores and a stable tie-breaker. Replicas add read capacity and availability, while querying many shards increases coordination work and tail latency.
Keep the index synchronized with the source database through change data capture or a transactional outbox. Apply versioned updates and deletion events so retries or delayed messages cannot restore old content. Index refresh makes changes searchable; use a consistent search snapshot and a cursor containing the last result’s sort values when pagination must remain stable.
A worked example
Searching a large post collection
Suppose a service stores 120 million posts and returns 20 results per query. Partition its search index into 12 primary shards, with one replica each. For “electric bike,” the coordinator queries all 12 primary-shard groups, selecting one available copy per group. Each shard returns its top 100 candidates, so the coordinator merges at most 1,200 candidates initially. It checks current visibility before returning text and fetches more candidates if filtering leaves fewer than 20 results. Database changes reach the index through CDC, with a target visibility lag below five seconds. Per-post versions reject older updates, while versioned deletion markers prevent delayed events from resurrecting deleted posts. Subsequent pages use a search snapshot and score-plus-ID cursor.
Trade-offs to say out loud
- Fast retrieval costs storage and write work
- Postings, token positions, stored fields, and replicas consume space, while indexing and segment merges compete for CPU and disk bandwidth.
- Freshness competes with indexing efficiency
- Frequent refreshes reduce search visibility lag but increase overhead, so define a freshness target instead of promising immediate visibility.
- More shards increase query coordination
- Shards distribute storage and indexing work, but broad searches must contact more workers and can become limited by the slowest response.
- Ranking adds relevance and complexity
- Text scoring, recency, and business signals need evaluation, and shard-local term statistics can make relevance scores less consistent.
- Stale indexes cannot enforce current permissions alone
- Sensitive results require current authorization checks before exposure, including snippets and any counts that could reveal private content.
- Analysis changes can require rebuilding the index
- Changing tokenization or indexed fields often requires a backfill into a new index, with extra capacity and a controlled cutover.
When to reach for it
- Users need keyword search across titles, descriptions, posts, or documents, with relevance ordering rather than exact-key lookup.
- Requirements include phrase matching, language-aware tokenization, field weighting, or combinations of text queries and structured filters.
- Scanning text with substring queries cannot meet the expected dataset size and query latency.
- Search must scale independently from transactional writes, and the product can tolerate a defined indexing delay.
How to bring it up in the interview
- Start with query semantics and scale: searchable fields, languages, filters, corpus size, query volume, and acceptable freshness. These determine whether database-native full-text search is sufficient or a separate search cluster is justified.
- Explain both paths: database changes flow into a derived index, while queries retrieve, filter, rank, and paginate candidates. Interviewers often probe what happens when indexing falls behind or events arrive out of order.
- Defend the partitioning choice and query fan-out. Expect questions about shard count, routing by tenant or region, score comparability, slow shards, and whether partial results are acceptable.
- Separate relevance from correctness. A slightly stale ranking may be acceptable, but deleted private content must not leak; discuss authorization, deletion handling, and refilling pages after candidates are rejected.
- Name operational targets: search latency, indexing lag, ingestion backlog, and result quality. Explain how you would rebuild an index without losing concurrent updates or interrupting queries.
Common mistakes
- Saying “use a search engine” without explaining the inverted index, ranking, update pipeline, or shard-level query execution.
- Assuming full-text search is the same as substring matching, prefix autocomplete, or semantic vector search.
- Dual-writing independently to the database and search index without a reliable recovery path for partial failures.
- Treating stale indexed visibility fields as sufficient authorization, or checking permissions only after snippets have already been returned.
- Using deep offset pagination on changing results without discussing its cost, duplicate or missing results, and snapshot-based cursors.
Where it comes up
Interview questions where full-text search is part of a strong answer, and why.
- Design Facebook post searchDistributed keyword retrieval, relevance ranking, fresh indexing, and permission-correct results are the core of the design.
- Design a restaurant review app like YelpRanked restaurant discovery combines text retrieval with geographic filters and search updates driven by changing restaurant and review data.
- Design Facebook MarketplaceListing discovery combines searchable titles and descriptions with location filters, stable pagination, and asynchronously updated listing state.
Questions people ask
What is the difference between full-text search and a database index?
Full-text search commonly uses an inverted index that maps terms to matching documents and supports relevance ranking. A typical B-tree index orders field values for exact matches and range scans. Databases can also provide native full-text indexes, so full-text search does not automatically require a separate service.
How do you design a scalable search system?
Separate durable source data from the derived search index, and use a reliable change pipeline to keep them synchronized. Partition the index, execute queries across relevant shards, and merge ranked candidates. Define explicit policies for freshness, authorization, pagination, and shard failures.
What is the difference between full-text search and vector search?
Full-text search matches analyzed words and commonly ranks results using lexical relevance signals such as BM25. Vector search retrieves items whose embeddings are close, which can find related meanings even without shared words. Hybrid search combines the two, but adds retrieval, ranking, and evaluation complexity.