Design a learning-to-rank system for search: system design interview question

How the interviewer opens it

“Design a service that orders retrieved search documents by relevance. Include how the service trains its model and learns from user feedback. The service must return results within a strict time limit.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

Relevance goal
Scale and candidates
Read and write mix
Latency
Feedback data
Features and document updates
Experiments and sessions
Availability and consistency
Event durability
System boundary

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

Core requirements

Order retrieved documents within the latency target.
Collect result impressions, clicks, and dwell reports.
Train a relevance model from feedback and human judgments.
Compare model versions and release safe updates.

Usually out of scope

Retrieve candidate documents and build the search index.
Check document access and safety.
Personalize results from user history.
Select advertisements or design the search interface.

What interviewers ask about

Time to return results: Each request has 200 candidates, but feature reads and score computation must share a 50 ms budget at peak traffic.
What clicks mean: Position and the old ranker change which documents receive clicks. A missing click does not prove that a document lacks relevance.
Data from collection to use: Features change after each query. Retries and delayed dwell reports must update one click without changing its identity or creating false evidence.
Evidence for a new result order: A model can gain clicks without better relevance. Aggregate metrics can also hide losses for rare queries and new documents.

Common mistakes

Treats every unclicked document as irrelevant.
Uses position as a relevance feature without a model of click bias.
Claims propensity weights recover evidence for documents with no chance of display.
Makes one remote feature request per document.
Counts dwell reports as extra clicks or discards valid dwell updates as duplicates.
Selects a model from raw click rate without latency or relevance checks.

The bar by level

Senior (L5/E5): Designs historical feature joins, a bounded score path, and daily model updates. Explains correction assumptions and missing support. Defines stable session experiments, offline evaluation, and rollback. Handles delayed reports without extra clicks or lost dwell updates.
Staff (L6+/E6+): States which relevance claims the available evidence can support. Allocates limited test traffic across risk and information needs. Connects model cost, tail latency, feedback bias, and release policy. Defines experiment units, uncertainty estimates, and dataset revision rules. Defines responses to query shifts and failures without unnecessary components.

Sources · 1 reference

These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.

SysRound · 1 record

No company association
Design a learning-to-rank system for search
Reported date not specified
Source snapshot: Nov 12, 2026
View source

How the interviewer opens it

“Design a service that orders retrieved search documents by relevance. Include how the service trains its model and learns from user feedback. The service must return results within a strict time limit.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

Scope of a strong answer

Core requirements

Order retrieved documents within the latency target.
Collect result impressions, clicks, and dwell reports.
Train a relevance model from feedback and human judgments.
Compare model versions and release safe updates.

Usually out of scope

Retrieve candidate documents and build the search index.
Check document access and safety.
Personalize results from user history.
Select advertisements or design the search interface.

What interviewers ask about

Time to return results

Each request has 200 candidates, but feature reads and score computation must share a 50 ms budget at peak traffic.

What clicks mean

Position and the old ranker change which documents receive clicks. A missing click does not prove that a document lacks relevance.

Data from collection to use

Features change after each query. Retries and delayed dwell reports must update one click without changing its identity or creating false evidence.

Evidence for a new result order

A model can gain clicks without better relevance. Aggregate metrics can also hide losses for rare queries and new documents.

Common mistakes

Treats every unclicked document as irrelevant.

Uses position as a relevance feature without a model of click bias.

Claims propensity weights recover evidence for documents with no chance of display.

Makes one remote feature request per document.

Counts dwell reports as extra clicks or discards valid dwell updates as duplicates.

Selects a model from raw click rate without latency or relevance checks.

The bar by level

Senior (L5/E5)

Designs historical feature joins, a bounded score path, and daily model updates. Explains correction assumptions and missing support. Defines stable session experiments, offline evaluation, and rollback. Handles delayed reports without extra clicks or lost dwell updates.

Staff (L6+/E6+)

States which relevance claims the available evidence can support. Allocates limited test traffic across risk and information needs. Connects model cost, tail latency, feedback bias, and release policy. Defines experiment units, uncertainty estimates, and dataset revision rules. Defines responses to query shifts and failures without unnecessary components.