# Design a learning-to-rank system for search: system design interview question

> Search relevance under a strict latency limit, with click feedback that reflects both document relevance and the previous result order.

Source: https://sysround.com/questions/design-learning-to-rank-system-for-search
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a service that orders retrieved search documents by relevance. Include how the service trains its model and learns from user feedback. The service must return results within a strict time limit."

## What to clarify first

- Relevance goal
- Scale and candidates
- Read and write mix
- Latency
- Feedback data
- Features and document updates
- Experiments and sessions
- Availability and consistency
- Event durability
- System boundary

## Scope of a strong answer

Core requirements:

- Order retrieved documents within the latency target.
- Collect result impressions, clicks, and dwell reports.
- Train a relevance model from feedback and human judgments.
- Compare model versions and release safe updates.

Usually out of scope:

- Retrieve candidate documents and build the search index.
- Check document access and safety.
- Personalize results from user history.
- Select advertisements or design the search interface.

## What interviewers probe

### Time to return results

Each request has 200 candidates, but feature reads and score computation must share a 50 ms budget at peak traffic.

### What clicks mean

Position and the old ranker change which documents receive clicks. A missing click does not prove that a document lacks relevance.

### Data from collection to use

Features change after each query. Retries and delayed dwell reports must update one click without changing its identity or creating false evidence.

### Evidence for a new result order

A model can gain clicks without better relevance. Aggregate metrics can also hide losses for rare queries and new documents.

## Common mistakes

- Treats every unclicked document as irrelevant.
- Uses position as a relevance feature without a model of click bias.
- Claims propensity weights recover evidence for documents with no chance of display.
- Makes one remote feature request per document.
- Counts dwell reports as extra clicks or discards valid dwell updates as duplicates.
- Selects a model from raw click rate without latency or relevance checks.

## The bar by level

- **Senior:** Designs historical feature joins, a bounded score path, and daily model updates. Explains correction assumptions and missing support. Defines stable session experiments, offline evaluation, and rollback. Handles delayed reports without extra clicks or lost dwell updates.
- **Staff+:** States which relevance claims the available evidence can support. Allocates limited test traffic across risk and information needs. Connects model cost, tail latency, feedback bias, and release policy. Defines experiment units, uncertainty estimates, and dataset revision rules. Defines responses to query shifts and failures without unnecessary components.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=search-result-ranking


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.