Machine learning system design interview

Clarify what the system predicts, who uses the prediction, and what action follows. Ranking a feed, estimating arrival time, and blocking a suspicious transaction have different latency requirements, error costs, and feedback loops.

Ask whether the interviewer wants an end-to-end ML system or a deeper discussion of modeling. Establish the simplest useful baseline, sketch the full architecture, then spend the remaining time on the hardest requirements rather than listing model families.

Choose a product objective before selecting a training objective. For a feed, clicks may be a useful signal but may not represent satisfaction. For an ETA service, prediction error matters, but so does whether estimates consistently understate arrival times.

Explain how you will judge the model offline, validate its impact online, and keep the serving system healthy. Improving one category does not guarantee improvement in the others.

Describe the events that create training examples. A recommendation system needs impression records showing what was displayed, not just clicks. Log the request identifier, candidate or item identifier, position, timestamp, and relevant model and experiment versions so outcomes can be attributed to the correct decision.

Define when a label becomes mature. A purchase may happen well after an impression, and a fraud label may arrive after an investigation. Missing feedback is not automatically a negative label. Training on immature outcomes can systematically mislabel examples.

A concrete data path is application events to a durable log, then validated raw storage, followed by batch or stream transformations and versioned training datasets. Define deduplication keys, schema checks, retention, access control, and how privacy deletions reach derived data.

Separate slow-changing features, such as account attributes or item embeddings, from fast-changing features, such as recent clicks or current traffic. Batch jobs can produce historical aggregates, while stream processors can maintain features that need low freshness lag.

Use point-in-time joins for training: each example must use only information available at its prediction time. Joining an old impression to today's user profile or a later outcome aggregate leaks future information and makes offline results misleading.

A feature store can provide historical retrieval and low-latency online lookups, but it is not mandatory. Shared transformation definitions, versioned schemas, freshness metadata, and explicit missing-value behavior matter more than adding a particular component.

Start with a baseline appropriate to the task: popularity ranking, a simple regression, rules, or a small classifier. Add complexity only when the expected quality gain justifies extra training cost, serving cost, and operational risk.

Use a workflow that creates a dataset snapshot, validates it, trains the model, evaluates it, and registers an immutable artifact with its feature schema and configuration. Checkpoint expensive jobs and make retries safe so failed orchestration does not publish an incomplete model.

Choose a split that matches deployment. A chronological holdout is often useful when predicting future behavior; grouping by user or entity may also be necessary to avoid inappropriate overlap. Evaluate important slices, cold-start cases, and recent data rather than reporting only one aggregate score.

For a recommendation example, draw: request service → candidate retrieval → feature lookup → ranking → eligibility checks → response. Candidate retrieval might combine precomputed lists, recent content, and vector search. Rank only a bounded candidate set rather than running an expensive model over the entire catalog.

For ETA prediction or fraud scoring, the path may instead be request validation → feature retrieval → model inference → decision policy. Keep model output separate from business rules, especially when the action affects money, safety, or access.

Give each stage a latency budget. Explain whether the model runs inside the application process or behind a shared inference service, and what happens when a dependency exceeds its deadline. A simpler model, cached result, or nonpersonalized ranking may be acceptable for discovery; a high-risk decision may need to fail closed or enter review.

Estimate peak prediction requests, candidates scored per request, feature lookups, model memory, and event volume. Request rate alone can hide the dominant cost: scoring hundreds of candidates per request is a different workload from producing one scalar prediction.

Use explicit assumptions instead of invented benchmarks. For example, at a hypothetical 2,000 requests per second and 200 ranked candidates per request, the system evaluates 400,000 candidate scores per second before accounting for retries or auxiliary models. Whether that fits on a machine depends on model complexity, batching, hardware, and latency targets.

Offline evaluation is a gate, not proof of product improvement. Shadow traffic can check prediction behavior and resource use without changing user responses. A canary limits initial exposure; a randomized A/B test estimates product impact when assignment and measurement are sound.

Version the model together with its input contract and required preprocessing. Rollback must restore a compatible serving configuration, not just replace a model file. Keep a known-good version available and define who or what triggers rollback.

Monitor service health separately from model health. Infrastructure alerts can detect timeouts immediately, while quality degradation may remain invisible until delayed labels arrive. Feature drift is a reason to investigate, not proof that accuracy has fallen.

Interviewers often test whether the architecture still works when the data pipeline lags, a feature service fails, a new model is incompatible, or traffic changes abruptly. Name the degraded behavior and recovery path for each critical dependency.

Also address feedback loops. A ranking policy influences which items collect labels, and repeatedly promoting popular items can reduce exploration. Explain how controlled exploration, exposure logging, and slice evaluation help, while recognizing that production experiments need safety limits.

To practice, take one prompt and walk through requirements, labels, features, training, serving, and monitoring. Then change one constraint, such as tighter latency, delayed labels, or no user history, and explain which parts of the design change.