# Design a continuous learning system: system design interview question

> Daily model updates with delayed labels, biased exposure, reproducible examples, and release controls that limit risk and detect harm.

Source: https://sysround.com/questions/design-continuous-learning-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a system that updates production models from new examples and labels that arrive later. The system must limit risks from harmful feedback loops and model regressions. It must detect harm and stop or reverse unsafe model releases."

## What to clarify first

- Prediction task
- Scale and request mix
- Label delay
- Update frequency
- Latency
- Availability and consistency
- Durability and retention
- Quality and feedback
- Release evidence and risk
- Scope limits

## Scope of a strong answer

Core requirements:

- Collect production examples and attach delayed labels.
- Create reproducible datasets with the feature values available at prediction time.
- Train and evaluate model candidates each day.
- Limit exposure to harmful updates, detect harm, and stop or reverse model releases.

Usually out of scope:

- Research new model algorithms.
- Build the upstream candidate item service.
- Define purchase attribution rules.
- Support model updates across multiple regions.

## What interviewers probe

### Examples and delayed outcomes

Missing labels can mean no purchase or an outcome that has not arrived. Features can also change before the label arrives.

### Which outcomes the model sees

The model changes which items users see. Later examples can reinforce those choices without evidence that those choices help users.

### Model release decisions

Production purchase labels mature after eight days. Daily candidates can overlap before the system measures their effects, which creates risk during experiments.

### History of model updates

Late events, retries, and concurrent runs can change historical results. An older run can also replace a newer release without explicit release checks.

## Common mistakes

- Treats absent labels as negative outcomes before the outcome window closes.
- Uses current feature values to reconstruct old examples.
- Assumes that a good offline score proves a better recommendation policy.
- Updates production weights directly from each label without release checks.
- Ignores the effect of model choices on future examples.
- Treats rollback as proof of safety and continues to reuse harmful data.

## The bar by level

- **Senior:** Defines point-in-time examples, reproducible datasets, and model lineage. Handles label outages, biased exposure, delayed experiments, and conditional release changes. Sets explicit limits and automatic stop rules.
- **Staff+:** Sets release cadence from label delay and experiment capacity. Defines exposure support, persistent controls, statistical decision rules, and acceptable risk. Plans recovery across model generations when harmful data affects later updates.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=continuous-model-learning


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.