# Design an active learning system: system design interview question

> A model improvement loop with a fixed annotation budget, delayed human judgments, and uncertain gains.

Source: https://sysround.com/questions/design-active-learning-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a system that selects unlabeled examples for human annotation and retrains a model from the results. The goal is the largest model improvement within a fixed annotation budget."

## What to clarify first

- Task and objective
- Example volume
- Annotation budget
- Human annotation
- Read and write mix
- Latency and cadence
- Consistency and availability
- Durability and retention
- Compute
- Scope limits

## Scope of a strong answer

Core requirements:

- Select examples within the annotation budget
- Assign annotation tasks and retain human judgments
- Retrain a model from approved labels
- Measure model improvement before model release

Usually out of scope:

- Recruit annotators or process payments
- Design a new model architecture
- Build the production prediction service

## What interviewers probe

### Which examples need labels

Uncertain examples can be duplicates, noise, or cases that the model cannot learn. Each poor choice consumes a limited budget.

### Annotation budget

Several jobs can assign work at once. Delayed results and retries can create charges after the service appears to have free funds.

### Human judgments

Human labels can conflict or arrive late. A label correction must not change the recorded inputs of a completed training run.

### Model improvement

Selected labels do not represent live traffic. Repeated evaluation can also make a model appear better without real gains.

## Common mistakes

- Select only low-confidence examples and ignore duplicates or confident errors.
- Count examples instead of paid judgments, including reviews.
- Release reserved funds on a timeout while a late result can still incur a charge.
- Train on every human judgment without checks for disagreement or annotator errors.
- Measure improvement on the examples that the system selects for annotation.
- Train from mutable tables and lose the exact inputs for an earlier model.

## The bar by level

- **Senior:** Defines concurrent budget reservations and task recovery. Combines selection signals with exploration. Handles label revisions, immutable snapshots, and release decisions based on independent model scores.
- **Staff+:** Designs equal-cost policy experiments and addresses evaluation reuse. Bounds selection compute as the pool grows. Defines recovery across task and budget systems. Adapts the loop to traffic changes without weakening the budget limit.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=active-learning


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.