Design an active learning system: system design interview question

How the interviewer opens it

“Design a system that selects unlabeled examples for human annotation and retrains a model from the results. The goal is the largest model improvement within a fixed annotation budget.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

Task and objective
Example volume
Annotation budget
Human annotation
Read and write mix
Latency and cadence
Consistency and availability
Durability and retention
Compute
Scope limits

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

Core requirements

Select examples within the annotation budget
Assign annotation tasks and retain human judgments
Retrain a model from approved labels
Measure model improvement before model release

Usually out of scope

Recruit annotators or process payments
Design a new model architecture
Build the production prediction service

What interviewers ask about

Which examples need labels: Uncertain examples can be duplicates, noise, or cases that the model cannot learn. Each poor choice consumes a limited budget.
Annotation budget: Several jobs can assign work at once. Delayed results and retries can create charges after the service appears to have free funds.
Human judgments: Human labels can conflict or arrive late. A label correction must not change the recorded inputs of a completed training run.
Model improvement: Selected labels do not represent live traffic. Repeated evaluation can also make a model appear better without real gains.

Common mistakes

Select only low-confidence examples and ignore duplicates or confident errors.
Count examples instead of paid judgments, including reviews.
Release reserved funds on a timeout while a late result can still incur a charge.
Train on every human judgment without checks for disagreement or annotator errors.
Measure improvement on the examples that the system selects for annotation.
Train from mutable tables and lose the exact inputs for an earlier model.

The bar by level

Senior (L5/E5): Defines concurrent budget reservations and task recovery. Combines selection signals with exploration. Handles label revisions, immutable snapshots, and release decisions based on independent model scores.
Staff (L6+/E6+): Designs equal-cost policy experiments and addresses evaluation reuse. Bounds selection compute as the pool grows. Defines recovery across task and budget systems. Adapts the loop to traffic changes without weakening the budget limit.

Sources · 1 reference

These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.

SysRound · 1 record

No company association
Design an active learning system
Reported date not specified
Source snapshot: Nov 12, 2026
View source

How the interviewer opens it

“Design a system that selects unlabeled examples for human annotation and retrains a model from the results. The goal is the largest model improvement within a fixed annotation budget.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

Scope of a strong answer

Core requirements

Select examples within the annotation budget
Assign annotation tasks and retain human judgments
Retrain a model from approved labels
Measure model improvement before model release

Usually out of scope

Recruit annotators or process payments
Design a new model architecture
Build the production prediction service

What interviewers ask about

Which examples need labels

Uncertain examples can be duplicates, noise, or cases that the model cannot learn. Each poor choice consumes a limited budget.

Annotation budget

Several jobs can assign work at once. Delayed results and retries can create charges after the service appears to have free funds.

Human judgments

Human labels can conflict or arrive late. A label correction must not change the recorded inputs of a completed training run.

Model improvement

Selected labels do not represent live traffic. Repeated evaluation can also make a model appear better without real gains.

Common mistakes

Select only low-confidence examples and ignore duplicates or confident errors.

Count examples instead of paid judgments, including reviews.

Release reserved funds on a timeout while a late result can still incur a charge.

Train on every human judgment without checks for disagreement or annotator errors.

Measure improvement on the examples that the system selects for annotation.

Train from mutable tables and lose the exact inputs for an earlier model.

The bar by level

Senior (L5/E5)

Defines concurrent budget reservations and task recovery. Combines selection signals with exploration. Handles label revisions, immutable snapshots, and release decisions based on independent model scores.

Staff (L6+/E6+)

Designs equal-cost policy experiments and addresses evaluation reuse. Bounds selection compute as the pool grows. Defines recovery across task and budget systems. Adapts the loop to traffic changes without weakening the budget limit.