Design a data labeling platform: system design interview question

How the interviewer opens it

“Design a platform that gives annotation tasks to human labelers and produces versioned training datasets. Balance task throughput, cost, and annotation quality.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

Annotation format
Scale and record size
Initial annotation policy
Quality and releases
Task time and cost
Read and write mix
Latency
Consistency and availability
Durability and retention
Scope limits

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

Core requirements

Import images with a fixed label schema.
Assign tasks to labelers and accept their annotations.
Check annotation quality and enforce cost limits.
Publish immutable dataset releases with source references and label history.

Usually out of scope

Train machine learning models.
Recruit labelers or process their payments.
Support video or geometric annotations.
Operate across several regions.

What interviewers ask about

Task ownership and submission retries: Assignments can expire while requests remain active. Concurrent requests can break label independence, and retries can duplicate accepted work or cost.
Annotation quality and cost: Agreement can hide shared errors. Required reviews and audits create future costs before the platform knows each item's final label.
Dataset releases: Labels and source files can change during export. A release must preserve one consistent set despite corrections, retries, and partial failures.

Common mistakes

Trusts a shared queue to prevent duplicate assignments.
Checks labeler eligibility without concurrency protection across both task slots.
Treats label agreement as proof of annotation accuracy.
Reserves only immediate task costs and leaves required reviews or audits without funds.
Overwrites accepted labels or source files after publication.
Adds database shards before checking the modest request rate and the much larger human labor cost.

The bar by level

Senior (L5/E5): Prevents expiry races and concurrent claims across independent slots. Separates agreement from audited accuracy. Funds mandatory downstream work before admission. Freezes release membership and makes export retries safe.
Staff (L6+/E6+): Defines statistical release gates and class evidence. Proves cost bounds across concurrent admission and review spikes. Defines funded recovery after failed audits. Resolves budget, throughput, and quality conflicts. Explains consistent release cuts, source retention, and failover without lost annotations.

Sources · 1 reference

These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.

SysRound · 1 record

No company association
Design a data labeling platform
Reported date not specified
Source snapshot: Nov 12, 2026
View source

How the interviewer opens it

“Design a platform that gives annotation tasks to human labelers and produces versioned training datasets. Balance task throughput, cost, and annotation quality.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

Scope of a strong answer

Core requirements

Import images with a fixed label schema.
Assign tasks to labelers and accept their annotations.
Check annotation quality and enforce cost limits.
Publish immutable dataset releases with source references and label history.

Usually out of scope

Train machine learning models.
Recruit labelers or process their payments.
Support video or geometric annotations.
Operate across several regions.

What interviewers ask about

Task ownership and submission retries

Assignments can expire while requests remain active. Concurrent requests can break label independence, and retries can duplicate accepted work or cost.

Annotation quality and cost

Agreement can hide shared errors. Required reviews and audits create future costs before the platform knows each item's final label.

Dataset releases

Labels and source files can change during export. A release must preserve one consistent set despite corrections, retries, and partial failures.

Common mistakes

Trusts a shared queue to prevent duplicate assignments.

Checks labeler eligibility without concurrency protection across both task slots.

Treats label agreement as proof of annotation accuracy.

Reserves only immediate task costs and leaves required reviews or audits without funds.

Overwrites accepted labels or source files after publication.

Adds database shards before checking the modest request rate and the much larger human labor cost.

The bar by level

Senior (L5/E5)

Prevents expiry races and concurrent claims across independent slots. Separates agreement from audited accuracy. Funds mandatory downstream work before admission. Freezes release membership and makes export retries safe.

Staff (L6+/E6+)

Defines statistical release gates and class evidence. Proves cost bounds across concurrent admission and review spikes. Defines funded recovery after failed audits. Resolves budget, throughput, and quality conflicts. Explains consistent release cuts, source retention, and failover without lost annotations.