# Design a data labeling platform: system design interview question

> Human annotation at scale: independent labels, submission retries, quality evidence, cost limits, and reproducible dataset releases.

Source: https://sysround.com/questions/design-data-labeling-platform
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a platform that gives annotation tasks to human labelers and produces versioned training datasets. Balance task throughput, cost, and annotation quality."

## What to clarify first

- Annotation format
- Scale and record size
- Initial annotation policy
- Quality and releases
- Task time and cost
- Read and write mix
- Latency
- Consistency and availability
- Durability and retention
- Scope limits

## Scope of a strong answer

Core requirements:

- Import images with a fixed label schema.
- Assign tasks to labelers and accept their annotations.
- Check annotation quality and enforce cost limits.
- Publish immutable dataset releases with source references and label history.

Usually out of scope:

- Train machine learning models.
- Recruit labelers or process their payments.
- Support video or geometric annotations.
- Operate across several regions.

## What interviewers probe

### Task ownership and submission retries

Assignments can expire while requests remain active. Concurrent requests can break label independence, and retries can duplicate accepted work or cost.

### Annotation quality and cost

Agreement can hide shared errors. Required reviews and audits create future costs before the platform knows each item's final label.

### Dataset releases

Labels and source files can change during export. A release must preserve one consistent set despite corrections, retries, and partial failures.

## Common mistakes

- Trusts a shared queue to prevent duplicate assignments.
- Checks labeler eligibility without concurrency protection across both task slots.
- Treats label agreement as proof of annotation accuracy.
- Reserves only immediate task costs and leaves required reviews or audits without funds.
- Overwrites accepted labels or source files after publication.
- Adds database shards before checking the modest request rate and the much larger human labor cost.

## The bar by level

- **Senior:** Prevents expiry races and concurrent claims across independent slots. Separates agreement from audited accuracy. Funds mandatory downstream work before admission. Freezes release membership and makes export retries safe.
- **Staff+:** Defines statistical release gates and class evidence. Proves cost bounds across concurrent admission and review spikes. Defines funded recovery after failed audits. Resolves budget, throughput, and quality conflicts. Explains consistent release cuts, source retention, and failover without lost annotations.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=training-data-annotation


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.