Design an end-to-end ML pipeline platform

How the interviewer opens it

“Design a platform for data preparation, model training, validation, and deployment. Teams must reproduce runs and recover from individual stage failures.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

• Workflow structure
• Scale and compute capacity
• Data size and retention
• Read and write mix
• Latency
• Consistency and durability
• Reproducibility
• Validation and approval
• Prediction service capabilities
• Existing systems and boundaries

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

What interviewers ask about

Run reproducibility

A workflow definition does not capture changes to data, libraries, or hardware behavior. A later run can produce different results without an obvious cause.

Stage failures and recovery

Long stages can finish work without a success report. Retries must not expose partial outputs or replace a result that the platform already accepted.

Shared compute capacity

Long training stages consume scarce GPUs. A large submission burst or repeated failures can delay every team.

Model release failures

Validation, approval, and deployment must refer to one artifact. A timeout can hide a completed release or a request that will act later.

Common mistakes

• The design records the workflow but omits the data and environment versions.
• The system repeats successful stages after any failure.
• Dependent stages read partial output files.
• The design assumes that each stage executes exactly once.
• The design treats resource requests as a rule that excludes CPU jobs from GPU nodes.
• The design assumes that local release IDs prevent duplicate remote deployments without a service contract.

The bar by level

Senior (L5/E5)

Defines attempt state, output acceptance, and recovery after ambiguous failures. Pins run inputs and environments. Uses GPU placement controls and team quotas. Handles deployment retries through explicit service guarantees.

Staff (L6+/E6+)

States correctness rules across controllers, workers, storage, and the prediction service. Resolves stale workers and concurrent releases. Defines safe limits when remote capabilities are absent. Explains reproducibility limits, retention duties, and capacity promises.