Design an LLM evaluation platform: system design interview question

How the interviewer opens it

“Design a platform that compares model and prompt versions with benchmark datasets, automated judges, and human review. Teams need reproducible results and control over evaluation cost and judge bias.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

Tasks and scope
Scale
Tokens and prices
Judges and human review
Traffic mix and latency
Reports
Reproducibility and retention
Consistency and durability
Cost limits
Data access

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

Core requirements

Compare model and prompt versions on a benchmark dataset
Run automated metrics and judges within a spend limit
Collect human reviews and report comparison results
Preserve run inputs, outputs, scores, and review history

Usually out of scope

Train models or judges
Build a model inference service
Deploy models to production
Create benchmark content

What interviewers ask about

Run identity and result history: Prompts, datasets, judges, and provider models can change independently. A repeated request does not guarantee the same answer.
Run progress and evaluation cost: Worker retries can cause duplicate charges. Active calls and unknown provider outcomes make spend limits harder than a final cost total.
Judge scores and human decisions: Judges can prefer answer position, length, or model style. Human reviewers can share biases or disagree with each other.
Result comparisons: Missing answers can distort scores. Small differences, repeated trials, and repeated benchmark use can produce false claims of improvement.

Common mistakes

The design stores mutable model aliases instead of complete run inputs.
The design treats a random seed as a guarantee of identical hosted outputs.
The design treats an automated judge as an unbiased source of truth.
Reports compare different example sets or silently omit failed calls.
The system checks costs only after calls finish.
Reports combine targeted human reviews with random samples without accounting for selection.

The bar by level

Senior (L5/E5): Defines immutable manifests and score history. Handles duplicate work, unknown provider outcomes, and atomic cost reservations. Uses blind review, judge calibration, paired comparisons, and explicit failure coverage.
Staff (L6+/E6+): Defines which reproducibility guarantees the platform can prove. Resolves quota, budget, and failure conflicts across organizations. Designs unbiased review samples and defensible release evidence. Explains how provider changes and data deletion affect past reports.

Sources · 1 reference

These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.

SysRound · 1 record

No company association
Design an LLM evaluation platform
Reported date not specified
Source snapshot: Nov 12, 2026
View source

How the interviewer opens it

“Design a platform that compares model and prompt versions with benchmark datasets, automated judges, and human review. Teams need reproducible results and control over evaluation cost and judge bias.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

Scope of a strong answer

Core requirements

Compare model and prompt versions on a benchmark dataset
Run automated metrics and judges within a spend limit
Collect human reviews and report comparison results
Preserve run inputs, outputs, scores, and review history

Usually out of scope

Train models or judges
Build a model inference service
Deploy models to production
Create benchmark content

What interviewers ask about

Run identity and result history

Prompts, datasets, judges, and provider models can change independently. A repeated request does not guarantee the same answer.

Run progress and evaluation cost

Worker retries can cause duplicate charges. Active calls and unknown provider outcomes make spend limits harder than a final cost total.

Judge scores and human decisions

Judges can prefer answer position, length, or model style. Human reviewers can share biases or disagree with each other.

Result comparisons

Missing answers can distort scores. Small differences, repeated trials, and repeated benchmark use can produce false claims of improvement.

Common mistakes

The design stores mutable model aliases instead of complete run inputs.

The design treats a random seed as a guarantee of identical hosted outputs.

The design treats an automated judge as an unbiased source of truth.

Reports compare different example sets or silently omit failed calls.

The system checks costs only after calls finish.

Reports combine targeted human reviews with random samples without accounting for selection.

The bar by level

Senior (L5/E5)

Defines immutable manifests and score history. Handles duplicate work, unknown provider outcomes, and atomic cost reservations. Uses blind review, judge calibration, paired comparisons, and explicit failure coverage.

Staff (L6+/E6+)

Defines which reproducibility guarantees the platform can prove. Resolves quota, budget, and failure conflicts across organizations. Designs unbiased review samples and defensible release evidence. Explains how provider changes and data deletion affect past reports.