# Design an LLM evaluation platform: system design interview question

> Model comparisons with reproducible experiments, fair judge scores, traceable human decisions, and bounded evaluation cost.

Source: https://sysround.com/questions/design-llm-evaluation-platform
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a platform that compares model and prompt versions with benchmark datasets, automated judges, and human review. Teams need reproducible results and control over evaluation cost and judge bias."

## What to clarify first

- Tasks and scope
- Scale
- Tokens and prices
- Judges and human review
- Traffic mix and latency
- Reports
- Reproducibility and retention
- Consistency and durability
- Cost limits
- Data access

## Scope of a strong answer

Core requirements:

- Compare model and prompt versions on a benchmark dataset
- Run automated metrics and judges within a spend limit
- Collect human reviews and report comparison results
- Preserve run inputs, outputs, scores, and review history

Usually out of scope:

- Train models or judges
- Build a model inference service
- Deploy models to production
- Create benchmark content

## What interviewers probe

### Run identity and result history

Prompts, datasets, judges, and provider models can change independently. A repeated request does not guarantee the same answer.

### Run progress and evaluation cost

Worker retries can cause duplicate charges. Active calls and unknown provider outcomes make spend limits harder than a final cost total.

### Judge scores and human decisions

Judges can prefer answer position, length, or model style. Human reviewers can share biases or disagree with each other.

### Result comparisons

Missing answers can distort scores. Small differences, repeated trials, and repeated benchmark use can produce false claims of improvement.

## Common mistakes

- The design stores mutable model aliases instead of complete run inputs.
- The design treats a random seed as a guarantee of identical hosted outputs.
- The design treats an automated judge as an unbiased source of truth.
- Reports compare different example sets or silently omit failed calls.
- The system checks costs only after calls finish.
- Reports combine targeted human reviews with random samples without accounting for selection.

## The bar by level

- **Senior:** Defines immutable manifests and score history. Handles duplicate work, unknown provider outcomes, and atomic cost reservations. Uses blind review, judge calibration, paired comparisons, and explicit failure coverage.
- **Staff+:** Defines which reproducibility guarantees the platform can prove. Resolves quota, budget, and failure conflicts across organizations. Designs unbiased review samples and defensible release evidence. Explains how provider changes and data deletion affect past reports.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=llm-evaluation


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.