Design an LLM serving platform: system design interview question

How the interviewer opens it

“Design a platform that serves large language models and streams each response to the client. Use GPUs efficiently while meeting latency targets as traffic and request lengths change.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

Model
Hardware
Traffic and request mix
Request lengths
Latency
Correctness and availability
Transport and durability
Client fairness
Cancellation and retries
Scope limits

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

Core requirements

Accept text prompts and stream output tokens
Share GPU capacity across requests with different lengths
Meet latency targets and reject excess load
Enforce client quotas and release resources after cancellation

Usually out of scope

Train or fine-tune models
Accept model uploads or change model versions
Retrieve documents or execute tools
Resume a stream after a failed request
Serve images, audio, or video

What interviewers ask about

Requests with different lengths: Prompt computation and token decode use GPU resources differently. A long prompt can delay output for many active requests.
GPU memory limits: The model occupies memory before any request starts. Each prompt and output token then adds request state with an uncertain lifetime.
Traffic bursts and client fairness: Equal request counts can represent very different GPU costs. Bursts arrive faster than new replicas can load the model.
Stream failures and client behavior: A client can observe part of an output before a failure. Slow or absent clients can also retain scarce GPU state.

Common mistakes

Uses GPU utilization as the success measure while admitted requests miss latency targets.
Treats request count as a reliable measure of GPU work.
Counts model weights but ignores request memory growth and temporary allocations.
Uses large fixed batches without addressing long prompts or requests that finish early.
Adds replicas without measuring model load time or combined prompt and output capacity.
Retries a partially visible output without defining stream failure behavior.

The bar by level

Senior (L5/E5): Defines how requests share GPU time and memory as they arrive and finish. Connects capacity estimates to both latency targets through benchmarks. Handles output growth, client fairness, cancellation, and slow clients.
Staff (L6+/E6+): Builds a quantitative capacity plan across request lengths and replica failures. Defines stable load controls and fair service under overload. Justifies the replica layout and any separate prompt or decode workers with measurements. Explains warm capacity costs and the limits of each failure guarantee.

Sources · 1 reference

These records explain where the question appears. A company tag or candidate report does not guarantee a future interview question.

SysRound · 1 record

No company association
Design an LLM serving platform
Reported date not specified
Source snapshot: Nov 12, 2026
View source

How the interviewer opens it

“Design a platform that serves large language models and streams each response to the client. Use GPUs efficiently while meeting latency targets as traffic and request lengths change.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

Scope of a strong answer

Core requirements

Accept text prompts and stream output tokens
Share GPU capacity across requests with different lengths
Meet latency targets and reject excess load
Enforce client quotas and release resources after cancellation

Usually out of scope

Train or fine-tune models
Accept model uploads or change model versions
Retrieve documents or execute tools
Resume a stream after a failed request
Serve images, audio, or video

What interviewers ask about

Requests with different lengths

Prompt computation and token decode use GPU resources differently. A long prompt can delay output for many active requests.

GPU memory limits

The model occupies memory before any request starts. Each prompt and output token then adds request state with an uncertain lifetime.

Traffic bursts and client fairness

Equal request counts can represent very different GPU costs. Bursts arrive faster than new replicas can load the model.

Stream failures and client behavior

A client can observe part of an output before a failure. Slow or absent clients can also retain scarce GPU state.

Common mistakes

Uses GPU utilization as the success measure while admitted requests miss latency targets.

Treats request count as a reliable measure of GPU work.

Counts model weights but ignores request memory growth and temporary allocations.

Uses large fixed batches without addressing long prompts or requests that finish early.

Adds replicas without measuring model load time or combined prompt and output capacity.

Retries a partially visible output without defining stream failure behavior.

The bar by level

Senior (L5/E5)

Defines how requests share GPU time and memory as they arrive and finish. Connects capacity estimates to both latency targets through benchmarks. Handles output growth, client fairness, cancellation, and slow clients.

Staff (L6+/E6+)

Builds a quantitative capacity plan across request lengths and replica failures. Defines stable load controls and fair service under overload. Justifies the replica layout and any separate prompt or decode workers with measurements. Explains warm capacity costs and the limits of each failure guarantee.