# Design an LLM serving platform: system design interview question

> GPU use under uneven load: first-token latency, token gaps, request memory, and fair client access.

Source: https://sysround.com/questions/design-llm-serving-platform
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a platform that serves large language models and streams each response to the client. Use GPUs efficiently while meeting latency targets as traffic and request lengths change."

## What to clarify first

- Model
- Hardware
- Traffic and request mix
- Request lengths
- Latency
- Correctness and availability
- Transport and durability
- Client fairness
- Cancellation and retries
- Scope limits

## Scope of a strong answer

Core requirements:

- Accept text prompts and stream output tokens
- Share GPU capacity across requests with different lengths
- Meet latency targets and reject excess load
- Enforce client quotas and release resources after cancellation

Usually out of scope:

- Train or fine-tune models
- Accept model uploads or change model versions
- Retrieve documents or execute tools
- Resume a stream after a failed request
- Serve images, audio, or video

## What interviewers probe

### Requests with different lengths

Prompt computation and token decode use GPU resources differently. A long prompt can delay output for many active requests.

### GPU memory limits

The model occupies memory before any request starts. Each prompt and output token then adds request state with an uncertain lifetime.

### Traffic bursts and client fairness

Equal request counts can represent very different GPU costs. Bursts arrive faster than new replicas can load the model.

### Stream failures and client behavior

A client can observe part of an output before a failure. Slow or absent clients can also retain scarce GPU state.

## Common mistakes

- Uses GPU utilization as the success measure while admitted requests miss latency targets.
- Treats request count as a reliable measure of GPU work.
- Counts model weights but ignores request memory growth and temporary allocations.
- Uses large fixed batches without addressing long prompts or requests that finish early.
- Adds replicas without measuring model load time or combined prompt and output capacity.
- Retries a partially visible output without defining stream failure behavior.

## The bar by level

- **Senior:** Defines how requests share GPU time and memory as they arrive and finish. Connects capacity estimates to both latency targets through benchmarks. Handles output growth, client fairness, cancellation, and slow clients.
- **Staff+:** Builds a quantitative capacity plan across request lengths and replica failures. Defines stable load controls and fair service under overload. Justifies the replica layout and any separate prompt or decode workers with measurements. Explains warm capacity costs and the limits of each failure guarantee.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=llm-inference


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.