# Design an LLM fine-tuning platform: system design interview question

> A private model training service must share scarce GPUs, protect customer data, and recover long jobs without exposing incomplete artifacts.

Source: https://sysround.com/questions/design-llm-fine-tuning-platform
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a platform that lets customers fine-tune foundation models on private datasets. Customers share GPU capacity, but each customer's data and model artifacts must stay private."

## What to clarify first

- Scale
- Supported jobs
- GPU capacity
- Data size and retention
- Read and write mix
- Latency
- Fairness and cancellation
- Privacy
- Consistency and durability
- Boundaries

## Scope of a strong answer

Core requirements:

- Upload and validate private datasets.
- Submit, inspect, and cancel fine-tune jobs.
- Share GPU capacity across customers without exposing their data.
- Retrieve private model artifacts and evaluation results.

Usually out of scope:

- Online inference.
- Full-parameter tuning and jobs that need several GPUs.
- Customer code and customer-supplied foundation models.
- Payment collection.
- Recovery across regions.

## What interviewers probe

### GPU capacity and job delays

Jobs occupy scarce GPUs for long periods. Input size, sequence length, and customer bursts make job duration and resource use uneven.

### Customer data boundaries

The platform reuses hardware across customers. Private data can remain in memory, temporary files, checkpoints, logs, and model state.

### Job failure, cancellation, and final results

Long GPU jobs can outlive their assigned workers. Cancellation must stop GPU execution promptly while final results compete for publication.

## Common mistakes

- Treats fine-tuning as a short HTTP request and keeps job state only in worker memory.
- Uses one global queue without customer quotas or limits on job duration.
- Assumes that a container or an encrypted bucket prevents every form of data exposure.
- Treats audit records as permission for staff to access customer data without prior approval.
- Shares mutable model state or exposes artifacts before publication commits.
- Reports cancellation without a bounded path that stops GPU execution.

## The bar by level

- **Senior:** Defines fair GPU allocation, memory limits, and safe worker reuse. Handles retries and artifact publication races. Gives cancellation intervals and termination deadlines. Requires approval before staff access and audits each grant.
- **Staff+:** Defines the isolation threat model and prevents staff access paths that bypass approval. Sets measurable capacity and fairness policies. Explains ownership and termination during partitions. Separates stop confirmation from cleanup. Adapts the design to larger models or strict start deadlines.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=llm-fine-tuning


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.