# Design an automatic video captioning system: system design interview question

> Caption accuracy, alignment with video time, and predictable completion times across languages, noise levels, and video lengths.

Source: https://sysround.com/questions/design-automatic-video-captioning-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a system that generates time-aligned captions for uploaded videos. It must support multiple languages and noisy audio while keeping processing time manageable."

## What to clarify first

- Workload
- Read and write mix
- Languages
- Caption quality
- Latency
- Consistency and durability
- Access and retention
- Media and time
- Invalid media and absent speech
- Scope boundaries

## Scope of a strong answer

Core requirements:

- Generate captions for speech in the source language
- Create one translated track when the user requests it
- Keep caption times aligned with the video
- Report job status and return caption files

Usually out of scope:

- Live captions
- Model training
- Video playback infrastructure
- Human review services

## What interviewers probe

### Caption times

Audio transforms, parallel work, and translation can each break the relationship between caption times and the original video clock.

### Speech quality across languages

Noise and language changes produce different errors. A single average accuracy score can hide poor results for one language.

### Completion time under load

Job counts hide large differences in audio work. Queue delays can consume the latency budget before a model starts.

### Repeated requests and failures

Retries can repeat work. Zone failures can lose accepted jobs or expose incomplete captions unless each storage step has a clear durability contract.

## Common mistakes

- Runs the full caption job inside the upload request.
- Estimates capacity from video counts instead of audio duration.
- Divides audio without preserving source times or resolving boundary errors.
- Treats translation as direct speech recognition in the target language.
- Promises accurate text for unclear speech without uncertainty or language-specific evaluation.
- Ignores duplicate work, stale workers, partial caption files, or durability across zones.

## The bar by level

- **Senior:** Explains boundary reconciliation, language quality checks, and safe publication. Budgets queue and model time. Handles duplicate work and specifies storage behavior during zone failure.
- **Staff+:** Defines measurable quality and latency contracts. Balances model choice, fleet cost, and fairness. Specifies acceptance limits, revision ownership, and recovery across storage boundaries.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=video-captioning


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.