# Design a near-duplicate content detection system: system design interview question

> The service must find edited copies across billions of items without a full scan or long delays.

Source: https://sysround.com/questions/design-near-duplicate-content-detection-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a service that finds duplicate and slightly modified posts or images across a social platform. The platform adds new content at a high rate."

## What to clarify first

- Supported edits
- Ingestion scale
- Content size and storage
- Search window and deletion
- Read and write mix
- Latency
- Match quality
- Consistency and availability
- Durability
- Results and exclusions

## Scope of a strong answer

Core requirements:

- Find exact duplicates of text posts and images.
- Find copies with small text or image changes.
- Return matching item IDs and scores.
- Check new content against the most recent 30 days.

Usually out of scope:

- Detect semantic paraphrases or different images of the same subject.
- Compare text with images.
- Detect duplicate video or audio.
- Develop new models or build the original content store.
- Remove content or make moderation decisions.

## What interviewers probe

### Supported edits and match quality

Small changes can hide a copy. Similar subjects can also create false matches between content that has no common source.

### Search cost across the corpus

The service must search billions of items without losing edited copies. Common text and popular images can create very large candidate sets.

### High-rate ingestion and freshness

Two copies can arrive before either becomes searchable. Index delays and worker failures can turn a temporary gap into a permanent missed match.

## Common mistakes

- Candidates use exact hashes alone despite the required edits.
- Candidates treat semantic similarity as proof that one item copies another.
- Candidates choose a candidate index without a recall target or a verification step.
- Candidates ignore common captions and popular images that create large candidate sets.
- Candidates assume asynchronous index updates cannot cause permanent missed matches.
- Candidates keep every feature after the search window ends.

## The bar by level

- **Senior:** Selects features and indexes for the supported edits. Separates retrieval from verification. Quantifies search cost and handles simultaneous arrivals, hot content, and deletion.
- **Staff+:** Defines measurable quality and freshness contracts. Plans index recovery, feature migrations, and controlled degradation. Explains cost and recall trade-offs without promising perfect approximate search.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=duplicate-content-detection


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.