# Design an ML model monitoring system: system design interview question

> Model health checks with data drift, prediction drift, delayed labels, limits on quality claims, and alerts with useful evidence.

Source: https://sysround.com/questions/design-ml-model-monitoring-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a service that detects changes in production model inputs and predictions. Measure model quality only where the available evidence supports the claim. Labels can arrive days later, and the service must send useful alerts."

## What to clarify first

- Models and events
- Scale
- Read and write mix
- Delayed labels
- Reference data and segments
- Quality claims
- Latency
- Consistency and durability
- Event size and retention
- Scope and response

## Scope of a strong answer

Core requirements:

- Collect prediction events and delayed labels
- Detect input drift and prediction drift by model version and segment
- Measure quality where the available labels support the claim
- Show results and send alerts with useful evidence

Usually out of scope:

- Train models or serve predictions
- Change models automatically
- Build an email service or an on-call service
- Prove that data drift caused quality loss

## What interviewers probe

### Which traffic to compare

A new model version or a different traffic mix can change distributions without a model defect.

### Quality before all labels arrive

Label delays can distort quality estimates. Outcome-dependent label availability can make population quality changes impossible to determine from the available data.

### Event accuracy and cost

Labels can arrive days after predictions. Duplicate events and traffic skew can corrupt results or overload workers.

### Alerts that operators can use

Many related checks can overwhelm operators. Delayed or selective labels can prevent immediate claims about model quality.

## Common mistakes

- Treats data drift or prediction drift as proof of quality loss
- Compares model versions or traffic segments without a compatible reference
- Groups quality by label arrival time instead of prediction time
- Assumes missing labels are negative outcomes or that mature labels represent all predictions
- Ignores duplicate events, late labels, or the size of the join state
- Creates many alerts without sample limits, evidence limits, or a responsible owner

## The bar by level

- **Senior:** Handles prediction cohorts, label maturity, selection bias, replay, traffic skew, and bounded state. Separates drift warnings, observed-label quality changes, and unsupported population claims.
- **Staff+:** Defines which quality claims the evidence supports and when representative audit labels become necessary. Sets reference governance, cardinality limits, recovery guarantees, and alert policy.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=production-model-monitoring


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.