# Design a data lineage tracking system: system design interview question

> Dataset and column dependencies across pipeline changes: historical views, incomplete source reports, and large impact reports.

Source: https://sysround.com/questions/design-data-lineage-tracking-system
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a system that tracks how datasets and columns depend on upstream data. Users must explore dependencies and assess the effect of a change as pipelines evolve."

## What to clarify first

- Catalog size
- Traffic and read:write mix
- Latency and freshness
- Source capabilities
- Meaning of impact
- Current and historical views
- Identity
- Availability and consistency
- Durability and retention
- Scope limits

## Scope of a strong answer

Core requirements:

- Record dataset and column dependencies from source metadata.
- Explore upstream sources and downstream dependencies.
- Identify datasets and columns that a proposed change can affect.
- Query current dependencies and dependencies at a past time.

Usually out of scope:

- Track individual rows or data values.
- Build an analyzer for arbitrary program code.
- Execute pipelines or enforce deployment rules.
- Support multiple organizations with separate access rules.

## What interviewers probe

### Changes to pipelines

Reports can arrive out of order. A new pipeline definition can remove dependencies or restore old logic.

### Column dependencies

Column names can change. Filters and joins affect results without directly supplying output values. Some sources cannot describe their logic.

### Large dependency reports

Shared columns can reach huge dependency sets. Pipeline loops and concurrent changes make complete reports difficult.

## Common mistakes

- Treat the latest received event as the latest deployed definition.
- Keep old edges active forever and report dependencies that no longer exist.
- Treat absent column detail as proof of no dependency.
- Match columns only by name.
- Claim a complete impact result after a query reaches its work limit.
- Build a full SQL interpreter before defining the metadata contract.

## The bar by level

- **Senior:** Defines deployment-time history and atomic revision publication. Handles late reports, removals, and unknown column detail. Bounds report work and calculates storage costs.
- **Staff+:** Defines source authority and limits on completeness. Separates deployment history from execution provenance. Sets snapshot rules for large reports. Explains recovery and store partition choices.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=data-lineage


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.