Design a distributed build system like Bazel Remote

How the interviewer opens it

“Design a remote service that runs large software builds and reuses previous results. Builds contain dependent actions. The service must preserve reproducibility when build owners supply deterministic commands and complete input declarations.”

The prompt is vague on purpose. The first minutes are for you. Reduce the prompt to the core features and agree on the numbers before you draw.

What to clarify first

• Scale
• Cache reuse and read-to-write ratio
• Action resources
• File volume
• Latency
• Build graph and client behavior
• Reproducibility
• Correctness, availability, and durability
• Retention and access
• Scope boundaries

The interviewer has the answers. In a SysRound round, Alex gives these answers, as a real interviewer does.

Scope of a strong answer

What interviewers ask about

Repeated build results

A compiler, environment value, or hidden input can change outputs without changing the source. Incorrect reuse can silently corrupt a build.

Build progress

Large graphs expose parallel work, but each action needs complete inputs. Duplicate events or coordinator failures can release work too early.

Worker failures

A worker can finish an action without reporting success. Replacement attempts can overlap, but the service needs one authoritative outcome.

File storage and reuse

Reusable results depend on many files. Zone failures and deletion races threaten durability, while last-use retention prevents a fixed storage bound.

Common mistakes

• The action key omits toolchains, environment values, or transitive inputs.
• The design assumes that containers make arbitrary commands reproducible.
• The coordinator releases dependents before their input files become durable.
• The design treats worker retries as exactly-once execution.
• The capacity estimate counts actions but ignores file transfer and worker memory.
• The design assumes that seven-day retention starts at creation, or acknowledges data before separate zones durably store it.

The bar by level

Senior (L5/E5)

Explains reproducibility limits, durable dependency progress, and stale attempt rejection. Sizes workers and file traffic. Handles cancellation, incomplete results, last-use retention, and zone-safe acknowledgment.

Staff (L6+/E6+)

Defines correctness boundaries across SQL, workers, and object storage. Resolves publication and deletion races. Proves acknowledged data survives zone failure. Evaluates fairness, trust, and storage growth without assuming a fixed retention window.