How the interviewer opens it
“Design an A/B testing platform like Optimizely. Product teams should be able to run experiments, give users consistent experiences, and tell whether a change helped.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- scale
- read:write mix and event size
- identity and assignment
- experiment changes
- latency and freshness
- availability versus consistency
- exposures and outcomes
- statistical policy
- durability and late events
- product boundary
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Define and launch experiments with targeting rules, traffic allocation, and variants
- Assign users consistently and evaluate experiments in product applications
- Record exposures and outcomes
- Report per-variant conversion rates, uplift, uncertainty, and experiment health
Usually out of scope
- Visual webpage editing
- Anonymous-to-account identity stitching
- Arbitrary user-supplied code or SQL
- Bandits, automatic winner selection, and a general-purpose sequential testing engine
What interviewers probe
- Stable assignment across SDKs and configuration changes
- Assignment must agree across devices and languages without a central lookup, while configuration propagates asynchronously. A harmless configuration edit must not silently move users between treatments.
- Exposure attribution under retries, late events, and replay
- Assignment is not exposure, repeated events are not independent users, and ingestion order is not event order. A fast dashboard can still be wrong if its denominator or conversion window changes under retries and delays.
- Producing a valid answer to 'did it help?'
- Correct infrastructure can produce misleading conclusions through the wrong analysis unit, immature conversion windows, biased exposure logging, or repeated significance checks. Measurement health is part of the product.
Common mistakes
- Putting a central assignment service and per-user database read on every product request without showing why local deterministic evaluation is insufficient.
- Rehashing users after configuration edits or changing live variant weights without addressing crossover.
- Confusing assignment, exposure, and conversion, especially when choosing the denominator.
- Assuming durable ingestion prevents duplicate counting or guarantees delivery from every client.
- Keeping only aggregate counters, making late attribution, investigation, and replay impossible.
- Treating statistical significance as a dashboard formatting problem while ignoring randomization units, sample-ratio mismatch, immature outcomes, and peeking.
The bar by level
- Senior (L5/E5)
- Defines stable cross-SDK assignment and versioned run semantics, keeps serving independent of analytics, and handles retries, event-time attribution, and replay. Explains user-level analysis, sample-ratio mismatch, outcome maturity, confidence intervals, and fixed-horizon reporting.
- Staff (L6+/E6+)
- Establishes end-to-end invariants across SDK behavior, immutable run definitions, exposure semantics, and statistical policy. Identifies where data loss or treatment-dependent logging invalidates causality, makes stale-config and kill-switch trade-offs explicit, and adapts the design to ramps or interacting experiments without silently changing the estimand. Chooses operationally simple components at this scale and defines how results are corrected and audited.