How the interviewer opens it
“You own an existing ML prediction service whose latency and error rate get much worse during traffic spikes. How would you identify its problems and redesign it without disrupting callers?”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- Starting architecture — interviewer-defined fixture
- Prediction contract
- Traffic and read:write mix
- Models and inference cost
- Feature data
- Observed symptoms
- Latency and availability
- Freshness and consistency
- Model lifecycle and durability
- Boundaries and freedom to change
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Investigate the existing request path and identify bottlenecks using evidence
- Serve synchronous predictions using existing models and precomputed features
- Meet latency, availability, and freshness targets during spikes and a single-zone failure
- Migrate infrastructure, models, and compatible feature representations with a safe rollback path
Usually out of scope
- Model training, model selection, and improving prediction quality
- Computing features from raw events
- Asynchronous batch prediction
- Multi-region serving
- GPU kernel optimization
What interviewers probe
- Diagnosing capacity versus routing problems
- Uneven routing, expensive feature reads, and insufficient total CPU can produce the same latency symptom. Better balancing cannot make 64 cores deliver 200 cores of inference work.
- Keeping overload out of the synchronous latency budget
- An unbounded queue accepts work that callers will abandon before execution. Retrying that work can turn a capacity shortage into a cascading failure.
- Fresh feature reads through cache and zone failures
- Replicas and caches can each appear only slightly stale while their combined delay violates the five-second limit. A fallback to the old SQL primary is useless when that primary's entire zone has failed.
- Model rollout, feature-schema transition, and rollback
- Two model versions may need different feature representations at the same time. Retaining an old model artifact is not enough if its features have been overwritten or stopped receiving updates.
Common mistakes
- Treating the undisclosed reported architecture as known instead of using the explicit interview fixture.
- Starting with infrastructure names rather than measuring the existing request path.
- Using queues to hide sustained capacity shortages in a synchronous service.
- Sharding by model ID despite a small model footprint and highly skewed traffic.
- Counting serving-host redundancy as zone tolerance while feature reads still require a single-zone primary.
- Promising fast rollback while discarding or freezing the previous model's feature representation.
The bar by level
- Senior (L5/E5)
- Drives diagnosis and quantifies peak and zone-loss capacity. Handles routing skew, native thread oversubscription, deadlines, and retry amplification. Gives an explicit cross-zone feature-store failover path with freshness checks, plus a staged rollout that retains live compatible features for the previous model and meets the rollback target.
- Staff (L6+/E6+)
- Defines testable contracts across serving, storage, and upstream feature publication. Connects replication progress, cache expiry, fencing, failover capacity, and availability accounting without claiming impossible uninterrupted service. Designs schema ownership, dual-publication gates, rollback retention, and safe retirement across concurrent models; separates infrastructure migration from model changes and gives concrete failure drills and cost trade-offs.