How it works
Start with a falsifiable hypothesis and a measurable healthy baseline. For example: “If one cache node becomes unreachable, requests finish within our latency target, and database traffic stays below its safe capacity.” Measure user-visible behavior, not just whether replacement processes start.
Choose a realistic fault that challenges an architectural assumption. Examples include dropping packets between replicas, delaying a dependency, terminating a worker after a side effect but before acknowledgment, or exhausting disk space on one node. Slow or partially reachable services often reveal problems that clean shutdowns miss.
Define the experiment boundary before injecting the fault: affected instances or tenants, duration, traffic limits, an owner, and automatic stop conditions. Start in an isolated environment with representative traffic, then consider a tightly scoped production experiment if the remaining uncertainty justifies the risk.
Inject the fault and compare behavior against the hypothesis. Track latency, errors, retry volume, queue age, dependency load, and relevant correctness invariants. For a replicated store, successful requests alone are insufficient: also check whether acknowledged writes survive and whether competing leaders can accept conflicting writes.
Stop the injection, restore normal operation, and observe recovery. Verify that backlogs drain, replicas converge, and traffic does not trigger another overload cycle. Fix weaknesses and repeat the experiment; a successful run provides evidence for the tested conditions, not proof against every failure.
A worked example
Testing cache failover without overwhelming the database
Suppose a service handles 20,000 reads per second with a 95% cache hit rate, so the database normally receives about 1,000 reads per second. Its tested safe capacity is 4,000. The hypothesis is that losing one cache node keeps API p99 latency below 200 milliseconds and database reads below 3,000 per second. First test under representative staging traffic, then isolate one production cache node serving a small traffic slice for 60 seconds. Monitor cache misses, database reads, errors, and request coalescing. Automatically stop if database reads exceed 3,000 per second or latency breaches the limit for 10 seconds. After restoring connectivity, verify that cache warming does not cause another surge. A failure points to missing capacity, request coalescing, or load shedding, not merely a need for faster node replacement.
Trade-offs to say out loud
- Realism versus customer risk
- Production experiments exercise real traffic and configuration, but require tighter scope and safeguards because customers can be affected.
- Small scope versus correlated failures
- Testing one instance limits damage but does not establish that the system can survive an entire zone failure or a shared dependency outage.
- Instrumentation and engineering cost
- Useful experiments require trustworthy measurements, fault controls, correctness checks, and time to investigate and fix findings.
- Recovery is not automatic rollback
- Removing a fault does not undo duplicate external actions, lost data, or an accumulated backlog, so recovery procedures must be designed separately.
- Evidence has a limited scope
- Passing an experiment validates a particular workload, configuration, and fault model rather than proving general reliability.
When to reach for it
- The design depends on failover, leader election, or fencing, and you need to explain how those mechanisms will be validated.
- Retries, timeouts, and fallback paths could amplify load during a dependency failure.
- Workers can crash between performing a side effect and recording completion, creating uncertain or duplicate outcomes.
- The interviewer asks how you would verify availability targets, recovery behavior, or containment of a regional or control-plane failure.
How to bring it up in the interview
- Introduce it after explaining the resilience mechanism: “We expect replica promotion to preserve acknowledged writes; I would validate that by isolating the leader during writes.” Chaos engineering validates the design rather than replacing it.
- State one concrete hypothesis, the injected fault, the measured outcome, and the stop condition. Interviewers may ask whether the test checks user success, data correctness, or merely process health.
- Explain why the fault is representative. Be ready to distinguish process crashes from network partitions, slow responses, and failures where a write succeeds but its response disappears.
- Describe both containment and recovery. Expect follow-ups about spare capacity, retry storms, backlog draining, and whether the experiment controller still works when the target system is unhealthy.
- Keep the discussion proportional: a focused experiment tied to a major failure assumption is stronger than listing chaos tools or proposing random production outages.
Common mistakes
- Calling random instance termination chaos engineering without defining a hypothesis, measurements, or stopping rules.
- Testing only clean crashes and assuming the result also covers network partitions or slow dependencies.
- Watching infrastructure dashboards while ignoring lost writes, duplicate side effects, or incorrect user-visible results.
- Running a broad production experiment before validating containment and recovery at a smaller scope.
- Treating a successful experiment as proof that the system meets its availability target under every failure.
Where it comes up
Interview questions where chaos engineering is part of a strong answer, and why.
- Design a distributed cacheControlled node loss and network isolation test ownership changes, replica failover, and whether cache misses overload the backing database.
- Design a distributed message log like KafkaFaults during append acknowledgments and leader changes test durability guarantees, while consumer crashes exercise replay and recovery behavior.
- Design a control plane for a distributed databasePartitioning controllers and interrupting workflows test fencing, safe reconciliation, and containment of fleet-wide failures.
Questions people ask
What is chaos engineering?
Chaos engineering is the practice of running controlled experiments that introduce failures and test explicit expectations about system behavior. The goal is to find weaknesses in availability, correctness, containment, and recovery before an uncontrolled incident exposes them.
How is chaos engineering different from fault injection?
Fault injection is the mechanism used to introduce a problem, such as packet loss or a terminated process. Chaos engineering wraps that mechanism in a hypothesis, measurements, safety boundaries, and a process for learning from the result.
Does chaos engineering have to run in production?
No. Isolated tests and staging experiments are useful starting points, especially for checking correctness and validating safeguards. Production experiments can expose differences in traffic, configuration, and dependencies, but should only be used with justified risk, limited scope, and tested stop controls.