How it works
Keep the stable version running while deploying the new version to a separate pool. A load balancer, gateway, or service mesh can send a small traffic share to that pool; a fleet controller can instead select a small group of machines or database instances.
Choose the exposure unit deliberately. Weighted request routing is simple for stateless APIs, while assignment by user, tenant, or resource keeps related operations on one version. A small but representative cohort is more useful than a convenient cohort that never exercises risky paths.
Compare the canary with the stable version using version-labeled metrics. Watch error rates, latency, resource use, and application-specific correctness signals, such as failed reservations or invalid output. Overall service metrics can hide a serious regression affecting only a small canary.
Promote through explicit stages, such as 1%, 5%, 25%, and 100%, with observation windows and enough samples at each stage. These percentages are examples, not defaults: low-volume services may need longer observation, while scheduled workloads may need a full execution cycle.
Pause or roll back when predefined checks fail, and pause when telemetry is missing. For an API, rollback usually means routing new requests to the stable pool and draining existing work; for a fleet upgrade, it means stopping new targets and following a tested recovery procedure for affected targets.
Keep shared state compatible throughout the rollout. Old and new versions may read the same database, cache, or messages, so use backward-compatible changes and delay destructive cleanup. Routing traffic back does not undo writes, notifications, or external side effects.
A worked example
Canarying an API release at 10,000 requests per second
Suppose an API handles 10,000 requests per second. Route 1%, about 100 requests per second, to a new version while keeping enough stable capacity to take traffic back. Over ten minutes, the canary receives roughly 60,000 requests. For this example, pause promotion if its error rate exceeds the stable version by 0.5 percentage points, or its p95 latency is over 20% worse for a sustained window. Also check successful business operations, not just HTTP responses. If checks pass, expand to 5% and then 25%, observing each stage. If they fail, route new requests back and drain the canary. Keep database writes backward compatible so the stable version can continue serving affected users.
Trade-offs to say out loud
- Smaller initial exposure, not complete isolation
- A canary limits which requests or resources run new code, but a shared database change or dependency overload can still affect everyone.
- Detection takes traffic and time
- Small cohorts reduce exposure but may not produce enough samples to reveal rare failures, slow leaks, or periodic workload problems.
- Mixed-version complexity
- Both versions must tolerate shared schemas, message formats, cache entries, and protocol interactions during rollout and rollback.
- Additional capacity and routing
- Separate pools, spare stable capacity, traffic controls, and automated analysis add infrastructure cost and operational work.
- Cohort selection can mislead
- A canary containing only small tenants or one region may pass while missing failures triggered by large tenants or different workloads.
- Rollback has limits
- Stopping new traffic is often quick, but reversing persistent writes, external effects, or an incompatible storage upgrade may require repair rather than a simple version switch.
When to reach for it
- The interview asks how to release a continuously serving system without exposing all users to an unproven version at once.
- A control plane upgrades many databases, agents, or other resources, and one faulty release must not spread across the fleet.
- Production workload diversity makes staging tests insufficient, and you can measure regressions on a limited cohort.
- A new serving implementation or published data version can coexist with the stable version and be selected by request, tenant, or resource.
How to bring it up in the interview
- Start with the failure you want to contain and name the rollout unit: requests, users, tenants, instances, or regions. Interviewers may ask why that unit provides meaningful isolation.
- Describe the routing mechanism and cohort selection, not just a percentage. Explain whether related requests need stable assignment and how long-lived connections or background jobs move between versions.
- Name promotion and abort checks, their observation windows, and what happens when telemetry is unavailable. Expect questions about low sample counts, noisy baselines, and correctness failures that do not produce HTTP errors.
- Walk through rollback end to end: stop promotion, restore routing, drain in-flight work, and verify recovery. Expect the interviewer to ask what happens to data already written by the new version.
- For fleet upgrades, combine a representative canary with bounded concurrency and staged expansion across failure domains. A successful first batch does not justify upgrading every remaining resource simultaneously.
Common mistakes
- Assuming that deploying new code to 1% of instances guarantees exactly 1% of traffic or limits all damage to 1% of users.
- Watching only aggregate health metrics, which can conceal failures in a small canary cohort.
- Promoting after a fixed timer without enough traffic, representative operations, or a complete relevant workload cycle.
- Calling rollback safe while the new version makes destructive schema changes or writes data the old version cannot read.
- Treating a canary as an A/B experiment: release safety checks do not automatically provide valid evidence of product impact.
Where it comes up
Interview questions where canary deployment is part of a strong answer, and why.
- Design a control plane for a distributed databaseSafe database upgrades require representative canaries, bounded rollout concurrency, and recovery rules that account for persistent storage compatibility.
- Design a system to run commands across a fleetFleet-wide safety depends on trying risky commands on a small cohort, evaluating outcomes, and stopping expansion before a faulty command reaches every target.
- Design a weather appPublishing forecast releases safely can use a canary cohort to validate a new data version before making it the default for all readers.
Questions people ask
What is the difference between canary and blue-green deployment?
Canary deployment emphasizes gradual exposure and evaluation before expansion. Blue-green deployment maintains two environments and typically switches traffic from one to the other. You can combine them by shifting traffic gradually between the two environments.
How much traffic should a canary deployment receive?
There is no universally safe percentage. Choose a starting share based on acceptable exposure, traffic volume, workload diversity, and how quickly you can detect a regression. Require enough observations to evaluate the relevant failure modes before promoting.
Does Kubernetes automatically provide canary deployments?
A basic Kubernetes rolling update gradually replaces pods, but it does not by itself provide precise traffic weighting or metric-based promotion and rollback. A canary rollout usually needs separate stable and canary workloads plus routing and analysis controls. Pod counts alone do not guarantee a particular traffic split.