Chaos Engineering: Testing Resilience Safely Before Real Failures Do It for You

Distributed systems fail in surprising ways: a slow dependency exhausts connection pools, a region outage reveals a hidden single point of failure, a retry storm turns a small problem into a large one. Chaos engineering exposes these weaknesses deliberately, under controlled conditions, so teams can fix them before customers are affected.

Principles

1. Define steady state: measurable indicators of normal behaviour, such as successful checkout rate and latency.
2. Form a hypothesis: “If one instance of the payment service fails, checkout success remains above 99.5%.”
3. Introduce realistic failures: instance loss, latency, errors from dependencies, resource exhaustion.
4. Minimise blast radius: limit the scope and duration of experiments.
5. Observe and learn: compare actual behaviour with the hypothesis.
6. Improve and repeat: fix weaknesses and rerun to confirm.

Prerequisites

• Observability to see the impact quickly — see our guide to observability with OpenTelemetry.
• Service level indicators that define acceptable behaviour.
• Automated rollback and the ability to stop experiments instantly.
• Incident processes and on-call readiness.
• Stakeholder agreement on when and where experiments run.

Designing a safe experiment

Common scenarios to test

• Loss of an instance, container or availability zone.
• Increased latency or errors from a dependency.
• Database failover.
• Exhausted connections, threads or disk space.
• Network partitions between services.
• Expired certificates or credentials.
• Sudden traffic spikes — often combined with load testing.

Game days

Game days are scheduled sessions where teams run experiments together, often simulating a realistic incident. They test not only systems but people and processes: alerts, runbooks, communication and decision-making. Debrief afterwards and track actions.

Worked example: a first game day

A hypothetical SaaS company runs its first game day on its staging environment for the order service. The team agrees steady-state metrics — order success rate and p95 latency — and three experiments: terminate one order-service instance, add two seconds of latency to the payment provider stub, and fail over the database.

• Instance loss: traffic shifts to healthy instances within seconds; no errors. Hypothesis confirmed.
• Payment latency: order requests pile up because the payment client has no timeout; thread pools exhaust and errors spike. The experiment is aborted after three minutes, as planned.
• Database failover: recovery takes four minutes instead of the expected one, because connection pools do not reconnect promptly.

Actions: add timeouts and a circuit breaker for the payment client, adjust connection pool settings and update the failover runbook. A month later, the team reruns the experiments in staging, then runs the instance-loss experiment in production for 5% of traffic.

Building a resilience programme

Starting small

1. Choose one critical service with good observability.
2. Run in staging with simple failures such as terminating an instance.
3. Fix findings: timeouts, retries with backoff, circuit breakers, fallbacks.
4. Move to production with tiny blast radius and business-hours scheduling.
5. Automate recurring experiments where confidence is high.

Common findings and fixes

Frequently asked questions

Is chaos engineering only for large tech companies?

No. Any team running important services can benefit from small, controlled experiments, starting in staging.

Will it cause outages?

Poorly controlled experiments can. Limit blast radius, define abort conditions and run with people watching.

How is it different from load testing?

Load testing checks behaviour under traffic; chaos engineering checks behaviour when components fail. They complement each other.

How often should we run experiments?

Start with occasional game days, then automate regular experiments for critical failure modes.