Load Testing in Practice: Planning, Running and Learning from Performance Tests

Performance problems are among the most visible failures a product can have: a checkout that times out on launch day, an API that slows to a crawl at month-end, a system that degrades after running for a week. Load testing finds these problems before customers do.

Types of performance tests

Planning a load test

1. Define objectives: which journeys, which peak, which targets.
2. Model the workload: use analytics for journey mix, arrival rates, think times and session lengths.
3. Set targets: for example, 95th percentile response time under a set threshold for key pages, error rate below an agreed limit, at a defined requests-per-second rate.
4. Prepare data: realistic volumes and variety; enough unique users and items to avoid unrealistic caching.
5. Choose the environment: as close to production as possible in size and configuration.
6. Instrument everything: application, database, cache, queues, infrastructure and third-party calls.
7. Plan third parties: use sandboxes or stubs, and agree with providers before testing against them.

Metrics to watch

Running and analysing

• Ramp up gradually to see where performance starts to degrade.
• Hold at peak long enough to reach steady state.
• Correlate response-time increases with resource saturation to locate bottlenecks.
• Change one thing at a time when tuning, and rerun.
• Document results with configuration, data and versions so tests can be compared later.

Common bottlenecks

• Missing or inefficient database indexes and N+1 queries.
• Undersized connection pools or thread pools.
• Synchronous calls to slow external services.
• Caches that are cold, too small or invalidated too often.
• Auto-scaling that reacts too slowly to spikes.
• Locks and contention on shared resources.

Tools

Popular open-source tools include k6, JMeter, Gatling and Locust; cloud load-testing services can generate large distributed loads. Choose tools your team can script and maintain, and that integrate with your CI and monitoring.

Making it continuous

• Pipeline checks: short performance tests on key endpoints for each release, with thresholds that fail the build on regressions.
• Scheduled full tests: before major releases, seasonal peaks and infrastructure changes.
• Production monitoring: real-user metrics and alerts to validate assumptions.

Our guide to designing a CI/CD pipeline explains where performance gates fit.

Worked example: preparing for a seasonal peak

A hypothetical online retailer expects traffic four times its normal peak during a holiday sale.

1. Workload model: analytics show 60% browsing, 25% search, 10% basket and 5% checkout, with checkout as the critical journey.
2. Targets: 95th percentile page response under two seconds and checkout error rate below 0.5% at four times normal peak.
3. First run: response times rise sharply at three times peak; the database CPU saturates due to an unindexed search filter and inventory checks on every page view.
4. Fixes: an index, caching of inventory counts for browse pages and a larger connection pool.
5. Second run: targets met at four times peak; a stress test shows degradation begins at six times peak.
6. Spike and soak tests: auto-scaling responds within acceptable time; no memory growth over eight hours.

The team also prepares a runbook for the sale day with dashboards, alert thresholds and actions such as disabling non-essential features under extreme load.

Frequently asked questions

Can we load test in production?

Sometimes, carefully and with safeguards, for example during low-traffic periods with gradual ramp-up and kill switches. Most testing should happen in production-like environments.

How much load should we test?

At least the expected peak plus a safety margin, and a stress test to find the breaking point.

How often should we load test?

Automated small checks on every release; full tests before major launches, peaks or architecture changes.

Who should own performance?

Product teams own performance of their services, with QA and platform teams providing tools and expertise.