Soak Testing

Some defects don't appear in an hour-long test. They accumulate — byte by byte, handle by handle — until the system falls over at 3am on day four. Soak testing finds them first.

What it is

A soak (endurance) test runs a realistic, moderate load continuously for an extended period — typically 8 to 72 hours — while monitoring for slow degradation. The load level is usually 60–80% of peak: high enough to exercise the system meaningfully, low enough that observed degradation is attributable to time, not saturation.

What it catches

Memory leaks and heap fragmentation that force progressively longer GC pauses; connection, file-handle and socket leaks that exhaust OS or pool limits; unbounded growth in logs, temp files, queues and database tables; cache eviction pathologies; scheduled-job interference (backups, batch windows, certificate rotations); and counter overflows or token expiries that only occur after N hours of uptime.

How we detect degradation

The key analysis technique is trend detection on steady-state load. Because the applied load is constant, any upward drift in latency, heap-after-GC, open handles or queue depth is a defect signal. We fit trends across the full run and flag any metric whose steady-state baseline moves.

Healthy soak  : heap-after-GC flat at ~2.1 GB across 48 h
Leaking soak  : heap-after-GC 2.1 → 2.4 → 2.9 → 3.6 GB
                full GC frequency doubling every ~9 h
                → OOM projected at hour 61

Practicalities

Soak tests need stable, dedicated environments and disciplined monitoring — a 48-hour run invalidated at hour 40 by an unrelated deployment is an expensive mistake. We handle scheduling, environment freeze coordination, automated health checks and overnight alerting so the run completes cleanly.