Identify and Fix Memory Leaks in Load Testing

To find a memory leak with a load test, hold a steady, realistic load for several hours and watch the memory the application keeps after each garbage collection. Healthy memory rises, then levels off. A leak keeps climbing at the same load and does not come back down when the load stops. Then take heap dumps at two points, compare them to find the objects that keep growing and the reference that holds them, fix it, and re-run the same test to prove the slope is gone.

Key Takeaways

Start with a baseline

Before running any load test, record memory under known conditions: idle after startup, at normal load, and after the load stops. Without that reference you cannot tell abnormal growth from the normal warm-up of caches and pools.

Combine metrics, heap dumps and profilers

Metrics tell you that memory is growing. Heap dumps tell you what is growing and what holds on to it. Use both: a trend line to confirm the leak, and two dumps taken some time apart to find it.

Automate detection in CI pipelines

A short, repeatable load test with a memory check on every release catches the leaks that grow fast. The slow ones need a scheduled multi-hour soak test.

Separate leaks from legitimate growth

Caches, connection pools and JIT-compiled code all make memory grow under load, then stop. The signature of a leak is growth that continues at a constant load, measured on the memory that survives garbage collection.

Why memory leaks matter in load testing

A memory leak is memory the application keeps allocated after it no longer needs it. In garbage-collected languages such as Java, C#, JavaScript, Python and Go, that almost always means something still holds a reference: a static map that only grows, a listener that is never removed, a cache with no eviction. The collector cannot free what is still reachable.

Small leaks are invisible in short tests. A leak of a few hundred bytes per request is nothing in a five-minute smoke test and a lot after millions of requests. Under sustained load it shows up as longer and more frequent GC pauses, rising response times, and finally an OutOfMemoryError, a crash or a container killed by the OOM killer. In autoscaled systems it can also hide as cost: instances get replaced or added before anyone notices that each one degrades over time.

This is why leak hunting belongs in load testing. Load testing is the one place where you can reproduce production-like traffic for hours, at a known and constant rate, and measure what memory does.

Step 1: Establish a baseline for memory usage

A baseline shows what normal memory consumption looks like at known points, so you can tell expected growth (caches filling, pools opening, code being compiled) from a leak.

Run the application in an isolated environment that matches production: the same runtime version, memory limits and GC settings, with no unrelated jobs on the host. Then record memory at three points:

• Idle: just after startup, before any traffic.
• Nominal load: typical traffic, held long enough for caches and pools to warm up.
• Post-load: a few minutes after the load stops.

Which memory metrics to use for the baseline

Choose metrics that show what the application keeps, not only what it allocates:

• Heap after GC (live set). The heap right after a collection, for example old generation occupancy after a full or mixed GC on the JVM. Raw heap usage is a sawtooth that rises and drops with every collection; the low points of the sawtooth are the number that matters.
• Non-heap and native memory. On the JVM: Metaspace (which replaced PermGen in Java 8), code cache, thread stacks and direct buffers. Class loader leaks and native libraries show up here, not in the heap.
• GC activity. Collection frequency and the share of time spent in GC. When the live set grows, the collector runs more often and frees less each time.
• Process RSS or container working set. What the operating system sees. Useful for the memory limit, but noisy: many allocators keep freed memory instead of returning it to the OS, so RSS that stays high after the load is not proof of a leak on its own.
• Other resources that leak the same way. Open file descriptors, thread or goroutine count, open connections and disk usage. A leak is any count that grows with traffic and never comes back.

Our older guide on how to measure CPU and memory while running a load test shows a simple script-based way to log these on a Linux server.

Step 2: Run a soak test to stress the application

Short, bursty load tests rarely expose leaks. You need an endurance test, also called a soak test: ramp up to a realistic load, then hold it constant for hours. A constant load is the point. If the request rate does not change and memory keeps rising, the traffic is not the explanation.

Designing load scenarios for leak detection

• Concurrency and rate. Use the load you expect at a normal busy hour, not the breaking point. Leaks are easier to read when the system is not saturated, because saturation adds its own queueing and memory pressure.
• Session patterns. Script real journeys: log in, browse, search, add to cart, log out. Leaks are often tied to one code path (a session object, an upload handler), so a test that hits only the home page can miss them.
• Duration. Long enough for the slope to rise clearly above noise. That usually means several hours; a leak of a few megabytes an hour needs more time than a fast one. Plan to repeat the run after the fix with the same profile.
• Stable environment. Keep the instance size, container limits, GC flags and data set the same between runs, and note them with each result. A different heap size changes the GC pattern and makes runs hard to compare.

Watch the load generator as well. A JMeter or k6 machine that runs out of memory produces errors that look like a server problem; give the generator enough room (see how to increase heap size in JMeter) or run it from the cloud.

Step 3: Monitor memory metrics during the test

Collect memory metrics from the system under test while the load runs, from the same timeline as the load test results, so you can line them up.

Selecting monitoring tools

Use the tools your runtime already has, plus your existing monitoring:

• JVM: GC logs (-Xlog:gc* on JDK 9 and later), jstat -gcutil <pid> 10s, VisualVM, JDK Flight Recorder, or the JVM metrics your APM agent exports.
• .NET: dotnet-counters for live GC and heap counters; Visual Studio or the commercial .NET Memory Profiler for deeper analysis.
• Node.js: process.memoryUsage(), the inspector and Chrome DevTools memory panel.
• Python: the standard library’s tracemalloc module.
• Go: runtime/metrics or the net/http/pprof heap profile, plus the goroutine count.
• Containers and Kubernetes: the container working set from cAdvisor or metrics-server, and OOMKilled restarts. (Heapster, still named in older guides, was retired in 2018.)

LeakCanary, often listed in such guides, is a leak detection library for Android apps. It is the right tool for a mobile client, not for a server under load.

Key memory metrics and leak indicators

How to detect a slow memory or disk leak from metrics

Slow leaks hide in noise, so judge the trend, not single points:

1. Use the floor, not the peaks. Take the minimum memory per window (for example, per 10 minutes) or the heap after each full GC. That removes the sawtooth.
2. Hold the load constant. Compare memory at equal load. Growth during the ramp-up is expected; growth on a flat load line is the signal.
3. Fit a slope. Prometheus offers deriv() and predict_linear() for gauges: a positive deriv() on the floor series that holds for hours is a leak candidate, and predict_linear() tells you when it reaches the limit. The same method finds disk leaks: run predict_linear() on free disk space, or watch log, temp or upload directories grow while traffic stays flat.
4. Check that it does not recover. Stop the load, wait for a full GC, and measure again. A cache stays roughly where it was; a per-request leak means the next run starts higher. Repeat the load-and-rest cycle twice and compare the starting points.

A worked example, with illustrative numbers: at a constant 50 requests per second, the heap after full GC rises from 410 MB to 530 MB over a 4-hour soak test. That is 30 MB per hour. With a 2,048 MB maximum heap, the remaining 1,518 MB lasts about 50 hours at that rate, so the service would fail roughly two days after a restart at the same traffic. The test served 50 x 14,400 = 720,000 requests, so the leak is about 175 bytes per request (120 MB / 720,000). That size points to one small object or map entry per request, not a large buffer, which narrows the search before you open a heap dump.

To line up memory with the load, send the load test metrics into the monitoring tool that already collects your server memory. LoadFocus can push load test metrics to Datadog, New Relic, Dynatrace, Azure Application Insights or any OpenTelemetry backend, with every run tagged by test name and run ID (see LoadFocus integrations), so the virtual users and response times sit on the same chart as the heap.

Step 4: Capture and compare heap dumps

A heap dump is a snapshot of every object in memory and the references between them. One dump shows what is big. Two dumps taken at the same load some time apart show what is growing, which is what you need for a leak.

How to capture and analyze heap dumps

• JVM: jcmd <pid> GC.heap_dump /tmp/heap.hprof writes an HPROF dump of live objects (it forces a full GC first unless you pass -all); jcmd <pid> GC.class_histogram gives a quick count per class. Add -XX:+HeapDumpOnOutOfMemoryError so a crash during the test leaves a dump behind. Open dumps in Eclipse Memory Analyzer (MAT) or VisualVM.
• .NET: dotnet-gcdump collect -p <pid>, then open the .gcdump files in Visual Studio or PerfView to compare them. Microsoft warns that it triggers a full generation 2 collection, which can pause a process with a large heap.
• Node.js: start with --heapsnapshot-signal=SIGUSR2 and send the signal to write a snapshot, or use --heapsnapshot-near-heap-limit to capture one before a crash; compare snapshots in Chrome DevTools.
• Python: tracemalloc.take_snapshot() at two points and snapshot.compare_to() lists the lines that allocated the growth.

Taking a dump pauses the process and can take seconds or longer on a big heap. Take dumps on one instance while the others carry the load, and do not read the response times from that window.

A practical cadence:

• Baseline dump after warm-up, at nominal load.
• Later dump at the same load, after memory has grown noticeably (for example, an hour later).
• Post-test dump after the load stops and a full GC, to see what was never released.

In the comparison, look for:

• Types whose instance count grows between dumps, especially your own classes and collections.
• The path to GC roots of those instances: which static field, singleton, thread or cache holds them.
• Collections that only grow: maps and lists with no removal or eviction.

Before/after example of a heap dump finding

The strong example names the object, the reference that holds it, the size of the growth and the result after the fix. That is the level of detail a ticket needs. (The example is illustrative.)

Common leak patterns in heap dumps

• Static collections: maps or lists held in static fields live as long as the application, so anything added and never removed stays forever.
• Unremoved listeners and callbacks: an object registered as a listener stays reachable from the thing it listens to.
• Unclosed resources: connections, streams and sockets hold buffers and sometimes threads until closed.
• Caches without limits: a cache with no maximum size or expiry is a leak with a better name.
• ThreadLocal values in thread pools: pooled threads never die, so values set and never cleared stay attached to them.
• Class loader leaks: repeated redeploys in a long-running container that keep old classes alive, visible in Metaspace rather than the heap.

Step 5: Automate memory leak detection in CI/CD pipelines

Manual heap analysis does not scale, but a simple automated check catches the fast leaks before they ship.

Setting up automated leak checks

1. Record a baseline from a known-good release: live heap after GC, GC time and RSS at a fixed load and duration.
2. Run the same short load test in the pipeline against each release candidate, with the same data set, heap size and duration.
3. Compare against thresholds. For example: live heap at the end of the test no more than a set percentage above the baseline, or no upward slope during the steady phase. Pick the numbers from your own baseline runs so normal variation passes.
4. Fail the build or raise an alert when a threshold is breached, and keep the memory chart with the result so the next person can see the trend.
5. Review the thresholds after deliberate changes such as a bigger cache or a new framework version.

A 15 to 30 minute CI run catches leaks that grow quickly. Schedule the multi-hour soak test (nightly or weekly) for the slow ones. Our guide to integrating performance testing with DevOps toolchains covers the pipeline side in more detail.

With LoadFocus, the load test half of this can be automated: LoadFocus load testing runs saved tests on an hourly, daily, weekly or monthly schedule, gives each run a PASS or FAIL verdict against your P95, P99, error-rate and throughput thresholds, and gates deploys through a GitHub Action, a verdict API or the Jenkins plugin. Memory itself is measured on your servers, so the memory threshold lives in your monitoring tool; the load test provides the steady, repeatable traffic. The maximum test duration depends on the plan (loadfocus.com/pricing lists up to 5 hours per test on the larger plans, October 2026); a longer soak can be split into scheduled runs.

Step 6: Fix the leak and prevent new ones

The heap dump comparison gives you the reference chain. The fix is to break it: remove the entry, unregister the listener, close the resource, or bound the cache.

Leak prevention practices

• Release resources deterministically. Use try-with-resources in Java, using in C#, context managers in Python and defer in Go for connections, streams and files.
• Give every cache a limit. A maximum size, an expiry time or both. If you cannot say what removes an entry, it is not a cache.
• Unregister what you register. Listeners, callbacks, timers and subscriptions need a matching cleanup in the component’s dispose or shutdown path.
• Avoid static state for per-request data. Per-request objects should die with the request.
• Clear ThreadLocals in a finally block when threads come from a pool.
• Use weak references with care. They help for canonical maps and listener lists, but they are not a substitute for a bounded cache.

Memory leak prevention code review checklist

After the fix, re-run the exact soak test from Step 2. The proof is the same load, the same duration and a flat floor where there was a slope.

Step 7: Monitor for memory leaks in production

Some leaks only appear with production traffic: rare code paths, long-lived sessions, unusual inputs. Keep the same leak signals in production monitoring.

Setting up memory usage alerts

Use two kinds of alert. A threshold alert fires when live memory or the container working set stays above a level (for example, 85% of the limit) for a sustained period; it catches fast leaks and runaway processes. A trend alert catches slow leaks: alert when predict_linear() over several hours says the limit will be reached within a day, or when the post-GC floor has risen for several days in a row.

Two more signals help. Frequent OOMKilled restarts or “restarted to free memory” runbooks are leak symptoms even when no alert fires. And response time that degrades with uptime, then resets after a deploy, is a classic leak pattern worth checking against memory.

When memory growth is not a leak

Growth is normal when it stops. Common legitimate causes:

• Caches and buffers warming up. Memory rises quickly while the cache fills, then flattens at the configured size.
• Connection and thread pools growing to their maximum under load.
• JIT compilation and class loading in the first minutes of traffic.
• The runtime keeping freed memory. The JVM may not shrink the heap after load, and allocators often keep freed pages, so RSS stays high even though the live set dropped.

To tell the two apart, look at the live set after GC at constant load and at the post-load measurement. Flat after warm-up means healthy. Still rising after hours at the same load means a leak. If the picture is ambiguous, extend the test or run the load-and-rest cycle twice: a cache returns to the same level each cycle, a leak starts higher each time.

Why GC tuning does not fix a leak

A bigger heap or different GC settings change when the problem appears, not whether it appears. If the application keeps references it no longer needs, no collector can free that memory; a larger heap only moves the crash further out, and can make each full GC longer. Use GC tuning for pause times and throughput once the live set is flat, and fix leaks in the code.

Summary checklist

Frequently Asked Questions

What is a memory leak in performance testing?

A memory leak is memory an application keeps after it no longer needs it, usually because something still references it. In performance testing it shows up as memory that keeps rising at a constant load, followed by longer GC pauses, slower responses and finally out-of-memory errors or restarts. Load tests matter here because a small per-request leak only becomes visible after many requests.

How do I detect a slow memory leak from metrics?

Use the memory left after garbage collection (or the minimum per time window) rather than raw usage, hold the load constant, and look at the slope over several hours. A positive slope at steady load that does not recover after the load stops is a leak. Tools such as Prometheus deriv() and predict_linear() turn that slope into an alert and an estimate of when the limit will be reached.

How can I distinguish a memory leak from normal memory growth under load?

Normal growth stops: caches fill, pools reach their size, and the live set flattens. A leak keeps growing at the same load and does not return after a full GC once the load stops. Run the load-and-rest cycle twice; if each cycle starts from a higher floor, you have a leak.

What tools should I use to detect memory leaks during load testing?

A load testing tool to produce steady, repeatable traffic, your monitoring stack for the memory metrics, and the runtime’s heap tools for the analysis: jcmd, GC logs, Eclipse MAT or VisualVM for Java; dotnet-counters and dotnet-gcdump for .NET; heap snapshots and Chrome DevTools for Node.js; tracemalloc for Python; pprof for Go.

How long should a load test run to find memory leaks?

Long enough for the growth to rise clearly above noise at a constant load, which usually means several hours. Fast leaks show up in a 15 to 30 minute test; slow ones need a multi-hour soak. Keep the duration and profile the same before and after a fix so the two runs can be compared.

Can automated tests catch all memory leaks?

No. Automated checks catch leaks that grow within the test window and on the code paths the test exercises. Leaks in rare paths or that need days of uptime can still reach production, so keep threshold and trend alerts on memory in production as well.

Is garbage collection tuning a solution for memory leaks?

No. GC tuning changes when a leak causes trouble, not whether it does. If references are still held, the collector cannot free the memory. Find the reference with a heap dump comparison and fix it in the code.

Sources

• jcmd GC.heap_dump and GC.class_histogram: Oracle jcmd reference (checked October 2026)
• Removal of PermGen in Java 8: JEP 122
• dotnet-gcdump, including the full gen 2 GC warning: Microsoft Learn
• Node.js --heapsnapshot-signal and --heapsnapshot-near-heap-limit: Node.js CLI documentation
• Python tracemalloc: Python documentation
• Prometheus deriv() and predict_linear(): Prometheus query functions
• Heapster retirement: github.com/kubernetes-retired/heapster (archived December 2018)
• LeakCanary is a leak detection library for Android: square.github.io/leakcanary
• LoadFocus integrations, scheduling, thresholds and test durations: loadfocus.com/integrations, loadfocus.com/load-testing, loadfocus.com/pricing (checked October 2026)

Related reading

• How to Quickly Establish a Performance Baseline
• Optimize Database Connection Pools for Peak Performance