Continuous validation means checking that a system still meets its performance goals for as long as it runs, not only when someone merges a change. My opinion: a performance gate in CI is necessary and still leaves a gap, because several things that slow a system down never arrive as a commit. Validation has to continue after the deploy, with scheduled tests and production checks.
This post is the argument for that second half. The CI half, what to run per pull request and how to fail a build, is in continuous performance testing in CI/CD, so I will not repeat it here.
Key Takeaways
- A CI gate catches regressions that a commit introduces. It cannot catch what changes without a commit.
- Add two things after deploy: scheduled load tests against a stable environment, and production checks tied to your service level objectives.
- Keep both small and owned by a named person. A validation job nobody reads is worse than none.
- It complements pre-release testing. It does not replace it.
What changes without a commit?
A test run on every merge only sees the code at that moment. Meanwhile, a system keeps changing in ways no pull request records:
- Data volume. A query that scans 10,000 rows in staging scans 10 million a year later in production.
- Traffic mix. A marketing campaign or a new client shifts load toward endpoints you never profiled.
- Dependencies. A third-party API, a managed database or a CDN changes its behaviour or limits on its own schedule.
- Infrastructure and configuration. An instance type, an autoscaling rule or a certificate changes outside the application repository.
- Slow leaks. A memory or connection leak shows only after hours or days, longer than any CI job.
None of these makes a CI test fail, because nothing in the repository changed. That is the case for validation that runs on a clock and against production, in addition to the gate.
What does continuous validation add to a CI gate?
Think of three layers, each answering a different question:
| Layer | Question it answers | Runs when |
|---|---|---|
| CI gate | Did this change make the service slower? | Each pull request or merge |
| Scheduled load test | Does the system still hold at expected peak, today? | Nightly or weekly, on a stable environment |
| Production checks | Is the live service meeting its objectives right now? | Continuously |
The scheduled run is the CI result’s counterpart: same script, same thresholds, but triggered by time instead of by a change. If last night’s run is slower than the one before and nothing was merged, you have learned something CI could not tell you, such as data growth or a dependency change. Keep a history of those runs and compare against the median of recent ones, as the CI post suggests for gates.
Production checks cover what no test environment reproduces. Google’s SRE workbook makes the same point about releases: pre-production testing builds confidence but cannot catch everything, because test environments differ from production and some defects only appear under real traffic. Its answer is canarying, defined there as “a partial and time-limited deployment of a change in a service and its evaluation”, judged against indicators of user-facing health rather than noisy signals such as raw CPU. You do not need a full canary system to take the lesson: decide which user-facing signals (error rate, latency percentile, availability of key endpoints) define healthy, and watch exactly those after every deploy. For uptime and response-time checks on an API from outside your network, synthetic monitoring does this job.
How do you start without building a platform?
Start with the smallest version that would have caught your last incident:
- Pick two or three flows that matter commercially, such as login, search and checkout.
- Schedule one load test on them against staging or a production-like environment, nightly or weekly, with pass/fail thresholds on p95 latency and error rate. See scheduled load tests.
- Add API checks in production for the same endpoints, with an alert channel a person actually reads.
- Review the history once a month: did anything drift without a deploy? Tighten thresholds only when the numbers have settled.
- After an incident, add the check that would have caught it. The suite then grows from real failures instead of guesses.
With LoadFocus, load tests can be scheduled daily, weekly or monthly on a paid plan, for cloud, JMeter and k6 tests, and the test owner gets an email when a run fails its pass/fail thresholds or does not complete. API monitoring checks status code, response body, headers and response time, from a choice of 25 regions, with alerts to email, Slack, Microsoft Teams, Discord, PagerDuty, Opsgenie or a webhook (loadfocus.com, October 2026). Any tool that can run a script on a schedule and alert someone works for the same pattern.
Where does continuous validation fall short?
- Cost and upkeep. Scheduled tests use load generators and need a stable target. Scripts break when an API changes, and someone has to own them.
- Noise. A threshold that fires for no reason trains people to ignore it. Start loose, and review each false alarm.
- A shared environment. A nightly test on a staging system other teams use produces numbers you cannot interpret.
- Load in production. Running real load tests against production is a deliberate decision with its own risks: use read-only flows, low volume and an agreed window. Monitoring checks are the safer default there.
- It is not a replacement. A pre-release stress or soak test still answers questions that scheduled runs do not, such as where the breaking point is. See validating auto-scaling with load testing.
What about the counterargument for infrequent, manual testing?
The fair case for it is that a careful test before a big release gives deep insight into edge cases, and a small team may not have the time to maintain scheduled jobs. I agree for a small, stable app with light traffic: a pre-release test and a basic uptime check may be all it needs. My objection starts where the system, its data or its dependencies change between releases, because then the last manual test describes a system that no longer exists.
Frequently Asked Questions
What is continuous validation in performance testing?
It is the practice of checking performance repeatedly over the life of a system, through CI gates, scheduled tests and production monitoring, instead of only at a pre-release milestone.
How is it different from continuous performance testing?
Continuous performance testing usually refers to the CI side: tests triggered by changes. Continuous validation also covers what happens between changes, which is time-triggered tests and live checks.
Does continuous validation replace load testing before a release?
No. A pre-release test is still the place to find breaking points and capacity limits. Continuous validation tells you whether the system keeps meeting its goals afterwards.
How often should scheduled performance tests run?
Nightly or weekly is a sensible start for most teams. Pick the frequency that matches how fast your data and dependencies change, and keep each run short enough to run reliably.
Sources
- Canarying and pre-production limits: Google SRE Workbook, Canarying Releases (October 2026)
- LoadFocus scheduled tests and API monitoring: scheduled load tests guide and loadfocus.com/api-monitoring (October 2026)
Published with PostNext
