Retries Before Alerting
Retries Before Alerting
What are retries?
Retries re-attempt a failing check before we send you an alert. A single failed request is not always a real outage: a brief network blip, a slow cold start, or a one-off timeout can all fail a check while your API is healthy. Retries let you confirm a failure is real before anyone gets paged.
Retries only happen when a check fails. A passing check never retries, so the setting costs you nothing while everything is healthy.
Why use retries?
The main reason is alert noise. Without retries, every transient blip becomes an alert, and alerts that are usually false get ignored. With retries, you only hear about failures that survived several attempts.
Set Retries to 0 to turn the feature off. It is off by default.
The settings
| Setting | Range | Default | What it does |
|---|---|---|---|
| Retries | 0 to 5 | 0 (off) | How many extra attempts to make after the first failure |
| Interval (s) | 1 to 60 | 10 | The base wait between attempts |
| Backoff | Fixed, Linear, Exponential | Fixed | How the wait grows with each attempt |
| Region | Same region, Different region | Same region | Where the retry runs |
| Max retry time (s) | 1 to 300 | 60 | A total time budget for the whole retry sequence |
How backoff works
Backoff decides how long we wait before each retry, starting from your Interval. For an interval of 10 seconds:
| Attempt | Fixed | Linear | Exponential |
|---|---|---|---|
| 1st retry | 10s | 10s | 10s |
| 2nd retry | 10s | 20s | 20s |
| 3rd retry | 10s | 30s | 40s |
- Fixed waits the same interval every time. Use it when you expect a failure to clear quickly.
- Linear grows the wait steadily. A reasonable middle ground.
- Exponential doubles the wait each time. Use it when a struggling service needs room to recover, and you do not want your retries adding to the load.
Max retry time
Max retry time is a budget for the whole sequence, measured from the start of the first attempt. Before each wait, we check whether that wait would push the sequence past the budget. If it would, we stop and report the last failure.
This means the budget can end your retries before you have used all of them, which is intentional. For example, with an interval of 10 seconds, Exponential backoff, 3 retries, and a 60 second budget, the third retry would need a 40 second wait on top of roughly 30 seconds already spent. That exceeds the budget, so the check stops after the second retry.
If you want all your retries to run, make sure the budget is larger than the total of the waits above.
What gets recorded
Only the final attempt is saved as the check's result. This is what makes retries useful:
- Fails, then passes on a retry: the check is recorded as passing and no alert is sent. The blip never reaches you.
- Fails every attempt: the check is recorded as failing and your alert channels fire as usual.
Because only the final result is stored, retries do not distort your uptime figures with transient blips.
Region
Same region retries from the same location that saw the failure. Different region is reserved for confirming a failure from a second location. In the current version, "Different region" still retries in the same region.
Practical use cases
- Public APIs over the internet: 1 to 2 retries with Fixed backoff and a short interval filters out routine network noise.
- Services that cold start: use Linear or Exponential backoff so a slow first response has time to warm up.
- A service under load: use Exponential backoff so your retries give it room instead of adding pressure.
- Critical checks that must page immediately: leave Retries at
0so a single failure alerts straight away.
A note on frequency
Keep your retry budget below your check frequency. If a check runs every minute and your retries can run for up to 300 seconds, a failing check would still be retrying when its next run is due.