Failover Testing for Database Cluster Resilience

Measure the outage a database failover actually causes

A failover that "takes 30 seconds" on the database dashboard can take three minutes for users: connections in the pool point at the old primary until they fail, the ORM retries, DNS caches the old address, and the new primary starts cold. This template holds a steady, realistic load on the application while you trigger the failover, so the results show what users saw: how many seconds of errors, how long p95 stayed elevated, and whether anything failed to recover.

Configuration

Run this templateOpens the cloud test form with these values filled in. Free plan runs it at the free user limit; sign in or create a free account first.

The button prefills users, duration and ramp-up. Add the endpoints, start the test, wait five minutes for a flat baseline, then trigger the failover on staging and note the time. Read the results against that timestamp.

What to read in the results

• The error window. The Errors tab at 5-second granularity shows exactly when errors began and ended. That span is the outage as users experienced it, and it is usually longer than the database reports.
• Reads versus writes. Filter by request. Reads served by replicas may continue; writes fail until the pool reconnects to the new primary. If reads fail too, the app pins every connection to the primary.
• Recovery slope. After the errors stop, p95 should return to the baseline within a minute or two. A long tail is a cold cache or buffer pool on the new primary; a permanent step up means some pool members are still on a slow path.

Pass/fail thresholds for this template

These thresholds turn the failover drill into a number to compare across driver, pool and cluster changes.

The same scenario as a k6 script

As a k6 script the load is deliberately flat; the interesting part happens outside the script when you trigger the failover.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [{ duration: '2m', target: 300 }, { duration: '18m', target: 300 }],
  thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(95)<600'] },
};

const BASE = 'https://api.example.com';

export default function () {
  check(http.get(`${BASE}/accounts/${1 + Math.floor(Math.random() * 1000)}`), { 'read 200': (r) => r.status === 200 });
  check(http.post(`${BASE}/events`, JSON.stringify({ type: 'view', at: Date.now() }), { headers: { 'Content-Type': 'application/json' } }), { 'write 2xx': (r) => r.status >= 200 && r.status < 300 });
  sleep(1 + Math.random());
}

When to run it

• Before relying on automatic failover in a runbook or an SLA.
• After changing the driver, the pool, or DNS TTLs each changes how fast clients follow the new primary.
• Quarterly as a drill so the number is current.

FAQ on database failover testing

Can I run this on production?

Only as a planned drill in a maintenance window with a rollback plan. The template is designed for staging with a production-like cluster.

The database says failover took 20 seconds but the test shows 90 seconds of errors.

The difference is the client side: pooled connections that fail one by one, retries with backoff, and DNS caching. Set a short DNS TTL, validate connections on borrow, and retry writes with a bounded backoff.

Reads kept working but writes failed. Is that acceptable?

That is the expected shape when reads use replicas. Whether it is acceptable depends on the product; the test tells you how long the write outage lasted so you can decide.

How do I line up the failover time with the results?

Note the wall-clock time when you trigger it; the Timeline and Errors tabs are on the same clock at 5-second granularity. Add a run annotation with the time so it stays with the result.