What is Reliability Engineering?

What is reliability engineering?

Reliability engineering is the practice of designing, running and improving systems so they meet a measurable reliability target, such as the share of requests that succeed or a latency percentile. The target is written down and measured, and the team makes trade-offs against it, such as shipping faster versus fixing stability. Site Reliability Engineering (SRE) is the best-known software version.

Reliability engineering, SRE and DevOps

The terms overlap, but they point at different things. Reliability engineering is the goal: the system meets its targets. SRE is Google's operating model for reaching it. According to the SRE book, Benjamin Treynor Sloss started it at Google in 2003, and it describes SRE as "what happens when you ask a software engineer to design an operations team". Google caps operational work at 50% of an SRE's time so the rest goes to engineering. DevOps is a culture and toolset that closes the gap between development and operations.

A practical test: if your team has a reliability number it defends, you are doing reliability engineering, whatever the job titles say.

The core building blocks

• SLI (service level indicator). A carefully defined quantitative measure of some aspect of the service, such as request latency or error rate.
• SLO (service level objective). A target value or range for an SLI, for example "99.9% of requests succeed".
• SLA (service level agreement). A contract with consequences if the objectives are missed. If there is no explicit consequence, you most likely have an SLO. See SLA management.
• Error budget. One minus the availability target. A 99.99% target leaves a 0.01% budget that can be spent on releases and experiments. In Google's model, releases are halted when the budget is exhausted.
• Incident response and blameless postmortems. Postmortems focus on contributing causes without blaming a person or team, and are written for significant incidents.
• Toil reduction. Toil is manual, repetitive, automatable work that scales with the service. The SRE goal is to keep it below 50% of each engineer's time.
• Capacity and architecture. Redundancy, dependency isolation and graceful degradation, proven by capacity testing.

Example: one SLO and its budget

These numbers are illustrative. Say the SLI is the share of /checkout requests that succeed in under 1,500 ms, and the SLO is 99.9% over 30 days. The error budget is 0.1%. At 1,000,000 checkout requests in the window, that allows 1,000 bad requests. If the budget is half spent in the first week, the team slows risky releases and works on stability. If it is barely touched, the team can ship faster, or tighten the SLO.

The SRE book gives useful advice for choosing the numbers: start from what users care about, use a few indicators, prefer percentiles to averages (why percentiles beat averages), do not copy your current performance into an SLO, and start with a looser target you can tighten. Read it in the service level objectives chapter.

Metrics reliability teams track

• SLO attainment: the share of the SLO target met per service over the window.
• Error budget burn rate: how fast the budget is being spent, and when it runs out at that pace.
• MTTD (mean time to detect): from the start of an incident to the first alert.
• MTTR (mean time to restore): from the start of an incident to user-visible recovery.
• Change failure rate: the share of deployments that cause an incident or rollback.
• Toil share: the fraction of an engineer's time spent on manual, repetitive work.

Common mistakes

• Aiming for 100%. The SRE book says 100% is probably never the right reliability target: users cannot tell the difference and the cost climbs steeply.
• Too many indicators. A handful of representative ones is enough to judge a system's health.
• Judging by averages. A healthy mean can hide a slow tail that real users feel.
• An error budget nobody acts on. The budget only works if exhausting it changes what the team does.
• SLOs that never get reviewed. Revisit them against real user behavior and product plans.
• Postmortems with no follow-through. Assign every action item to an owner and track it to completion.
• Never testing under load. A latency SLO you have only seen at quiet hours is a guess.

Where load testing and monitoring fit

You cannot defend a latency SLO without evidence that the system delivers under realistic concurrency. Load testing before launches and after big changes supplies that evidence. With LoadFocus load testing you can set pass/fail thresholds on P95 and P99 response time, error rate and throughput, so a run gets a verdict, and a GitHub Action can block a deploy when a threshold is breached. That turns your SLO into a release gate.

In production, API monitoring checks your endpoints on a schedule from several regions and alerts you when assertions fail; LoadFocus API monitoring has a free plan with checks every 5 minutes. Status pages can show uptime built from those checks and let you post incidents. Metrics, logs and traces are covered under observability, and our monitoring best practices guide covers the habits around them.

FAQ

What is the difference between reliability engineering and SRE?

Reliability engineering is the discipline of making a system meet measurable reliability targets. SRE is one specific way to do it, started at Google, in which software engineers run production with SLOs, error budgets and a cap on operational work.

What is an error budget?

It is one minus the availability target. With a 99.99% availability target the budget is 0.01%. Teams spend it on releases and experiments, and slow down or stop risky changes when it runs out.

What is the difference between an SLO and an SLA?

An SLO is an internal target for a service level measured by an SLI. An SLA is a contract with users that includes consequences if the objectives are missed. If there is no explicit consequence, it is most likely an SLO.

Is 100% uptime a good goal?

Usually not. The SRE book argues that 100% is probably never the right target, because users rarely notice the difference and each extra nine costs more. Pick a target that matches what users need and what you can afford.

How do I get started?

Pick one user-critical journey, define an SLI for it, set a loose SLO and measure it from production data. Compute the error budget, agree what happens when it runs out, and run a blameless postmortem after every significant incident.