What Is Production Reliability in Software Engineering?

Production reliability is the ability of a software change to behave correctly and consistently once it reaches real production conditions.

For engineering teams, that means looking beyond whether code passes tests and asking what the change touches, what dependencies it affects, how the surrounding system behaves in production, and what evidence exists before it ships.

Why production reliability is different from test results

Test coverage tells you what was checked, not everything that could break. CI passing tells you the build and automated checks completed successfully. Code review approval tells you that someone examined the change. All three are real signals. None of them directly answers how a specific change will behave once it interacts with a running production system.

That gap has traditionally been partly covered by engineers who have built up experience with a particular codebase and its failure modes. That becomes harder to rely on as the volume of changes grows, whether the increase comes from a larger team or from AI agents generating more diffs. Experience still matters, but it is difficult to turn individual intuition into a repeatable signal that can be tracked before a change ships.

Why change level reliability matters

Production reliability is increasingly being considered at the level of individual changes, not only at the service level.

Meta has publicly described Diff Risk Score (DRS), a system built around a fine tuned LLM that evaluates code changes and their metadata to produce a risk score and flag potentially risky segments before a change ships.

The useful point here is not that every team should use the same method. It is that analyzing an individual code change for potential production impact is an active engineering problem, not just a theoretical idea.

Service health metrics tell you what is happening now, usually in aggregate. Change level analysis asks a different question:

What could this particular, not yet shipped change mean for the system once it reaches production?

What can affect production reliability?

There isn't one universally accepted checklist for evaluating the reliability of an individual software change. In practice, teams tend to look at several kinds of evidence:

A change can look fine on the dimensions you're not tracking and still be the one that causes a production problem, because the signal that mattered was not part of the review.

A simple example

Imagine a pull request that changes a payment service. The tests pass. CI is green. A reviewer approves it.

The same change also modifies a shared database query that three other services depend on, touches a dependency with a recent incident history, and ships without a clear rollback path.

Both the green CI result and the approved review are useful signals. They just don't tell you much about the dependencies, production history, or rollback conditions surrounding this particular change.

That is the kind of context production reliability analysis is meant to surface before a change reaches users.

How production reliability can be measured

DORA's software delivery performance metrics measure how software changes move through production over time. The current model covers change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate.

These metrics are useful for understanding delivery throughput and instability across a team or application. They don't answer whether a specific pull request should be merged, which is a different question.

SRE error budgets operate at a similar service level. They help teams decide how much unreliability they can absorb while balancing delivery speed against service objectives. They are not designed to score an individual change before it ships.

There is no single universally adopted standard that scores an individual change using all of the evidence described above before merge. That is the specific gap that change level reliability analysis, using whatever method a particular team or vendor chooses, is trying to address.

Change level vs service level

Service level metrics tell you how an application is behaving overall. Change level analysis narrows the question to a particular modification.

For example, an application's error rate might look normal while a new pull request changes a shared dependency or introduces a new path through a critical component.

• The service level signal tells you what is happening now.
• Change level analysis gives you another way to examine the change before it becomes part of that production state.

Production reliability vs code review

Code review and production reliability analysis answer related but different questions.

Neither replaces the other. They are built to surface different classes of problems.

If you want to go deeper on this distinction, see Production Reliability vs Code Review.

Production reliability vs observability

AI observability tells you what a system or agent did and why: prompts, tool calls, outputs, and runtime traces. It provides evidence about system behavior.

Reliability analysis can use evidence like that as one input among several, alongside dependency mapping, change history, and incident history. Observability data on its own does not provide a complete assessment of a specific change before it ships.

For more on observability, see What Is AI Observability?. You can also read LLM Observability Tools Compared and Agentic AI Observability for the more specific observability side of the problem.

How Tomosu uses the Production Reliability Index

The Production Reliability Index (PRI) is Tomosu's way of turning multiple reliability signals into a single score that can be tracked over time.

Tomosu's platform describes PRI as a master score rolled up from seven sub indices, calibrated per organization, operating across development, pre merge, and production.

The point of the score is not to replace the underlying evidence. It is to give a team one trendable signal while keeping the individual indexes and findings available underneath it.

See how the Production Reliability Index works →

How to assess production reliability yourself

You don't need PRI specifically to start doing this better. Start with the evidence you already have:

• Dependencies
• Testing coverage
• Production behavior
• Code churn
• Rollback conditions
• Incident history

Look at how those signals relate to the specific change rather than treating any one of them as a complete measure of reliability.

Then decide what should happen when the evidence is weak. Should the change require another review? More testing? A smaller rollout? A rollback plan? Or simply a record for later analysis? The important part is connecting the signal to an engineering action. For a step by step version of this, see How to Assess Production Reliability Before Deployment.

Frequently asked questions

Production reliability becomes much more useful when you can measure it against the changes moving through your systems. The Production Reliability Index is Tomosu's approach: multiple reliability signals, one score you can track over time. See the Production Reliability Index →