How to Measure Software Change Reliability

Knowing that a change should be assessed before deployment is one thing. Measuring it consistently is harder.

A pull request can have passing tests, a clean review, and a successful CI run while still introducing uncertainty around dependencies, production behavior, or blast radius.

So how do you actually measure the reliability of a software change? There isn't one universally accepted formula. Instead, change reliability can be measured by bringing together several signals about the change, the system it affects, and the evidence available before and after deployment.

The useful question is:

What can we measure about a software change that tells us something meaningful about its production reliability?

If you're starting from the assessment process, see How to Assess Production Reliability Before Deployment.

What does software change reliability mean?

Software change reliability is about how reliably a particular change behaves once it interacts with the production system. That makes it different from measuring the reliability of the application as a whole.

For example, a service might have 99.9% availability, low error rates, stable latency, and healthy resource usage. Those are useful service level signals. But they don't tell you whether the next pull request will preserve that behavior.

Change level measurement narrows the question to the modification itself:

What evidence do we have about this change, and how does that evidence relate to production behavior?

What can you measure?

A practical model can start with several categories of evidence:

These signals don't all have to become one formula. Some are useful before deployment. Others become useful after the change reaches production. The important part is understanding what each signal actually tells you.

1. Measure the scope of the change

The first signal is the change itself. You can measure things such as:

• Number of files changed
• Number of components affected
• Number of services touched
• New execution paths
• Database or schema changes
• Configuration changes
• Shared components modified

But raw change size is only a starting point. A 1,000 line change isolated to one internal component may have less production impact than a small change to a shared authentication or database component.

So instead of asking only “How large is the diff?” also ask:

How much of the production system can this diff affect?

That gives you a more useful measure of change scope.

2. Measure dependency impact

Dependencies are another important part of change reliability. A change can affect production indirectly through:

• Internal services
• APIs
• Databases
• Queues
• Shared libraries
• Third party services
• Infrastructure
• Configuration

One useful approach is to map the components affected by the change and identify how many other production components depend on them. The more connected a component is, the more context you may need when evaluating a change.

This is also where blast radius becomes useful. A change affecting one isolated component has a different potential production surface from a change touching a shared component used across multiple services. For a deeper look at this, see How to Assess the Blast Radius of a Code Change.

3. Measure what was actually verified

Testing provides some of the most obvious signals around a software change. You can measure:

• Test coverage
• Number of tests added or changed
• Unit test results
• Integration test results
• End to end test results
• Static analysis results
• Failed and passed checks
• Untested paths

But test coverage shouldn't be treated as a direct reliability score. A change can have high coverage while missing an important integration. Another change can have lower coverage because it modifies a well isolated component with a small production surface.

The useful question is not simply “What percentage of the code is covered?” It is:

What important behavior introduced by this change has actually been verified?

4. Measure production behavior

Once a change reaches production, you can observe what actually happened. Useful signals can include:

• Error rate
• Latency
• Resource usage
• Dependency failures
• Traffic changes
• Saturation
• Alerts
• SLO performance

These measurements are valuable because they provide evidence about the real production outcome of the change. For example, a deployment might pass every pre deployment check but produce an unexpected increase in latency. That production behavior becomes evidence about the change.

This is where observability and change reliability connect. Observability tells you what happened in the running system. Change reliability analysis connects that evidence back to the change that produced it. See What Is AI Observability? for more on the observability side.

5. Measure code volatility

Some parts of a codebase change constantly. Others barely change. That difference can provide useful context. You can look at:

• Commit frequency
• Lines added and removed
• Number of changes to a component
• Repeated modifications to the same area
• Recent churn
• Concentration of changes around specific files

High churn doesn't automatically mean low reliability. But frequent changes can indicate that an area deserves more context when evaluating another modification.

The useful measurement isn't simply “How much code changed?” It's:

How much change activity is happening around the area this pull request touches?

6. Measure incident history

Past incidents provide another source of evidence. For the components affected by a change, you can look at:

• Number of previous incidents
• Severity of previous incidents
• Frequency of incidents
• Recent incidents
• Repeated failure patterns
• Previous rollbacks

Historical incidents shouldn't be treated as a prediction that the next change will fail. They provide context.

For example, if a service has repeatedly experienced problems around connection management and a new change modifies that same area, the historical signal becomes relevant to the assessment.

7. Measure deployment outcomes

After deployment, you can compare what happened against what was expected. Useful measurements include:

• Whether the deployment succeeded
• Whether a rollback occurred
• Whether an incident followed the deployment
• Whether error rates changed
• Whether latency changed
• Whether resource consumption changed
• Whether the change required remediation

This creates an important feedback loop:

Over time, those outcomes can become evidence for evaluating future changes.

Production risk scoring vs reliability measurement

You will sometimes see this problem described as production risk scoring or software change risk scoring. The framing is slightly different:

The important thing is not the label. It is whether the measurement gives engineers useful information about the change.

Is there a formula for production risk?

There isn't a single universally accepted formula for calculating production risk for an individual software change. A simple conceptual model could look like:

The exact weighting depends on the system. A dependency change might matter much more for one architecture than another. A rollback failure might be critical for one service and relatively minor for another.

This is why a fixed formula can be misleading when it is disconnected from the environment where the change will run.

Why a single number can still be useful

If there are many signals, engineers can end up with a different problem: too much information. You might have test results, static analysis, dependency data, incident history, production metrics, code churn, and deployment data.

All of these can be useful. But if every pull request requires someone to manually combine them, the process becomes difficult to apply consistently.

This is where a summary score can become useful. The score shouldn't replace the underlying evidence. It should make the overall signal easier to track while allowing engineers to inspect the factors behind it.

How Tomosu measures reliability with PRI

The Production Reliability Index (PRI) is Tomosu's approach to summarizing production reliability signals into a single trendable score. Tomosu describes PRI as a master score built from seven sub indices:

• Fragility Index
• Drift Index
• Governance Compliance
• Runtime Signals
• Code Volatility
• Deployment Velocity
• Escalation Index

The score is calibrated per organization and intended to provide a consistent reliability signal while the underlying indexes provide more detail.

This is different from simply counting test failures or looking at one production metric. The goal is to bring multiple dimensions of production reliability into one measurement that can be tracked over time.

See the Production Reliability Index →

How to build your own software change reliability measurement

You don't need to start with a complex scoring system. Start by collecting the signals you already have.

Then connect the two. The important question becomes:

Did the signals we had before deployment tell us anything useful about what happened afterward?

That feedback loop is what makes the measurement more useful over time.

What makes a reliability measurement useful?

A useful measurement should be:

What to read next

If you want to understand the process before getting into measurement, read How to Assess Production Reliability Before Deployment. If you want to go deeper into one of the most important inputs, read How to Assess the Blast Radius of a Code Change. And if you want to see how Tomosu combines reliability signals into one measurement, see the Production Reliability Index.

Frequently asked questions

A score that cannot be explained is difficult for engineers to trust. The Production Reliability Index rolls seven sub indices into one trendable measurement, with every underlying signal still there to inspect. See the Production Reliability Index →