How to Assess Production Reliability Before Deployment

A pull request can pass CI, get approved in code review, and still introduce a production problem.

The reason is simple: most pre-deployment checks look at individual parts of a change. Production reliability depends on how those parts interact with the system around them.

A useful production reliability assessment asks a broader question:

What evidence do we have that this specific change is ready to reach production?

This doesn't mean predicting exactly what will happen after deployment. It means bringing the relevant evidence together before the change ships and identifying where uncertainty remains.

If you haven't read it yet, start with What Is Production Reliability in Software Engineering?.

What should you assess before deployment?

There isn't one universal checklist that works for every application. The useful signals depend on the architecture, the change, the team's deployment process, and the production history of the system.

But a practical assessment can start with six areas:

The goal isn't to produce a perfect score from these signals. The goal is to understand what you know, what you don't know, and whether the remaining uncertainty is acceptable before deployment.

1. Start with the change itself

First, understand what the change actually modifies. Look at:

• Files changed
• Components affected
• Services involved
• New execution paths
• Database or schema changes
• Configuration changes
• External integrations
• Shared libraries or dependencies

A small diff isn't necessarily a small production change. A five line modification to a shared authentication library can have a much larger production surface than a 500 line change isolated to an internal tool.

This is why line count alone isn't a useful measure of production reliability. The important question is:

What parts of the production system can this change influence?

2. Map the dependencies

Next, look at what the change depends on and what depends on it. This can include:

• Internal services
• APIs
• Databases
• Queues
• Third party services
• Shared libraries
• Infrastructure
• Configuration
• Authentication or authorization systems

Dependencies matter because a change can be correct in isolation while interacting badly with another component.

For example, a service might introduce a new database query that works correctly against its own test data. But if the query changes load characteristics on a shared database, the production impact may extend beyond the service that owns the pull request.

Dependency context helps expose that kind of relationship before deployment.

3. Look at what was actually tested

“Tests passed” is useful information. It isn't the same thing as “the change has been fully verified.” Ask:

• Which paths are covered?
• Which paths aren't?
• Are integration tests involved?
• Are production like datasets being used?
• Did the tests exercise the affected dependencies?
• Were failure conditions tested?
• Are there known gaps in this part of the system?

The important distinction is between tests that passed and behavior that has actually been verified. A change can have excellent test coverage in general while still introducing an untested interaction.

4. Check current production behavior

Production already contains useful evidence about the system. Depending on the application, that might include:

• Error rates
• Latency
• Resource usage
• Traffic patterns
• Dependency health
• Recent alerts
• Existing incidents
• Service level objectives

This is where observability becomes useful. Observability tells you what the system is doing now. That information can provide context for understanding what a new change is about to modify.

But observability by itself doesn't answer whether the change should ship. That requires connecting the production evidence to the specific change. For more on this distinction, see What Is AI Observability?.

5. Check incident and change history

Past production behavior can be useful evidence when evaluating a new change. Look for:

• Previous incidents in the affected component
• Recent changes to the same area
• Repeated failures
• Known fragile dependencies
• Previous rollbacks
• Recent configuration changes
• Areas with high code churn

For example, imagine a pull request changes a service that has already experienced several production incidents related to connection pooling.

That history doesn't mean the new change is automatically unsafe. It does mean there is useful context that shouldn't disappear simply because the current pull request passes its tests.

Historical evidence helps answer:

Has this part of the system caused problems before, and does this change interact with the same conditions?

6. Check deployment and rollback conditions

A reliability assessment shouldn't stop at the code. Ask what happens if the change doesn't behave as expected. Consider:

• Can the deployment be rolled back?
• How quickly can it be rolled back?
• Does rollback require a database migration?
• Can the old and new versions run at the same time?
• Is a feature flag available?
• Can the rollout be limited?
• What production signal would indicate a problem?

A change with a clear rollback path has a different operational context from a change that is difficult to reverse.

This doesn't automatically make one reliable and the other unreliable. It changes the consequences of a failure and the options available to the team.

Put the signals together

The six areas above shouldn't be treated as six independent checkboxes. The useful part is the relationship between them. Consider a hypothetical pull request:

None of those signals alone tells you what will happen. Together, they give an engineer much more context.

That is the point of production reliability assessment.

What evidence should block deployment?

This is where teams need an explicit policy. Not every weak signal should automatically block a deployment. The same evidence — missing integration coverage — can call for different responses:

The important part is defining these responses before the situation happens. A reliability signal is much more useful when it is connected to an action.

How to think about production reliability before deployment

A simple model is:

The goal isn't to eliminate all uncertainty. That's not realistic in a production system.

Where the Production Reliability Index fits

The Production Reliability Index (PRI) is Tomosu's approach to turning multiple reliability signals into a single trendable score. Tomosu describes PRI as a master score built from seven sub indices:

• Fragility Index
• Drift Index
• Governance Compliance
• Runtime Signals
• Code Volatility
• Deployment Velocity
• Escalation Index

The underlying signals provide the detail. PRI provides a way to track the overall reliability picture over time.

For teams working with a large number of changes, the idea is to move from manually piecing together reliability context toward having a consistent signal attached to the production environment and the changes moving through it.

See the Production Reliability Index →

Production reliability assessment vs code review

Code review remains an important part of the process. But code review primarily examines the change itself. Production reliability assessment expands the context around that change.

A reviewer might ask:

Does this implementation do what it is supposed to do?

A production reliability assessment also asks:

• What does this change touch?
• What depends on it?
• What has happened in this part of the system before?
• What production signals should we pay attention to?
• What happens if the change causes a problem?

These are complementary questions. You can go deeper in Production Reliability vs Code Review.

A practical pre deployment checklist

Before deploying a significant change, ask:

What to read next

This article covers the process of assessing a change before deployment. The next step is understanding how those signals can actually be measured: How to Measure Software Change Reliability. Or see how Tomosu puts these signals together in the Production Reliability Index.

Frequently asked questions

This article covers the process of assessing a change before deployment. The next step is understanding how those signals can actually be measured — or seeing how Tomosu puts them together in one trendable score. How to Measure Software Change Reliability →