Building an automated escalation pipeline with Datadog and PagerDuty

Engineering teams deploying software faster often run into a familiar wall. As change volume grows, alert volume follows. On-call rotations turn into reactive triage sessions dominated by transient L1 and L2 alerts. When engineers spend their shifts suppressing noise, critical outages take longer to diagnose and fix.

Solving pager fatigue requires changing how alerts move from production monitors to human responders. Placing a governance layer between monitoring tools and incident response platforms decouples telemetry from immediate paging. Here is how to construct an automated escalation stack using Datadog, Tomosu AI, and PagerDuty.

The mechanics of an automated escalation stack

A resilient incident workflow relies on three distinct layers working in sequence:

• Detection: Datadog monitors service-level objectives (SLOs), latency spikes, and error rates in real time.
• Governance and remediation: Tomosu AI evaluates incoming telemetry against the application's Production Reliability Index (PRI) and handles routine remediation via its multi-tier agentic system.
• Escalation: PagerDuty pages the on-call engineer only when automated remediation fails or when risk metrics cross critical thresholds.

This separation ensures that high-volume runtime signals do not immediately trip the pager. Instead, routine failures are filtered and addressed at the governance layer.

Step 1: Route Datadog SLO monitors to the governance layer

Start by configuring Datadog to send SLO breach alerts and runtime signals directly to Tomosu AI rather than routing them straight to an on-call schedule. Datadog captures operational parameters like checkout latency regressions or elevated error budgets.

When an SLO degrades, Datadog posts an alert payload to Tomosu AI. The platform ingests this signal alongside the current PRI score and historical repo data. Rather than treating every anomaly as a human emergency, the system checks whether the fault matches known fragility patterns or recent deployment drift.

2 Automate L1 and L2 remediation

Once Tomosu AI receives the Datadog alert, its multi-tier agentic system attempts automated L1 or L2 resolution. If an application experiences a known transient regression or secondary configuration fault, the platform executes a pre-approved remediation path to protect the service-level agreement.

This step reduces manual triage effort. For a deeper breakdown on setting up runtime telemetry triggers for automated tier-one fixes, read our guide on how to automate L1 and L2 incident resolution with runtime telemetry.

During this phase, Tomosu AI tracks the Escalation Index. The Escalation Index measures how often an alert requires human intervention versus automated resolution. If the platform successfully stabilizes the service without human help, the incident closes automatically, and the on-call engineer remains unpaged.

Step 3: Trigger PagerDuty on true escalations

PagerDuty should act as a true emergency system, not a notification log. Configure PagerDuty integration rules within Tomosu AI to trigger under specific conditions:

• The multi-tier agent fails to resolve the runtime regression within a defined window.
• The incident severity indicates potential data corruption or core infrastructure loss.
• The application's Escalation Index breaches acceptable operational thresholds.

When an alert reaches PagerDuty, it carries full governance context. The paged engineer sees not just the Datadog stack trace, but the exact commits, PRI impact, and automated remediation steps that were already attempted. This eliminates initial investigation churn during an off-hours call.

Closing the runtime-to-IDE feedback loop

An automated escalation pipeline should do more than suppress pages. It must prevent identical failures from recurring in future releases. When an incident is resolved—whether automatically or by an engineer—Tomosu AI converts the failure context into active guardrails.

These guardrails sync to developer environments through VS Code and Cursor extensions, as well as the Tomosu MCP Server. Developers writing new changes receive immediate local feedback if their diff introduces the same risk profile that previously triggered a Datadog alert. To understand how teams capture runtime signals to inform pre-merge reviews, see our summary on closing the loop from runtime to IDE.

Trade-offs and operational realities

Automating incident escalations requires explicit boundaries. Before turning on automated L1 and L2 remediation, teams must establish strict policies around which actions agents can take automatically. Granting agents unchecked write or deployment permissions in production environments introduces unnecessary risk.

Initial calibration also takes time. If Datadog SLO thresholds are set too aggressively, Tomosu AI will spend compute processing trivial noise. If rules are too loose, true outages may delay human paging. Teams should run the stack in read-only mode initially to baseline their Escalation Index and verify agent remediation logic before enabling automated execution.