How to automate L1 and L2 incident resolution with runtime telemetry

The cost of manual L1/L2 triage

Senior engineers spend too many hours handling repetitive operational failures. When a production service throws an unhandled exception or hits a latency spike, standard monitoring platforms trigger a page. An engineer wakes up, inspects logs, applies a known fix, and deploys a patch. This manual loop consumes engineering capacity that belongs on feature development.

Observability tools like Datadog and Sentry excel at showing where code breaks. However, they act as rear-view mirrors rather than enforcement systems. They alert your team after an outage begins, leaving resolution entirely to human operators. To reduce page volume, teams must move beyond passive monitoring toward active governance. A dedicated AI governance layer sits between generated code and production, handling tier-one and tier-two incidents automatically while building evidence trails. For a broader breakdown of how governance layers differ from linters or bots, see our comparison of AI code risk tools: Static analyzers, PR bots, or governance layers.

Connecting runtime signals from Datadog and Sentry

Automating incident response requires feeding live operational telemetry into your governance control plane. Tomosu AI connects directly with runtime monitoring platforms like Datadog and Sentry, alongside alerting channels like PagerDuty.

Setting up the telemetry stream involves three basic steps:

• Link runtime providers: Connect your Sentry organization and Datadog workspace inside the Tomosu control plane settings to stream error events and performance regressions.
• Map escalation targets: Ingest PagerDuty service routes so the governance system knows which alerts traditionally wake up on-call teams.
• Establish baseline metrics: Allow the platform to calculate initial scores for Runtime Signals (RS) and the Escalation Index across your active repositories.

Once connected, incoming error spikes and service-level regressions bypass noisy operational channels. Instead of sending a raw alert to a Slack channel or paging an engineer at midnight, Sentry exceptions and Datadog latency signals route directly to Tomosu AI for agentic evaluation.

Routing and resolving L1 and L2 incidents automatically

Not every incident requires human intervention. Tier-one (L1) issues—such as minor unhandled exceptions or simple payload mismatches—and tier-two (L2) issues—like checkout latency regressions—often have deterministic fixes.

When a runtime signal fires, Tomosu AI executes a structured resolution sequence:

1. Intercept and classify: The platform captures the error trace from Sentry or the metric anomaly from Datadog. It evaluates the impact against the current Production Reliability Index (PRI) score and Fragility Index (FI).
2. Formulate fix: The multi-tier agentic system analyzes the stack trace against recent pull requests and runtime state. It crafts a targeted patch to fix the underlying fault.
3. Apply and test: The fix is verified against governance policies to ensure it does not introduce secondary drift or compliance violations.
4. Resolve without escalation: If the patch meets safety bounds, the system resolves the incident automatically. Senior engineers are never paged, protecting team bandwidth while maintaining operational stability.

Closing the loop from runtime to IDE

Resolving an incident once is not enough. If the root cause remains unchecked in local developer environments, the same fragile code pattern will reemerge in future pull requests. The real power of an AI governance layer is converting runtime fixes into permanent development guardrails.

When Tomosu AI resolves an L1 or L2 incident, it logs the failure pattern into its risk ledger. That runtime learning propagates back upstream to local editor plugins in VS Code and Cursor. The next time a developer generates code that mirrors the precise logic pattern that caused the production error, local guardrails highlight the risk instantly before pre-merge review. You can read more about this architectural shift in our AI code governance monthly digest: Closing the loop from runtime to IDE.

This continuous feedback loop directly improves your Production Reliability Index. Instead of treating production incidents as isolated fires to extinguish, your engineering organization uses automated resolution to systematically harden codebases against recurring failures.