Once an LLM application reaches production, normal APM usually isn't enough to understand every failure. A request can succeed while the model gives the wrong answer. An agent can complete a workflow using the wrong tool. A RAG application can return a confident response built on badly retrieved context.
That's where LLM observability comes in. Three names dominate the category: Datadog, Arize, and Langfuse. This isn't a ranking. The right choice depends mostly on the stack you already have and what you're trying to observe.
What each tool is
Datadog
Started with infrastructure and application monitoring, and added LLM and agent observability to the broader platform. The natural fit if you already run Datadog for infra and application telemetry and want model traces sitting in the same environment as everything else.
Arize
Started around ML observability and expanded into LLM observability and evaluation. Arize also maintains Phoenix as an open source component, which matters if you want part of the stack self-hosted without moving off the platform entirely.
Langfuse
LLM-native and open source at its core. Langfuse focuses on tracing, prompt management, evaluations, and LLM application observability, with self-hosting as a first-class option rather than an enterprise afterthought.
Feature sets and pricing in this category change fast. Verify current details against the vendors' own documentation before you commit to anything, including the summary below.
Where they overlap
All three provide some version of prompt and response tracing, multi-step traces, and evaluation support. The overlap is large enough that a feature-by-feature grid will mislead you. The useful difference is contextual: what you already run, and how much of it you need to control.
| Tool | General fit | Open source |
|---|---|---|
| Datadog | Teams already standardized on Datadog for infra and app telemetry | No |
| Arize | Teams working across both ML and LLM systems | Phoenix component |
| Langfuse | Teams wanting LLM-native tooling and a self-hosted option | Core is open source |
Positioning as of writing. Check each vendor's current docs before deciding.
What observability answers
- What prompt produced this response?
- Which model was used?
- What context was retrieved?
- Which tools did the agent call, and in what order?
- Where did latency or cost increase?
- Did output quality change over time?
Those are good questions, and having real answers to them is the difference between debugging an AI system and guessing at it. For the definition underneath all of this, see what AI observability actually means.
Observability isn't a merge decision
Imagine an AI coding agent generates a configuration change. An observability platform helps you understand what happened when that agent ran: the prompt, the model, the tool calls, the output.
But the fact that you can reconstruct the run doesn't mean the change should have been merged. That's a different control point entirely. Observability is descriptive. A merge gate is a decision.
Where production risk fits
A production risk check looks at the change itself and its surrounding context rather than at the model run.
- What does the change touch?
- What depends on it?
- Has this area caused incidents before?
- Is it crossing a shared boundary?
- What is the potential blast radius?
That information combines with observability rather than replacing it. Observability answers what happened. Production risk analysis answers how much risk are we accepting by shipping this change.
Knowing after the fact that an agent's output was wrong is useful. It isn't the same as catching it before it merges.
What none of these tools do
All three answer “what did the model or agent do.” None of them answer “should this have shipped.”
That's a governance and verification problem, and it sits on top of whichever observability tool you pick rather than inside it. Tomosu sits in that gap: not another observability tool, but the merge-gate check that runs after whichever one of these you choose. If you're evaluating that layer, the buyer's checklist lists the seven questions that separate a real gate from a dashboard.
Pick whichever of these fits your stack. That decision is worth getting right, and none of the three are bad answers. Just don't file it under the same heading as the decision about what ships.
Frequently asked questions
What does LLM observability mean?
Tracking what a model or agent did and why, at the level of prompts, tool calls, and outputs, rather than just server uptime. See What Is AI Observability? A Definition for Engineers for the full definition.
Are there open source LLM observability tools?
Yes. Langfuse is open source at the core, and Arize ships Phoenix as a separate open-source project alongside its paid platform. Datadog's LLM Observability is not open source.
Is Grafana an observability tool?
Yes, but not an LLM-specific one out of the box. Grafana is a visualization and dashboarding layer, most often paired with Prometheus for metrics, built for infrastructure and application observability rather than prompt-level LLM tracing.
What's the difference between Prometheus and Grafana?
Prometheus collects and stores metrics as time series and handles alerting. Grafana visualizes that data, and data from other sources, as dashboards. They're usually run together, not as substitutes for each other.
Why is observability called o11y?
It's a numeronym: o, eleven letters, y. Same pattern as i18n for internationalization.
What's the best tool for AI observability?
There isn't a single best one. It depends on what you're already running and whether you need open source. The comparison above covers how Datadog, Arize, and Langfuse actually differ.
Pick whichever of these fits your stack — that decision is worth getting right. Just don't mistake it for the governance decision. Tomosu runs at the merge gate, checking whether a change should have shipped, not only recording that it did. Book a call →
