Hub / Guides / Agent observability

Agent observability

Agent loops burning cash? Trace every call.

Yes. You can catch a runaway agent loop before it drains the budget when every model and tool call lands on one trace with an owner and a hard cap. We wire OpenTelemetry GenAI spans, run-level limits, and a live work queue that names who stops the run.

Why didn't our dashboards stop the loop?

Because dashboards watch, and nothing in them is allowed to say no. The best-known case is a LangChain agent-to-agent loop that ran up roughly $47,000 before anyone pulled the plug. The write-up puts it bluntly: "The team had observability. They did not have enforcement." (case study)

The harder lesson is that every single call in that loop looked healthy. Each request was well-formed, stayed under max_tokens, and returned a 200 (analysis). Per-call logging can't see a problem that only exists at the level of the whole run. You need traces shaped around runs, and you need a rule that acts on them.

What should an agent trace actually record?

There is now a shared standard for this. The OpenTelemetry GenAI semantic conventions define agent spans (invoke_agent, create_agent) and tool spans named execute_tool {gen_ai.tool.name} (spec). The attributes your reviewer will lean on most are:

Tool-call arguments and results are opt-in for privacy, so decide up front who is allowed to turn them on. One caution: the spec is still marked Development, which means attribute names can change. Pin a version and plan to re-check it.

Microsoft's guidance for agents adds the part a trace alone can't give you: log every tool invocation with its inputs, outputs, the identity it ran under, and the rationale (Azure shared responsibility). Identity is the field most teams skip, and it's the one you'll want first when someone asks which agent changed what.

Do our frameworks and tools already emit these spans?

Many do. The OpenTelemetry reference table lists execute-tool spans from AutoGen, CrewAI, Google ADK, LangChain, LlamaIndex, OpenAI Agents, and Pydantic AI, among others (coverage table). The OpenAI Agents SDK also ships built-in tracing for agent runs, tool calls, handoffs, and guardrails (SDK tracing).

On the backend side, LangSmith ingests OTLP and maps gen_ai.operation.name to its run types (LangSmith OTel), and Langfuse accepts OTLP and reads cost from gen_ai.usage.cost (Langfuse OTel). A collector can fan the same spans out to more than one backend, so you don't have to pick one tool forever. We don't rank these; your stack decides the fit.

How should one agent run be shaped?

Make one trace equal one agent run, and tie turns together with a session. Inside the run, record one generation per model call, interleaved with the tool observations between them. Langfuse's guidance warns against wrapping the whole loop in a single parent generation, because that hides which tool call blew up the context or the cost (trace best practices).

When the shape is right, you can answer the question that matters during an incident: which step, by which agent, under whose credentials, started the burn.

What signals tell us a run is going wrong?

Watch the shape of the graph, not single calls. Three numbers carry most of the warning: calls per run, the rate of 429 responses, and spend per run. A steady ramp usually means a loop. A staircase usually means retry amplification, where three layers each retrying three times turn one failed request into 27 (runaway loops).

Put a hard ceiling on each of those numbers, and make crossing it stop new work rather than just send an alert. That's the enforcement the $47K case was missing. For the budget side of this, see our guide on how to govern agent model spend.

Where do call traces stop being enough?

A call trace tells you what happened inside one run. It doesn't tell you who owns the work, what stage it's in, or who is supposed to act when it stalls. That second record is accountability, and it lives on the control plane, not in the tracing backend. Our guide to map the agentic control plane splits those layers apart.

In practice, the two records work as a pair. The trace shows the eng lead which tool call ran away. The work queue shows which agent owned that work card, when it last checked in, and whether anyone has been named accountable. In the open-source 0xRay agent kit, each cloud agent gets its own row on a live board, installed hooks post activity to it, and a missed health beat exits non-zero so a stall is loud instead of silent. Neither record replaces the other.

What does 0xRay set up for this?

Through AI Fleet OPS, we stand up the open-source agent kit with your team and wire ownership, review gates, and a live work queue next to the tracing you already run. Concretely, that means OpenTelemetry spans shaped per run, a run-level cap with a named owner who can stop it, and release checks before anything merges, as covered in release gates for agents. We don't promise savings figures; we make the burn visible and give one person the right to halt it.

How do you request a consult?

Email consult@0xray.dev.

Request a consult

What else do operators ask?

Is agent observability the same as LLM observability?

Mostly, but agent observability also traces tool calls and handoffs between agents, not just model prompts and completions.

Does OpenTelemetry have a standard for AI agents?

Yes. The OpenTelemetry GenAI semantic conventions define agent and tool spans, though the spec is still marked Development and names can change.

Why didn't max_tokens stop our runaway loop?

Because each call stayed under the limit and returned 200. The problem only shows at the run level, so you need run caps that stop new work.

Should we log tool-call arguments and results?

Only by deliberate choice. They are opt-in in the spec for privacy, so name who may enable them and where that data is kept.