Agentic Data Plane

Trace an Agent Run

A trace is the record of one agent run, broken into spans such as a model call, a tool call, or a subagent hand-off. A call that only AI Gateway handled gets a trace too. Use traces to find which LLM call, tool call, subagent, or policy decision made a run slow, expensive, or wrong, without opening each agent separately.

After reading this page, you will be able to:

  • Filter traces by status, agent, user, conversation, MCP server, LLM provider, and time range

  • Expand a trace into its span waterfall and read one span’s detail

  • Choose traces, transcripts, or the audit log for an investigation

For what traces and spans are, see How Observability Works.

Prerequisites

Your account needs these permissions:

Permission What it covers

dataplane_adp_trace_list

Open the traces list and its timeline

dataplane_adp_trace_get

Expand one trace and read its spans

dataplane_adp_trace_content_read

Read the content of spans that no agent or person owns, and of other people’s AI Gateway calls

Without dataplane_adp_trace_list the page reports that permission was denied and hides the timeline. The built-in Read only template, and every template built on top of it, grants all three. To grant them with an access policy, name Action::"Trace.list", Action::"Trace.get", and Action::"Trace.read_content". See Trace permissions.

Content you can reach without dataplane_adp_trace_content_read follows who owns it:

  • An agent’s spans need dataplane_adp_transcript_get on that agent, or dataplane_adp_transcript_list for a call made outside any conversation.

  • Your own AI Gateway calls are yours to read.

A span you can’t read shows as Restricted, and a trace whose content you can’t read shows Restricted in place of its title. See Restricted spans.

Open the traces list

  1. Open Traces under Observability in the sidebar.

  2. Review the most recent traces. The default time range is Last 1 hour, and the newest trace is first. The table loads more traces as you scroll toward the end, and its count line shows how many traces have loaded, out of the total when it’s known.

  3. Click a row to expand it into its span waterfall. Loading more traces leaves an expanded trace open.

The table shows these columns:

Column Description

Started

When the trace started, as a relative time. Hover over it to see the UTC timestamp.

Status

The outcome of the run. See Trace status. When a run that didn’t fail outright holds failed tool calls, the number of failed calls appears beside its status.

Trace

The trace’s title, taken from the first line of the run’s opening message, with the conversation ID underneath when the run belongs to a conversation.

Conversation

The conversation the run belongs to. If the agent still exists, click it to open that conversation’s transcript. To narrow the list to one conversation instead, use the Conversation filter.

Agent

The agent that ran. The name links to the agent while the agent still exists. The cell is empty for a call that only AI Gateway saw, unless an agent made that call.

User

The authenticated person who made the call, or the end user the runtime asserted.

Activity

How many LLM calls and tool calls the run made, over a bar split between the two.

Duration

How long the run took. A running trace counts up instead.

Tokens

Total tokens the run used, as the provider reported them. Hover over it for the exact count.

Cost

What the run’s model calls cost. See Cost.

Sort on any column except Conversation and Activity by clicking its header.

Not every column starts visible. Open the columns control to show or hide Agent, User, Activity, Duration, Tokens, Cost, and Conversation. Conversation, Tokens, and User start hidden, and the browser you are working in remembers what you choose.

Trace status

Status Meaning

Completed

The run finished.

Error

The run itself failed. A run that finished with a failed tool call inside it is Completed, not Error.

Running

The run’s root span is still open, or the trace has no root span and spans arrived in the last 15 minutes.

Incomplete

The trace has no root span and no span arrived in the last 15 minutes, so the run’s record is partial.

A trace that only AI Gateway saw is never Running or Incomplete, because the gateway records each call after it serves it.

Cost

A trace’s cost covers the run’s model calls, in US dollars, as AI Gateway priced each call. A tool call carries no cost of its own.

  • An amount below $0.0001 reads as <$0.0001. Hover over it for the exact figure.

  • An asterisk means that only some of the run’s calls could be priced. Hover over it to see how many of them were.

  • A run whose calls couldn’t be priced at all, such as one on a model with no price, shows no cost.

For costs across runs, agents, and providers, see Analyze Cost and Usage.

Filter and narrow the list

Narrow the list by status from the status tabs: All, Errors, Running, or Incomplete. Each tab counts the traces that match it within everything else you have filtered on, and one tab applies at a time. The tabs don’t single out completed runs, so to find a run that recovered from a failed tool call, read the Status column.

Add the other filters from the filter menu:

  • Agent: One or more agents in this environment.

  • User: One or more people, matched on the authenticated email or on the end-user ID the runtime asserted. Type a value the list hasn’t seen yet to filter on it.

  • Conversation: One conversation ID.

  • MCP server: One or more MCP servers, matched on the servers whose tools the run called.

  • LLM provider: One or more of OpenAI, Anthropic, Google Gemini API, Bedrock, and Gemini Enterprise Agent Platform, matched on the type of provider that served the run’s model calls. OpenAI also matches OpenAI-compatible providers, which have no option of their own.

Each filter you add appears as a chip, which you can remove on its own.

Search by title or ID in the search box, up to 256 characters. The search matches trace titles, trace IDs, and conversation IDs.

Pick the time range with the range control. The presets are Last 1 hour, Last 24 hours, Last 7 days, and Last 30 days, and you can set a custom range.

Each filter accepts up to 20 values. Adding more makes that filter stop applying rather than reporting an error, so keep each one within the cap.

Zoom the timeline

The traces timeline counts the traces in each time bucket of the range and marks how many of them failed. Drag across it to zoom into a window, which also narrows the table, then zoom out to return to the range you came from. Bucket width follows the range, at about 60 buckets across it. Collapse the timeline when you don’t need it.

Read a trace

The expanded row holds the span waterfall beside one pane, which shows either the trace summary or the selected span’s detail. Selecting a span opens its detail, Trace summary in the trace’s toolbar switches back, and you can drag the divider between the waterfall and the pane to resize them. Close the pane to give the waterfall the full width, and drag the trace’s resize handle to make the trace itself taller or shorter.

The summary pane opens with the run’s status and title, over the rows that identify it: the agent that ran it, or the person who made it for a call that only AI Gateway saw, then its source, its conversation, and its trace ID. You can narrow the list to the row’s agent, user, or conversation without leaving the trace, and copy the conversation or trace ID. To read a title longer than the pane shows, click Show full prompt.

When the run failed and recorded its error, the summary pane also shows the error code and message it ended with. A run that recorded no error code shows Error in place of the code.

A details list gives:

  • Duration: How long the run took.

  • Tokens sent and Tokens received: The tokens the run’s model calls sent and received.

  • Cached: How many of the sent tokens were cached.

  • Cost: What the run’s model calls cost.

  • Model calls: How many model calls the run made.

  • Tool calls: How many tool calls the run made, with how many of them failed.

  • Spans: How many spans the run recorded.

  • Started: When the run started.

The Where the time went section breaks the duration into waiting for the model, running tools, agent orchestration, and other, listing only the parts that took at least a few milliseconds. The Tools used section lists each tool the run called, the MCP server it came from, and how many times the run called it. The Policies section lists each guardrail or data policy decision the run met, with the number of spans it applied to.

From the summary pane you can copy a link to the trace, and copying a link while a span is selected gives a link that reopens the trace with that span selected. For a trace that ran an agent, Open transcript in the trace’s toolbar opens that agent’s transcript at the turn holding the span you selected.

When a trace holds more spans than one read returns, a note says that only the first spans are listed. A trace that is no longer stored reports that the trace isn’t available.

Span kinds

Each span in the waterfall carries its kind:

Kind What it covers

AGENT

One agent run or turn.

SUBAGENT

A run handed off to a subagent.

LLM

One call to a model.

TOOL

One tool call.

GATEWAY

What AI Gateway saw of a call it served, nested under the agent’s own call.

COMPACTION

A pass where the agent reduced its own context. See Context compaction.

SPAN

Anything else the run recorded.

Filter the waterfall from the span kind menu, which labels these kinds a little differently (LLM call, Tool call, Subagent, Agent run, Gateway, and Compaction) and adds Errors and Blocked by policy. It offers no option for SPAN, and it lists Gateway, Compaction, and Blocked by policy only when the trace holds spans of that kind, so every option it offers carries a count of at least one. Switch the bars between Time, Tokens, and Cost to compare spans by duration, by token use, or by what they cost. The Tokens scale draws only the spans that used tokens, and the Cost scale only the spans whose calls were priced.

One span’s detail

Select a span to read what it recorded. The detail opens with where the span sits in the run, what it called, how long it took, and its span ID, over tabs that follow the span’s kind:

Kind What its tabs hold

LLM

Overview gives the tokens and cost of the call, broken out by token type, and the share of the trace’s cost it accounts for, followed by the gateway call that served it. Messages gives the system prompt, the last user message, the model’s response with the reason it stopped, and the tool calls that followed. Tools lists the tools the call offered the model. See Tools offered to a model call.

TOOL

Input & output gives the call arguments under gen_ai.tool.call.arguments and the result under gen_ai.tool.call.result, marked as either an error or a success, then the gateway call behind it, and the model call that made this call and the one that read its result.

SUBAGENT

Input & output gives what the agent handed the subagent and what came back, as for a TOOL span. When the subagent’s run is folded into the hand-off, Overview also summarizes that run, as for an AGENT span.

AGENT

Overview counts the failed spans beneath the run, reports how long it spent in model calls and in tool calls, and lists its slowest tool calls and its largest model calls.

GATEWAY

Gateway gives the caller, the route, the HTTP status, how long the call took, the upstream time and the gateway’s own overhead, and any guardrail that ran. For a model call it adds the provider and model, the tokens, and the reason the model stopped, and the span also has a Messages tab and, when the offered tools were recorded, a Tools tab. For an MCP call it adds the MCP method, the MCP server, and the tool.

COMPACTION

Compaction reports what the pass reclaimed. See Context compaction.

SPAN

Only the Attributes and Raw tabs.

Every span you can read also carries an Attributes tab, which lists the span’s OpenTelemetry attributes, and a Raw tab, which shows the stored span as JSON. A restricted span shows no tabs.

A failed span shows its error code and message on the tab that holds its content: Messages for a model call, Input & output for a tool call or a subagent hand-off, and Gateway for a gateway span. A call AI Gateway refused shows what it rejected, such as a budget block or a model that policy doesn’t allow.

Where a span names a resource that still exists, the detail links out to it: the agent, the LLM provider that served the model, or the managed MCP server behind the tool. A resource that no longer exists is named without a link.

Tools offered to a model call

A model can only call a tool it was offered, so a run that skipped a tool you expected is often a run that was never offered it. The Tools tab on a model call reports how many tools the call offered the model and how many of them it used, grouped by where each came from. Each group names how many tools it holds:

  • One group per MCP server the tools came from, named after the server, counting the tools called from it.

  • Subagents: The agents this one can hand off to. The call offers each as a tool, this group counts how many were called, and the summary line counts subagents separately from tools.

  • Not via gateway: The remaining tools the agent defines itself. This group is marked as defined by the agent rather than counting calls.

Each tool lists its description and, when you expand it, the input schema the model was given. A tool the model called on this turn carries a Called badge, which takes you to that tool call’s span. For a subagent, that badge takes you to the hand-off span.

A gateway span for a model call can carry a Tools tab too. It lists the same offered tools but reports only how many there were, with no used count, no per-group counts, and no Called badges.

A model call whose offered tools weren’t recorded has no Tools tab. If the offered tools can’t be retrieved, the tab reports that they couldn’t be loaded.

Policy decisions on a span

A span that met a guardrail or a data policy lists each decision with the policy’s name, its outcome, and why. The outcome is one of:

  • Allowed: The policy ran and let the content through.

  • Blocked: The policy stopped the content.

  • Masked: The policy rewrote part of the content.

  • Error: The policy couldn’t reach a verdict.

A guardrail decision also reports whether it ran on the request or the response. To follow a blocked request further, see Review Blocked Requests.

Runs whose root span hasn’t arrived

Spans can reach the service before the span that opened the run. Until that root span arrives, its spans sit under a stand-in row labeled Run in progress, or Run incomplete when the run stopped sending spans. A subagent run whose own root span is missing appears as subagent.

Restricted spans

A span whose content your permissions don’t reach shows as Restricted. Its shape stays visible, so you can still see where the time went: the span’s kind, timing, token use, the agent that emitted it, an error code, and an AI Gateway call’s HTTP status and upstream latency. Prompts, messages, tool arguments, and tool results stay hidden.

A trace whose content you can’t read shows Restricted in place of its title, and reports no user.

What traces leave out

AI Gateway does upkeep work that no one ran on purpose, and recording it would bury the calls you care about. A gateway record that holds only upkeep is dropped before storage, so the list never shows it. That covers MCP session setup and keepalives, tool and resource discovery, token issuance, and internal Agentic Data Plane calls. A record that holds a real model or tool call is kept whole, keepalives included.

Traces compared to transcripts and the audit log

The three surfaces answer different questions:

  • Traces give one row per run across every agent and AI Gateway, with timing, token use, and the span tree. Start here to find where a run spent its time or which call failed.

  • Transcripts give one agent’s conversation in full, turn by turn. Start there to read what the agent and the person said to each other. See how to read a transcript.

  • The audit log records authorization decisions, meaning who acted on which resource and whether the request was allowed. Start there for an access question. See how to review the audit log.

A trace and a transcript cover the same run from different angles. For a trace that ran an agent, Open transcript in the trace’s toolbar takes you from the selected span to the matching turn of the transcript.

Next steps