Trace an Agent Run
A trace is the record of one agent run, broken into spans such as a model call, a tool call, or a subagent handoff. A call that only AI Gateway handled gets a trace too. Use traces to find which LLM call, tool call, subagent, or policy decision made a run slow, expensive, or wrong, without opening each agent separately.
After reading this page, you will be able to:
-
Filter traces by agent, status, user, source, conversation, and time range
-
Expand a trace into its span waterfall and read one span’s detail
-
Choose traces, transcripts, or the audit log for an investigation
For what traces and spans are, see How Observability Works.
Prerequisites
Your account needs these permissions:
| Permission | What it covers |
|---|---|
|
Open the traces list and its timeline |
|
Expand one trace and read its spans |
|
Read the content of spans that no agent or person owns, and of other people’s AI Gateway calls |
Without dataplane_adp_trace_list the page reports that permission was denied and hides the timeline. The built-in Read only template, and every template built on top of it, grants all three. To grant them with an access policy, name Action::"Trace.list", Action::"Trace.get", and Action::"Trace.read_content". See Trace permissions.
Content you can reach without dataplane_adp_trace_content_read follows who owns it:
-
An agent’s spans need
dataplane_adp_transcript_geton that agent, ordataplane_adp_transcript_listfor a call made outside any conversation. -
Your own AI Gateway calls are yours to read.
A span you can’t read shows as Restricted, and a trace whose content you can’t read shows Restricted in place of its title. See Restricted spans.
Open the traces list
-
Open Agents in the sidebar, then click Traces.
-
Review the most recent traces. The default time range is Last 1 hour, and the newest trace is first.
-
Click a row to expand it into its span waterfall.
The table shows these columns:
| Column | Description |
|---|---|
Started |
When the trace started, as a relative time. Hover over it to see the UTC timestamp. |
Status |
The outcome of the run. See Trace status. A trace that holds a failed tool call carries a marker beside its status. |
Trace |
The trace’s title, taken from the first line of the run’s opening message, with the trace ID underneath. A trace that only AI Gateway saw carries a Gateway badge. |
Conversation |
The conversation the run belongs to. Click it to open that conversation’s transcript, while the agent still exists. To narrow the list to one conversation instead, use the Conversation filter. |
Agent |
The agent that ran. The name links to the agent while the agent still exists. A dash means no agent ran and only AI Gateway saw the trace. |
User |
The authenticated person who made the call, or the end user the runtime asserted. |
Duration |
How long the run took. A running trace counts up instead. |
LLM |
How many LLM calls the run made. |
Tools |
How many tool calls the run made. |
Tokens |
Total tokens the run used, as the provider reported them. Hover over it for the exact count. |
Sort on any column except Conversation by clicking its header.
Trace status
| Status | Meaning |
|---|---|
Completed |
The run finished. |
Error |
The run itself failed. A run that finished with a failed tool call inside it is Completed, not Error. |
Running |
The run’s root span is still open, or the trace has no root span and spans arrived in the last 15 minutes. |
Incomplete |
The trace has no root span and no span arrived in the last 15 minutes, so the run’s record is partial. |
A trace that only AI Gateway saw is never Running or Incomplete, because the gateway records each call after it serves it.
Filter and narrow the list
Filter the list by:
-
Agent: One or more agents in this environment.
-
Status: One or more of Completed, Error, Running, and Incomplete, plus Has failed calls for a run that holds at least one failed tool call.
-
User: One or more people, matched on the authenticated email or on the end-user ID the runtime asserted. Type a value the list hasn’t seen yet to filter on it.
-
Source: Agent run for a run an agent made, or Gateway only for calls that only AI Gateway saw.
-
Conversation: One conversation ID.
Search by title, trace ID, or conversation ID in the search box, up to 256 characters.
Pick the time range with the range control. The presets are Last 1 hour, Last 24 hours, Last 7 days, and Last 30 days, and you can set a custom range.
Each filter accepts up to 20 values. Adding more makes that filter stop applying rather than reporting an error, so keep each one within the cap.
Zoom the timeline
The traces timeline counts the traces in each time bucket of the range and marks how many of them failed. Drag across it to zoom into a window, which also narrows the table, then zoom out to return to the range you came from. Bucket width follows the range, at about 60 buckets across it. Collapse the timeline when you don’t need it.
Read a trace
The expanded row holds the span waterfall beside one pane, which shows either the trace summary or the selected span’s detail. Selecting a span opens its detail, Trace summary in the trace’s toolbar switches back, and you can drag the divider between the waterfall and the pane to resize them.
The summary pane reports what the run did in one sentence, lists the trace and conversation IDs for copying, and breaks the duration down under Where the time went, into waiting for the model, running tools, agent orchestration, and other. Any guardrail or data policy decision the run met is listed under Policies.
From the summary pane you can copy a link to the trace, and copying a link while a span is selected gives a link that reopens the trace with that span selected. For a trace that ran an agent, Open transcript in the trace’s toolbar opens that agent’s transcript at the turn holding the span you selected.
When a trace holds more spans than one read returns, a note says that only the first spans are listed. A trace that is no longer stored reports that the trace isn’t available.
Span kinds
Each span in the waterfall carries its kind:
| Kind | What it covers |
|---|---|
|
One agent run or turn. |
|
A run handed off to a subagent. |
|
One call to a model. |
|
One tool call. |
|
What AI Gateway saw of a call it served, nested under the agent’s own call. |
|
A pass where the agent reduced its own context. See Context compaction. |
|
Anything else the run recorded. |
Filter the waterfall from the span kind menu, which labels these kinds a little differently (LLM call, Tool call, Subagent, Agent run, Gateway, and Compaction) and adds Errors and Blocked by policy. It offers no option for SPAN, and it lists Gateway, Compaction, and Blocked by policy only when the trace holds spans of that kind, so every option it offers carries a count of at least one. Switch the bars between Time and Tokens to compare spans by duration or by token use.
One span’s detail
Select a span to read what it recorded. An LLM span shows the system prompt, the last user message, and the model’s response. A tool span shows the call arguments under gen_ai.tool.call.arguments and the result under gen_ai.tool.call.result, marked as either an error or a success. A failed span shows its error code, and a call AI Gateway refused shows what it rejected, such as a budget block or a model that policy doesn’t allow. An agent span counts the failed spans beneath it.
Where a span names a resource that still exists, the detail links out to it: the agent, the LLM provider that served the model, or the managed MCP server behind the tool. A resource that no longer exists is named without a link.
Policy decisions on a span
A span that met a guardrail or a data policy lists each decision with the policy’s name, its outcome, and why. The outcome is one of:
-
Allowed: The policy ran and let the content through.
-
Blocked: The policy stopped the content.
-
Masked: The policy rewrote part of the content.
-
Error: The policy couldn’t reach a verdict.
A guardrail decision also reports whether it ran on the request or the response. To follow a blocked request further, see Review Blocked Requests.
Runs whose root span hasn’t arrived
Spans can reach the service before the span that opened the run. Until that root span arrives, its spans sit under a stand-in row labeled Run in progress, or Run incomplete when the run stopped sending spans. A subagent run whose own root span is missing appears as subagent.
Restricted spans
A span whose content your permissions don’t reach shows as Restricted. Its shape stays visible, so you can still see where the time went: the span’s kind, timing, token use, the agent that emitted it, an error code, and an AI Gateway call’s HTTP status and upstream latency. Prompts, messages, tool arguments, and tool results stay hidden.
A trace whose content you can’t read shows Restricted in place of its title, and reports no user.
What traces leave out
AI Gateway does upkeep work that no one ran on purpose, and recording it would bury the calls you care about. A gateway record that holds only upkeep is dropped before storage, so the list never shows it. That covers MCP session setup and keepalives, tool and resource discovery, token issuance, and internal Agentic Data Plane calls. A record that holds a real model or tool call is kept whole, keepalives included.
Traces compared to transcripts and the audit log
The three surfaces answer different questions:
-
Traces give one row per run across every agent and AI Gateway, with timing, token use, and the span tree. Start here to find where a run spent its time or which call failed.
-
Transcripts give one agent’s conversation in full, turn by turn. Start there to read what the agent and the person said to each other. See See What Your Agent Did.
-
The audit log records authorization decisions, meaning who acted on which resource and whether the request was allowed. Start there for an access question. See Review the Audit Log.
A trace and a transcript cover the same run from different angles. For a trace that ran an agent, Open transcript in the trace’s toolbar takes you from the selected span to the matching turn of the transcript.
