Agentic Data Plane

How Observability Works

Redpanda provides complete observability and governance for AI agents through automated transcript capture. Every agent execution, from simple tool calls to complex multi-agent, multi-turn workflows, generates a record stored on Redpanda’s log. For an agent that records transcripts, the structure of every execution (its reasoning steps, tool invocations, model interactions, and data flows) is captured with 100% sampling and no gaps. A per-agent transcript recording mode sets how much each agent records, including whether the full message content of each step (the prompts sent to the model and the responses it returns) is kept. It defaults to recording everything, so if an agent handles regulated or sensitive traffic, set its recording mode to record metadata only. The input and output capture settings on an LLM provider apply to the AI Gateway’s own trace data, not to what an agent’s transcript shows. See Configure an LLM Provider.

With transcripts, organizations gain the ability to debug agent behavior, identify performance bottlenecks, meet regulatory compliance requirements, and maintain accountability for AI-driven decisions. Transcripts use OpenTelemetry standards and Raft-based consensus for correctness, establishing a trustworthy foundation for agent governance.

After reading this page, you will be able to:

  • Explain how transcripts and spans capture execution flow

  • Interpret transcript structure for debugging and monitoring

  • Distinguish between transcripts and audit logs

What are transcripts

A transcript records the complete execution of an agentic behavior from start to finish. It captures every step (across multiple agents, tools, models, and services) in a single, traceable record. The AI Gateway and every managed agent and MCP server in your Agentic Data Plane automatically emit OpenTelemetry traces to a topic called redpanda.otel_traces. Redpanda’s immutable distributed log stores these traces. Self-managed agents are the exception: they export their own spans through the environment’s authenticated OTLP endpoint. See Self-Managed Agent Telemetry Reference.

Transcripts capture:

  • Tool invocations and results

  • Agent reasoning steps

  • Data processing operations

  • External API calls

  • Error conditions

  • Performance metrics

With an agent set to record metadata only, transcripts hold the execution structure and metadata but not the raw prompt and response bodies. The underlying storage uses a distributed log built on Raft consensus (with TLA+ proven correctness), giving transcripts a trustworthy record for governance, debugging, and performance analysis.

Traces and spans

OpenTelemetry traces provide a complete picture of how a request flows through your system:

  • A trace represents the entire lifecycle of a request (for example, a tool invocation from start to finish).

  • A span represents a single unit of work within that trace (such as a data processing operation or an external API call).

  • A trace contains one or more spans organized hierarchically, showing how operations relate to each other.

Agent transcript hierarchy

Agent executions create a hierarchy of spans that reflect how agents process requests. Understanding this hierarchy helps you interpret agent behavior and identify where issues occur.

Agent span types

Agent transcripts contain these span types:

Span Type Description Use To

invoke_agent

Root span of one managed-agent invocation, following the OpenTelemetry agent invocation semantic conventions. Covers the entire request, from receiving it through the reasoning loop, tool calls, and the final response. When an agent delegates to a subagent, the delegated work appears as a nested invoke_agent span. Self-managed agents emit it from their own instrumentation.

Measure total request duration, identify slow agent invocations, and trace delegation between an agent and its subagents.

chat {model}

One LLM call, named after the model (for example, chat gpt-5.2 or chat claude-sonnet-4-6). The provider appears as an attribute (gen_ai.provider.name), not in the span name. Multiple chat spans appear when the agent iterates through its reasoning loop.

Identify which model was called, measure LLM response time, track reasoning iterations, and debug LLM API errors.

execute_tool {tool}

One tool execution as recorded by the managed agent runtime (for example, execute_tool get_quote).

Measure tool execution time and identify slow tools.

In raw trace data, spans from the managed agent runtime carry the resource attribute service.name: ai-agent. That value identifies the emitter, not a span type. Raw trace data can also hold a tools/call {tool} span for the same tool execution as seen by the AI Gateway’s MCP proxy. Transcripts don’t include gateway spans.

Typical agent execution flow

A simple agent request creates this hierarchy:

invoke_agent (6.65 seconds)
├── chat gpt-5.2 (4.1 seconds)
├── execute_tool get_quote (0.4 seconds)
└── chat gpt-5.2 (2.1 seconds)

This hierarchy shows that the two LLM calls (6.2 seconds combined) account for most of the total agent invocation time (6.65 seconds), revealing the bottleneck in this execution flow.

Trace layers and scope

The trace an agent exports contains more than one layer of instrumentation, from HTTP transport through application logic to outbound calls. The scope.name field in each span identifies which instrumentation layer created that span. A transcript keeps only the agent-level spans (invoke_agent, chat, and execute_tool). The HTTP spans stay in the raw trace data and don’t appear in the Transcripts view.

Instrumentation layers

An agent’s exported trace includes these layers:

Layer Scope Name Purpose

HTTP Server

go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

HTTP transport layer receiving requests. Shows request/response sizes, status codes, client addresses, and network details.

AI SDK (Agent)

github.com/redpanda-data/ai-sdk-go/plugins/otel

Agent application logic. Shows agent invocations, LLM calls, tool executions, conversation IDs, token usage, and model details. Includes gen_ai.* semantic convention attributes.

HTTP Client

go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Outbound HTTP calls from agent to MCP servers. Shows target URLs, request methods, and response codes.

How layers connect

Layers connect through parent-child relationships in a single trace. Only the AI SDK spans appear in the transcript:

ai-agent-http-server (HTTP Server layer)
└── invoke_agent customer-support-agent (AI SDK layer)
    ├── chat gpt-5-nano (AI SDK layer, LLM call 1)
    ├── execute_tool get_order_status (AI SDK layer)
    │   └── HTTP POST (HTTP Client layer)
    └── chat gpt-5-nano (AI SDK layer, LLM call 2)

The request flow demonstrates:

  1. HTTP request arrives at agent

  2. The agent invocation starts

  3. Agent makes first LLM call to decide what to do

  4. Agent executes tool, making HTTP call to MCP server

  5. Agent makes second LLM call with tool results

  6. Response returns through HTTP layer

Key attributes by layer

Different layers expose different attributes:

HTTP Server/Client layer (following OpenTelemetry semantic conventions for HTTP):

  • http.request.method, http.response.status_code

  • server.address, url.path, url.full

  • network.peer.address, network.peer.port

  • http.request.body.size, http.response.body.size

AI SDK layer (following OpenTelemetry semantic conventions for generative AI):

  • gen_ai.operation.name: Operation type (invoke_agent, chat, execute_tool)

  • gen_ai.conversation.id: Links spans to the same conversation session. A conversation may include multiple agent invocations (one per user request). Each invocation creates a separate trace that shares the same conversation ID. Every span an agent exports must carry this attribute; spans without it don’t associate with any transcript.

  • gen_ai.agent.name: The agent’s name

  • gen_ai.provider.name, gen_ai.request.model: LLM provider and model

  • gen_ai.usage.input_tokens, gen_ai.usage.output_tokens: Token consumption

  • gen_ai.tool.name, gen_ai.tool.call.arguments: Tool execution details

  • gen_ai.input.messages, gen_ai.output.messages: Full LLM conversation context

The scope.name field identifies which instrumentation layer created each span.

Understand the transcript structure

Each span captures a unit of work. Here’s what a tool execution recorded by the managed agent runtime looks like:

{
    "traceId": "71cad555b35602fbb35f035d6114db54",
    "spanId": "43ad6bc31a826afd",
    "parentSpanId": "ed45544a7d7b08d4",
    "name": "execute_tool get_quote",
    "attributes": [
        {"key": "gen_ai.operation.name", "value": {"stringValue": "execute_tool"}},
        {"key": "gen_ai.tool.name", "value": {"stringValue": "get_quote"}},
        {"key": "gen_ai.conversation.id", "value": {"stringValue": "4c1f0b9e-2d7a-4f3e-9a51-8b6d2e0c7f14"}}
    ],
    "startTimeUnixNano": "1765198415253280028",
    "endTimeUnixNano": "1765198415653280028",
    "instrumentationScope": {"name": "github.com/redpanda-data/ai-sdk-go/plugins/otel"},
    "status": {"code": 0, "message": ""}
}

  • traceId links all spans in the same request.

  • spanId uniquely identifies this span.

  • parentSpanId links this span to its parent, here the invoke_agent span for the invocation. Spans that share a traceId and parent links reconstruct the complete operation.

  • name identifies the operation and, for a tool execution, the tool.

  • instrumentationScope.name identifies which layer created the span.

  • attributes contain operation-specific metadata. gen_ai.conversation.id ties the span to its conversation.

  • status.code indicates success (0) or error (2). When an operation fails, status.message carries the error.

How Redpanda stores trace data

The redpanda.otel_traces topic stores OpenTelemetry spans using Redpanda’s Schema Registry wire format, with a custom Protobuf schema named redpanda.otel_traces-value that follows the OpenTelemetry Protocol (OTLP) specification. Spans include attributes following OpenTelemetry semantic conventions for generative AI, such as gen_ai.operation.name and gen_ai.conversation.id. The schema is automatically registered in the Schema Registry with the topic, so Kafka clients can consume and deserialize trace data correctly.

Redpanda manages both the redpanda.otel_traces topic and its schema automatically. If you delete either the topic or the schema, they are recreated automatically. However, deleting the topic permanently deletes all trace data, and the topic comes back empty. Do not produce your own data to this topic. It is reserved for OpenTelemetry traces. To send traces from a self-managed agent, use the environment’s authenticated OTLP endpoint instead of producing to a topic; see Self-Managed Agent Telemetry Reference.

Topic configuration and lifecycle

The redpanda.otel_traces topic has a predefined retention policy. Configuration changes to this topic are not supported. If you modify settings, Redpanda reverts them to the default values.

The topic persists even after all agents and MCP servers are deleted, allowing you to retain historical trace data for analysis.

Transcripts may contain sensitive information from your tool inputs and outputs. Review the data in transcripts before sharing or exporting to external systems.

Read access to transcript data is governed by the transcript permissions rather than by topic ACLs on redpanda.otel_traces. Among the built-in roles provisioned today, only Admin holds them, and every built-in policy template from Read only up also grants them. See Transcript permissions.

How Redpanda stores audit events

When audit recording is enabled for a deployment, Agentic Data Plane structures audit events according to the Open Cybersecurity Schema Framework (OCSF), an open, vendor-neutral schema for security events. OCSF gives events from different subsystems a shared vocabulary. Agentic Data Plane uses the OCSF API Activity event class for authorization decisions on API operations and the Entity Management event class for configuration changes. This keeps fields such as actor, resource, activity, and outcome consistent across the audit log.

OCSF standardizes what an event means; it does not define how events are stored or exported. In Agentic Data Plane, use the Audit Log view to review these records.

Redpanda keeps recent events in streaming storage and archives them in an Apache Iceberg-backed historical store in object storage. The Audit Log view presents both as one time-ordered history, so you can investigate new and older activity from the same view.

Captured request, response, or configuration data may contain sensitive information. Agentic Data Plane redacts supported secret and sensitive configuration fields. Identities and other audit attributes may remain visible, and captured data may be truncated or omitted. Review an event before sharing it outside your organization.

The Audit Log view returns events for your Agentic Data Plane environment. Access requires an audit log permission. See Audit log permissions.

Transcripts compared to audit logs

Transcripts and audit logs serve different but complementary purposes.

Transcripts provide:

  • A complete, immutable record of every execution step, stored on Redpanda’s distributed log with no gaps

  • Hierarchical view of request flow through your system (parent-child span relationships)

  • Detailed timing information for performance analysis

  • Ability to reconstruct execution paths and identify bottlenecks

Transcripts are optimized for execution-level observability and governance.

The audit log provides:

  • The user accountable for an action, plus the agent or caller that acted for the user when available

  • The affected resource, operation, subsystem, and time

  • The allowed, masked, partial, or denied outcome and the policies that decided it, including any guardrail or data policy that ran

For access and accountability questions, use the audit log. For questions about an agent’s model turns, tool calls, timing, or errors, use transcripts. An investigation may use both views.

Next steps