# Redpanda Documentation - Full Markdown Export > This file contains all documentation pages in markdown format for AI agent consumption. > Generated from 2101 pages on 2026-08-13T21:30:29.566Z > Site: https://docs.redpanda.com ## About This Export This export includes only the **latest version** of each component's documentation: - **Home**: version - **Data Platform**: version - **Self-Managed**: version - **Streaming**: version 26.2 - **shared**: version - **search**: version - **Labs**: version - **Cloud**: version - **Connect**: version - **Agentic Data Plane**: version ### AI-Friendly Documentation Formats We provide multiple formats optimized for AI consumption: - **https://docs.redpanda.com/llms.txt**: Curated overview following the llms.txt standard - start here for a quick introduction - **https://docs.redpanda.com/llms-full.txt**: Complete documentation export (this file) - comprehensive reference with all pages - **Component-specific exports**: Focused documentation for individual products: - `https://docs.redpanda.com/home-full.txt`: Home - `https://docs.redpanda.com/data-platform-full.txt`: Data Platform - `https://docs.redpanda.com/self-managed-full.txt`: Self-Managed - `https://docs.redpanda.com/streaming-full.txt`: Streaming - `https://docs.redpanda.com/shared-full.txt`: shared - `https://docs.redpanda.com/search-full.txt`: search - `https://docs.redpanda.com/labs-full.txt`: Labs - `https://docs.redpanda.com/cloud-data-platform-full.txt`: Cloud - `https://docs.redpanda.com/connect-full.txt`: Connect - `https://docs.redpanda.com/agentic-data-plane-full.txt`: Agentic Data Plane - **Individual markdown pages**: Each HTML page has a corresponding .md file (e.g., `/docs/page.html` → `/docs/page.md`) ### Accessing Versioned Content For components with versioned documentation (like Redpanda Self-Managed), older versions can be accessed by replacing the version segment in the URL: - Latest: `https://docs.redpanda.com/current/page-path` - Specific version: `https://docs.redpanda.com/24.3/page-path`, `https://docs.redpanda.com/25.1/page-path`, etc. Available versioned components: streaming --- # Page 1: Use the Agentic Data Plane CLI **URL**: https://docs.redpanda.com/agentic-data-plane/cli.md --- # Use the Agentic Data Plane CLI > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Use the Agentic Data Plane CLI latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/cli/pages/index.adoc description: Manage Agentic Data Plane resources from the terminal with rpk ai. Install the CLI, connect it to an AI Gateway, and script LLM providers, MCP servers, OAuth, and agents. page-topic-type: how-to personas: platform_engineer, agent_builder learning-objective-1: Install the Agentic Data Plane CLI and connect it to an AI Gateway learning-objective-2: "Manage Agentic Data Plane resources from the terminal: LLM providers, MCP servers, OAuth providers and clients, and agents" learning-objective-3: Format command output for scripts and pipelines page-git-created-date: "2026-06-26" page-git-modified-date: "2026-08-10" --- The Agentic Data Plane command-line interface, `rpk ai`, manages [Agentic Data Plane](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#agentic-data-plane) resources from your terminal. It drives the same control surface as the Agentic Data Plane UI: [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) providers, Model Context Protocol (MCP) servers, OAuth providers and clients, and agents. Because every action is a command, you can script it and run it in continuous integration. The Agentic Data Plane CLI, also called the Redpanda AI CLI, runs as an [`rpk`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) plugin, so you install and run it through your existing `rpk` setup. After reading this page, you will be able to: - Install the Agentic Data Plane CLI and connect it to an AI Gateway - Manage Agentic Data Plane resources from the terminal: LLM providers, MCP servers, OAuth providers and clients, and agents - Format command output for scripts and pipelines ![rpk ai workflow. Install the rpk ai plugin, sign in and select an Agentic Data Plane environment, then manage LLM providers, MCP servers, OAuth resources, and agents; list or call MCP tools and send A2A messages; or launch supported coding agents through AI Gateway. These workflows share the selected identity and environment.](_images/rpk-ai-workflow.svg) Figure 1. `rpk ai` reuses one authenticated environment across resource, protocol, and agent workflows ## [](#install-the-cli)Install the CLI The Agentic Data Plane CLI is an `rpk` managed plugin. Install `rpk` first (see [Install or Update rpk](https://docs.redpanda.com/agentic-data-plane/reference/rpk-install/)), then install the plugin: ```bash rpk ai install ``` Update the plugin later with [`rpk ai upgrade`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-upgrade/), and remove it with [`rpk ai uninstall`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-uninstall/). > 📝 **NOTE** > > Upgrading `rpk` does not upgrade the plugin. The plugin is a separate binary that stays at its installed version, even across `rpk` upgrades, until you run `rpk ai upgrade`. New commands and flags in the reference marked "introduced in ai version X" require the plugin at that version or later. ## [](#connect-to-an-ai-gateway)Connect to an AI Gateway The `rpk ai` command is self-contained: it has its own login and its own environment selection, independent of any `rpk cloud` session. You sign in once, choose an Agentic Data Plane environment, and `rpk ai` targets that environment’s AI Gateway for every command. 1. Sign in to Agentic Data Plane. This runs an OAuth device-authorization flow in your browser, caches credentials in `~/.rpai/credentials` (readable only by you), then lists the Agentic Data Plane environments in your organization so you can pick one: ```bash rpk ai auth login ``` 2. List the environments available to you, and switch the active one by name or ID: ```bash rpk ai env list rpk ai env use ``` In these commands, `` is the name or ID of an Agentic Data Plane environment. `rpk ai env use` switches the active environment, pointing the CLI at that environment’s AI Gateway. Inspect the resolved environment and token state at any time with `rpk ai env show` and `rpk ai auth status`. 3. Verify the connection: ```bash rpk ai llm-provider list ``` For the full setup walkthrough, including token expiry behavior and the supported environment variables, see [Use `rpk ai` for local development](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/#authenticate-with-rpk-ai). To target a different AI Gateway URL for a single command without switching environments, pass `--rpai-endpoint`: ```bash rpk ai --rpai-endpoint https://aigw..clusters.rdpa.co llm-provider list ``` This overrides the selected environment’s AI Gateway URL for that one command; the flag is not bound to an environment variable. For a local or manual gateway you use repeatedly, define it once as an environment with `rpk ai env add`: ```bash rpk ai env add local --ai-gateway-url http://localhost:8090 --auth-mode none rpk ai env use local ``` ## [](#manage-resources)Manage resources Each Agentic Data Plane resource has its own command group, and each group supports the standard `create`, `get`, `list`, `update`, and `delete` actions. The model catalog is read-only. | Command | Manages | | --- | --- | | rpk ai llm-provider | LLM providers (OpenAI, Anthropic, Google, AWS Bedrock, and OpenAI-compatible endpoints) | | rpk ai mcp-server | MCP servers and their tools | | rpk ai oauth-provider | OAuth providers for user-delegated MCP connections | | rpk ai oauth-client | OAuth clients that external tools use to reach an MCP server | | rpk ai agent | Agents in the agent registry | | rpk ai model | The model catalog (read-only discovery) | For example, to create and inspect an LLM provider: ```bash rpk ai llm-provider create --name openai --type openai --api-key-ref OPENAI_API_KEY rpk ai llm-provider get openai ``` In this command, `--api-key-ref` names a secret already stored in Agentic Data Plane. The CLI never takes a raw API key, so secrets stay out of your shell history and out of any file you commit. To create, update, and delete resources declaratively from YAML manifests instead of imperative flags, see [Manage Resources with GitOps](gitops/). ## [](#format-command-output)Format command output Every `list` and `get` command honors `-o` (`--format`): | Format | Use | | --- | --- | | table | Default. Human-readable columns. | | wide | Table with extra columns. | | json | Machine-readable JSON for scripts and jq. | | yaml | YAML. A get -o yaml dump is a complete manifest you can edit and reapply. | | markdown | Pipe-friendly tables for documents. | Set a default for the session with the `RPAI_FORMAT` environment variable. The `-o` flag always wins when both are set. ## [](#next-steps)Next steps - [Manage Resources with GitOps](gitops/) - [Configure an LLM Provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) - [rpk ai](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) --- # Page 2: Manage Resources with GitOps **URL**: https://docs.redpanda.com/agentic-data-plane/cli/gitops.md --- # Manage Resources with GitOps > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Manage Resources with GitOps latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: gitops page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: gitops.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/cli/pages/gitops.adoc description: "Keep Agentic Data Plane resources in Git and reconcile them with rpk ai apply and diff: declarative create-or-update, drift detection, and CI gating." page-topic-type: how-to personas: platform_engineer, agent_builder learning-objective-1: Export an Agentic Data Plane resource to a YAML manifest you can commit to Git learning-objective-2: Reconcile resources with apply, and preview changes with diff learning-objective-3: Gate continuous integration on configuration drift page-git-created-date: "2026-06-26" page-git-modified-date: "2026-07-30" --- GitOps keeps the definition of your [Agentic Data Plane](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#agentic-data-plane) resources in Git and reconciles the live environment toward those files, instead of running imperative `create`, `update`, and `delete` commands by hand. With the [Agentic Data Plane CLI](../), `rpk ai apply -f` creates what is missing and updates what has drifted, and `rpk ai diff -f` is a read-only dry-run that reports what `apply` would change. In these commands, `` is one of the resource command groups that support GitOps: - LLM providers (`rpk ai llm-provider`) - MCP servers (`rpk ai mcp-server`) - OAuth providers (`rpk ai oauth`) - OAuth clients (`rpk ai oauth-client`) - Agents (`rpk ai agent`) After reading this page, you will be able to: - Export an Agentic Data Plane resource to a YAML manifest you can commit to Git - Reconcile resources with apply, and preview changes with diff - Gate continuous integration on configuration drift ## [](#prerequisites)Prerequisites - The [Agentic Data Plane CLI installed and connected](../) to an Agentic Data Plane environment. - Permission to create and update the resources you manage. The reconcile commands call the same APIs as `create` and `update`. See [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/). - Any secrets your manifests reference already created in Agentic Data Plane. Manifests name secrets by reference, for example, `OPENAI_API_KEY`; the CLI does not create secrets. ## [](#how-apply-and-diff-reconcile)How apply and diff reconcile A manifest is plain resource YAML: the same shape a `get -o yaml` dump produces. The CLI compares each manifest against the live resource of the same name and resolves one of three outcomes: Create No resource of that name exists, so `apply` creates it. Update The resource exists and a field in the manifest differs from the live value, so `apply` updates the differing fields. Unchanged The resource exists and every field the manifest names already matches. The reconcile rules are deliberate, and they are not the same as a full-object replace: Presence drives updates A field that is present in the manifest and differs from the live resource is updated. A field you omit is left untouched. To clear a field, write it explicitly with an empty or zero value. Collections replace wholesale Lists, maps, and provider or backend variants are replaced as a unit, not merged element by element. Create-only fields are immutable A field that can be set only at creation time, such as an LLM provider’s `type`, an MCP server’s backend kind, an OAuth provider’s `client_id`, or an agent’s managed-or-self-managed kind, cannot change on an existing resource. Changing one is an error that tells you to delete and recreate the resource. Request-only fields are instructions, not state A few fields tell the gateway to do something rather than describe what a resource is. An MCP server’s `remote.user_oauth.automatic_setup` asks the gateway to set up OAuth for the server itself instead of attaching a provider by name. Set it in a manifest when you want automatic setup, and leave `provider_name` out: on an existing server an empty `provider_name` preserves the provider already attached, and setting both is an error. Omitting `automatic_setup` leaves an existing attachment untouched, so an unrelated edit such as a scopes change never re-runs OAuth setup. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). Secrets stay by reference Manifests reference secrets by name, for example, `api_key_ref` and `client_secret_ref`, and never contain secret values, so a manifest is safe to commit to Git. The `apply` command does not delete resources that are absent from your manifests; there is no prune. The `diff` command checks only the fields a manifest names, so it does not detect a resource that exists in the environment but is missing from your manifests, nor drift in a field a manifest omits. ## [](#export-a-resource-to-a-manifest)Export a resource to a manifest Start from a live resource so the manifest is complete. Dump it to YAML and redirect it to a file: ```bash rpk ai llm-provider get openai -o yaml > openai.yaml ``` A dumped OpenAI provider looks like this, ready to commit: ```yaml '@type': type.googleapis.com/redpanda.api.adp.v1alpha1.LLMProvider created_at: "2026-06-20T10:15:30Z" display_name: OpenAI enabled: true name: openai openai_config: api_key_ref: OPENAI_API_KEY provider_models: - name: gpt-4o - name: gpt-4o-mini type: LLM_PROVIDER_TYPE_OPENAI updated_at: "2026-06-20T10:15:30Z" url: https://openai.aigw.d0example1cluster234.clusters.rdpa.co/openai/v1 ``` The `@type` line records the resource kind. It is optional when you apply with a resource command, because `rpk ai llm-provider apply` already implies the kind, but keeping it lets a reader and any validator know what the file describes. > 📝 **NOTE** > > The `created_at`, `updated_at`, and `url` fields are server-managed and read-only. The `apply` command ignores them, so you can leave them in the file or strip them. The `api_key_ref` value is a reference to a secret, not the key itself. ## [](#preview-changes-with-diff)Preview changes with diff Edit the manifest, then preview the effect before you touch the environment. For example, change the display name: ```yaml display_name: OpenAI (production) ``` Run `diff` to see the plan: ```bash rpk ai llm-provider diff -f openai.yaml ``` ```text ~ openai (update: display_name) ``` The `diff` command marks each manifest with one of three symbols and changes nothing: | Symbol | Meaning | | --- | --- | | + | The apply command would create the resource. | | ~ | The apply command would update the resource. The changed fields follow in parentheses. | | = | The resource already matches; apply would leave it unchanged. | The `diff` command exits with a non-zero status when any change is pending, and zero when the environment already matches every manifest. That exit code is what lets continuous integration gate on drift. ## [](#apply-changes)Apply changes Reconcile the environment toward the manifest: ```bash rpk ai llm-provider apply -f openai.yaml ``` The `apply` command prints one line per manifest as it works: ```text updated openai (display_name) ``` A first-time apply of a resource that does not yet exist prints `created openai` instead, and a manifest that already matches the environment prints `unchanged openai`. The CLI plans every manifest before it changes anything, so a malformed manifest aborts the run before any write. If a later write fails, the lines already printed tell you exactly what was applied. ## [](#apply-many-manifests-at-once)Apply many manifests at once The `-f` flag is repeatable and accepts a file, a directory, or a stream: ```bash # A directory of manifests; the CLI reads every .yaml and .yml file, sorted by name. rpk ai mcp-server apply -f ./manifests/ # Several paths in one run. rpk ai llm-provider apply -f openai.yaml -f anthropic.yaml # Standard input, for piping a manifest from another tool. rpk ai oauth apply -f - ``` A single file can hold more than one manifest. Separate documents with a line containing only `---`. ## [](#gate-continuous-integration-on-drift)Gate continuous integration on drift Because `diff` exits non-zero when the environment differs from your manifests, a continuous-integration job can fail the build whenever the live environment has drifted from Git: ```bash # Fails the job if an apply would change anything. rpk ai llm-provider diff -f ./llm/ rpk ai mcp-server diff -f ./mcp/ ``` A common pipeline runs `diff` on a pull request to preview changes, then runs `apply` after the merge to roll them out: ```bash # Deploy step, after merge to the main branch. rpk ai llm-provider apply -f ./llm/ rpk ai mcp-server apply -f ./mcp/ ``` ## [](#manifests-are-declarative)Manifests are declarative A manifest describes the full intended state, so an omitted field means zero, not the convenience default that `create` fills in. A manifest that omits `enabled` creates a disabled resource, and `diff` then reports no drift, because the disabled state matches the manifest. To avoid surprises: - Start from a `get -o yaml` dump, which is already complete, rather than hand-writing a manifest from scratch. - Set `enabled: true` explicitly when you want an active resource. One subtlety follows from this: a dump of an already-disabled resource omits `enabled`, because `false` is the field’s zero value and the dump omits zero values. Reapplying that dump elsewhere also produces a disabled resource. ## [](#troubleshooting)Troubleshooting | Symptom | Resolution | | --- | --- | | The apply or diff command reports that a field is immutable and tells you to delete and recreate the resource. | You changed a create-only field, such as an LLM provider’s type or an OAuth provider’s client_id. Restore the original value, or delete the resource and recreate it from the new manifest. | | The apply or diff command fails to decode the manifest and mentions unknown or misspelled fields. | A key in the manifest is not a field of the resource. The CLI rejects unknown keys rather than dropping them silently. Fix the key. Start from a get -o yaml dump to get the exact field names. | | The diff command reports no drift, but you know the environment has extra resources. | The diff and apply commands never prune. They act only on the resources your manifests name. Delete unwanted resources with rpk ai delete. | | A field you changed in the environment keeps coming back after apply. | The apply command overwrites a field only when the manifest names it and the value differs. If the change is not in your manifest, add it, then reapply. | ## [](#next-steps)Next steps - [Use the Agentic Data Plane CLI](../) - [Configure an LLM Provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 3: Connect Data & Tools **URL**: https://docs.redpanda.com/agentic-data-plane/connect.md --- # Connect Data & Tools > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect Data & Tools latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/index.adoc description: Connect agents, MCP servers, and the apps and data sources behind them. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-06-08" --- Bring your agents, tools, and data into the Agentic Data Plane. Build agents, connect MCP servers, set up authentication, and integrate with your existing stack. - [Connect Tools with MCP Servers](mcp-servers/) Learn how MCP servers expose tools and data to agents, then plug in a managed type or build, register, and test your own. - [Set Up Authentication](authentication/) Configure OAuth providers, user-delegated OAuth, and client connections so agents and tools authenticate securely. - [Integrate with Your Stack](integrations/) Choose an integration pattern and connect the Agentic Data Plane to your existing stack, Redpanda pipelines, and developer tools. --- # Page 4: Connect Agents with A2A **URL**: https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts.md --- # Connect Agents with A2A > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect Agents with A2A latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: a2a-concepts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: a2a-concepts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/a2a-concepts.adoc description: The A2A protocol enables agents to discover and call each other across platforms. Learn how agent cards, authentication, and protocol versioning work in Agentic Data Plane. page-topic-type: concepts personas: agent_builder learning-objective-1: Describe the A2A protocol and its role in agent communication learning-objective-2: Explain how agent cards enable discovery learning-objective-3: Identify how authentication secures agent communication page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- The Agent-to-Agent (A2A) protocol is an open standard that defines how agents discover, communicate with, and call each other across platforms. Each A2A-compliant agent exposes a machine-readable agent card at a well-known URL, describing its capabilities and how to authenticate with it. Callers fetch the card at runtime, so they reach any A2A agent without prior knowledge of its implementation, and agents can update their capabilities without breaking existing callers. In Redpanda Agentic Data Plane, A2A enables two integration paths: external applications and agents calling Agentic Data Plane agents over HTTP, and Redpanda Connect pipelines invoking agents for each event in a stream. After reading this page, you will be able to: - Describe the A2A protocol and its role in agent communication - Explain how agent cards enable discovery - Identify how authentication secures agent communication ## [](#what-is-the-a2a-protocol)What is the A2A protocol? The Agent-to-Agent (A2A) protocol is an open standard that defines how agents discover, communicate with, and invoke each other. Agents that implement A2A expose their capabilities through a standardized agent card. This allows other systems to interact with them without prior knowledge of their implementation. The protocol provides: - Standardized discovery: Agent cards describe capabilities in a machine-readable format. - Platform independence: Any system can call any A2A-compliant agent. - Version negotiation: Protocol versions ensure compatibility between agents. - Communication mode flexibility: Supports synchronous request/response and streaming. Current A2A versions make the interface and protocol version explicit during discovery and standardize how callers receive messages, task status, and artifacts. Callers can integrate with an agent without knowing which framework or runtime implements it. ![A2A lifecycle. An application, agent, or Redpanda Connect pipeline fetches an Agentic Data Plane agent card, chooses a supported interface and protocol version, obtains an access token, sends a message, and receives a direct response or streamed task status and artifacts.](https://docs.redpanda.com/agentic-data-plane/connect/_images/a2a-message-lifecycle.svg) Figure 1. A2A separates discovery from authenticated message and task exchange For the complete specification, see [A2A protocol specification](https://a2a-protocol.org/latest/specification/). ## [](#agent-cards)Agent cards Every A2A-compliant agent exposes an agent card at a well-known URL. The agent card is a JSON document that describes what the agent can do and how to interact with it. For the complete agent card specification, see [A2A Agent Card specification](https://a2a-protocol.org/latest/specification/#44-agent-card). ### [](#agent-card-location)Agent card location Agentic Data Plane agents expose their agent cards at the `/.well-known/agent-card.json` subpath of the agent URL. You can find the agent URL on the **Agents** page. For example, if your agent URL is `https://my-agent.ai-agents.abc123.clusters.rdpa.co`, your agent card URL is `https://my-agent.ai-agents.abc123.clusters.rdpa.co/.well-known/agent-card.json`. The `.well-known` path follows internet standards for service discovery, making agents discoverable without configuration. The agent card is served without authentication. The A2A protocol requires the discovery card to be publicly reachable, because a caller fetches it to learn how to authenticate with the agent. Invoking the agent still requires authentication. For details, see the [Authentication](#authentication) section. To configure the agent card, see [Configure A2A discovery metadata](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/#configure-a2a-discovery-metadata-optional). ## [](#where-a2a-is-used-in-agentic-data-plane)Where A2A is used in Agentic Data Plane Agentic Data Plane uses the A2A protocol in two contexts: ### [](#external-integration)External integration External applications and agents hosted outside Agentic Data Plane use A2A to call Agentic Data Plane agents. This includes backend services, CLI tools, custom UIs, and agents hosted on other platforms. For integration pattern guidance, see [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/). ### [](#internal-pipeline-to-agent-integration)Internal pipeline-to-agent integration Redpanda Connect pipelines use the [`a2a_message`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/a2a_message/) processor to invoke agents for each event in a stream. This enables real-time interaction between streaming data and AI agents, enabling use cases like: - Real-time fraud detection on every transaction. - Streaming data enrichment with AI-generated fields. - Event-driven agent invocation for automated processing. The `a2a_message` processor uses the A2A protocol internally to discover and call agents. For pipeline patterns, see [Integrate with Redpanda Pipelines](https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns/). ## [](#how-agents-discover-each-other)How agents discover each other A2A enables dynamic discovery without hardcoded configuration: 1. The caller fetches the agent card from the well-known URL. 2. The caller checks the supported interfaces, protocol versions, capabilities, and skills. 3. The caller formats message parts for the content types declared by the selected interface and skill. 4. The caller sends the request to the agent’s endpoint. This discovery model allows: - New agents to become available immediately after deployment - Existing agents to update their capabilities while callers adapt dynamically - Callers to understand exactly what agents do through self-describing agent cards ## [](#authentication)Authentication Agentic Data Plane agents require authentication to prevent unauthorized access. The agent card is the exception. Agentic Data Plane serves the card at the `/.well-known/agent-card.json` subpath without authentication, so that callers can discover how to authenticate before they hold a token. Every other request to the agent, including agent invocation, requires a valid access token. Callers authenticate as themselves, never as the agent they’re calling. Each caller needs its own service account and permission to invoke the agent. See [Service account authorization](https://docs.redpanda.com/agentic-data-plane/connect/concepts/#service-account-authorization) and [A2A runtime permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#a2a-runtime-permissions). Callers use OAuth2 client credentials to obtain access tokens: 1. The caller exchanges its client ID and secret for a time-limited access token through OAuth2. 2. The caller includes the access token in the Authorization header when calling the agent endpoint. 3. When the token expires, the caller exchanges credentials again for a new token. This flow ensures: - Credentials stay secure: Applications never send them directly to agents, only access tokens. - Exposure is limited: Tokens expire, reducing the window for compromised credentials. - Integration is standard: Applications can use existing OAuth2 libraries. ### [](#external-integration-2)External integration External applications and agents authenticate with their own service account credentials, not the target agent’s. Grant the caller permission to invoke the agent with an access policy or a custom role. See [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). ### [](#internal-integration)Internal integration The `a2a_message` processor handles authentication automatically. Pipelines don’t need to manage credentials explicitly because they run within Agentic Data Plane with appropriate permissions. ## [](#protocol-versions)Protocol versions The A2A specification uses semantic release versions, but protocol compatibility uses major.minor. Each supported interface in the agent card declares the `protocolVersion` it exposes. Clients select a compatible interface and send that version on requests; patch versions do not affect protocol compatibility. ## [](#next-steps)Next steps - [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 5: How Agents Work **URL**: https://docs.redpanda.com/agentic-data-plane/connect/agents-overview.md --- # How Agents Work > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: How Agents Work latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: agents-overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: agents-overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/agents-overview.adoc description: Learn how Agentic Data Plane agents use a declarative approach backed by a broad library of prebuilt tools and integrations to replace custom agent code. page-topic-type: overview personas: agent_builder, platform_engineer learning-objective-1: Describe what AI agents are and their essential components learning-objective-2: Explain how Agentic Data Plane streaming infrastructure benefits agent architectures learning-objective-3: Identify use cases where Agentic Data Plane agents provide value page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-24" --- Redpanda Agentic Data Plane AI agents take a declarative approach: instead of writing Python or JavaScript agent code, you declare the behavior you want by selecting an LLM, writing a system prompt, and attaching prebuilt tools and integrations. The framework handles execution, tool orchestration, and scaling, backed by real-time streaming infrastructure and built-in filtering and data enrichment. After reading this page, you will be able to: - Describe what AI agents are and their essential components - Explain how Agentic Data Plane streaming infrastructure benefits agent architectures - Identify use cases where Agentic Data Plane agents provide value ## [](#what-is-an-ai-agent)What is an AI agent? An AI agent is a system built around a [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) that interprets user intent, selects the right tools, and chains multiple steps into a workflow. In Agentic Data Plane, agents are declarative: you configure the agent’s role, constraints, and available tools rather than writing imperative agent code. Agents reach tools through MCP servers, and Redpanda Connect provides the real-time streaming and data-processing capabilities the framework orchestrates for you. ## [](#declarative-agent-components)Declarative agent components When you create an agent, you configure the components through Agentic Data Plane rather than writing code: - **System prompt**: Defines the agent’s role, responsibilities, and constraints - **LLM**: Interprets user intent and decides which tools to invoke - **Tools**: External capabilities exposed through the [Model Context Protocol (MCP)](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - **Context**: Conversation history, tool results, and real-time events from Redpanda topics Agents reach these tools through MCP servers. Redpanda Connect pipelines can also invoke agents for event-driven processing, so agents support both interactive workflows and automated streaming. When a user makes a request, the LLM receives the system prompt and context, decides which tools to invoke, and processes the results. This cycle repeats until the task completes. For a deeper understanding of how agents execute, manage context, and maintain state, see [Understand Agent Concepts](https://docs.redpanda.com/agentic-data-plane/connect/concepts/). ## [](#key-benefits)Key benefits Agentic Data Plane agents are designed to reduce the infrastructure and integration work of building production AI systems. - Declarative configuration: Configure agent behavior instead of coding it, with access to a broad library of prebuilt tools and integrations for data sources, APIs, and services. - Real-time streaming data: Agents access live events instead of batch snapshots. - [Remote MCP](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) support: Standardized tool access across any MCP-compatible server. - Managed infrastructure: Deployment, scaling, and security handled for you. - Low-latency execution: Tools run close to your data. - Integrated secrets management: API keys and credentials stored securely. ## [](#use-cases)Use cases Agentic Data Plane AI agents unlock new capabilities across multiple fields. ### [](#for-ai-agent-developers)For AI agent developers Build agents grounded in real-time data instead of static snapshots. Connect your agent to live order status, inventory levels, and customer history so responses reflect current business state, not stale training data. ### [](#for-application-developers)For application developers Add conversational AI to existing applications without rebuilding your backend. Expose your services as MCP tools and let agents orchestrate complex multi-step workflows through natural language. ### [](#for-streaming-developers)For streaming developers Process every event with AI reasoning at scale. Invoke agents automatically from pipelines for fraud detection, content moderation, or sentiment analysis. No batch jobs, no delayed insights. ## [](#limitations)Limitations - MCP servers must be hosted in Agentic Data Plane - Cross-agent calling between separate agents is not supported. Use internal subagents for delegation within a single agent. ## [](#next-steps)Next steps - [Agentic Data Plane Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) - [Understand Agent Concepts](https://docs.redpanda.com/agentic-data-plane/connect/concepts/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 6: Build Agents **URL**: https://docs.redpanda.com/agentic-data-plane/connect/agents.md --- # Build Agents > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Build Agents latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: agents page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: agents.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/agents.adoc description: Understand how AI agents work in the Agentic Data Plane, then create, register, and design them. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-06-18" --- Agents are the workloads that call LLMs and tools through the Agentic Data Plane. Start with how agents work, then create a declarative agent, set up an agent you host yourself, and apply architecture and system-prompt best practices. - [How Agents Work](https://docs.redpanda.com/agentic-data-plane/connect/agents-overview/) Learn how Agentic Data Plane agents use a declarative approach backed by a broad library of prebuilt tools and integrations to replace custom agent code. - [Understand Agent Concepts](https://docs.redpanda.com/agentic-data-plane/connect/concepts/) Understand how declaratively configured agents execute reasoning loops, manage context, invoke tools, and handle errors. - [Connect Agents with A2A](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/) The A2A protocol enables agents to discover and call each other across platforms. Learn how agent cards, authentication, and protocol versioning work in Agentic Data Plane. - [Choose an Agent Architecture](https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns/) Design maintainable agent systems with single-agent and multi-agent patterns based on domain complexity. - [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) Write system prompts that produce reliable, predictable agent behavior through clear constraints and tool guidance. - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) Configure a managed Agentic Data Plane agent declaratively on a single create canvas. No Python or JavaScript code required. - [Draw Charts from an Agent](https://docs.redpanda.com/agentic-data-plane/connect/draw-charts/) Make an agent render inline bar and line charts in the Playground by emitting a chart code block that follows the Agentic Data Plane chart contract. - [Set Up a Self-Managed Agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/) Register a self-managed agent, issue it a client credential, and route its LLM and tool calls through the AI Gateway so spend, traces, and transcripts attribute back to the agent. - [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/) Triggers invoke a deployed agent without calling its API directly, from Microsoft Teams or on a schedule. --- # Page 7: Choose an Agent Architecture **URL**: https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns.md --- # Choose an Agent Architecture > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Choose an Agent Architecture latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: architecture-patterns page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: architecture-patterns.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/architecture-patterns.adoc description: Design maintainable agent systems with single-agent and multi-agent patterns based on domain complexity. page-topic-type: best-practices personas: agent_builder learning-objective-1: Evaluate single-agent versus multi-agent architectures for your use case learning-objective-2: Choose appropriate LLM models based on task requirements learning-objective-3: Apply agent boundary design principles for maintainability page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-02" --- Design agent systems that are maintainable, discoverable, and reliable by choosing the right architecture pattern and applying clear boundary principles. After reading this page, you will be able to: - Evaluate single-agent versus multi-agent architectures for your use case - Choose appropriate LLM models based on task requirements - Apply agent boundary design principles for maintainability ## [](#why-architecture-matters)Why architecture matters Agent architecture determines how you manage complexity as your system grows. The right pattern depends on your domain complexity, organizational structure, and how you expect requirements to evolve. Starting with a simple architecture is tempting, but can lead to unmaintainable systems as complexity increases. Planning for growth with clear boundaries prevents technical debt and costly refactoring later. Warning signs that you need architectural boundaries, not just better prompts: - System prompts exceeding 2000 words - Too many tools for the LLM to select correctly - Multiple teams modifying the same agent - Changes in one domain breaking others Match agent architecture to domain structure: | Domain Characteristics | Architecture | Pros | Cons | | --- | --- | --- | --- | | Single business area, stable requirements | Single agent | Simple to build and maintain, one deployment, lower latency | Limited flexibility, difficult to scale to multi-domain problems | | Multiple business areas, shared infrastructure | Root agent with internal subagents | Separation of concerns, easier debugging, shared resources reduce cost | Single point of failure, subagents share the parent’s credentials and budget | | Cross-organization workflows, independent evolution | External agent-to-agent | Independent deployment and scaling, security isolation, flexible infrastructure | Network latency, authentication complexity, harder to debug across boundaries | Every architecture pattern involves trade-offs. - **Latency versus isolation:** Internal subagents have lower latency because they avoid network calls, but they share a failure domain. External agents have higher latency due to network overhead, but they provide independent failure isolation. - **Shared state versus independence:** Single deployments share provider credentials, budget, and policies but offer less flexibility. Multiple deployments allow independent scaling and updates but add coordination complexity. - **Complexity now versus complexity later:** Starting simple means faster initial development but may require refactoring. Starting structured requires more upfront work but makes the system easier to extend. For foundational concepts on how agents execute and manage complexity, see [Understand Agent Concepts](https://docs.redpanda.com/agentic-data-plane/connect/concepts/). ## [](#single-agent-pattern)Single-agent pattern A single-agent architecture uses one agent with one system prompt and one tool set to handle all requests. This pattern works best for narrow domains with limited scope, single data sources, and tasks that don’t require specialized subsystems. ### [](#when-to-use-single-agents)When to use single agents Use single agents for focused problems that won’t expand significantly. Examples include order lookup agents that retrieve history from a single topic, weather agents that query APIs and return formatted data, and inventory checkers that report stock levels. ### [](#trade-offs)Trade-offs Single agents are simpler to build and maintain. You have one system prompt, one tool set, and one deployment. However, all capabilities must coexist in one agent. Adding features increases complexity rapidly, making single agents difficult to scale to multi-domain problems. > 💡 **TIP** > > You can migrate from a single agent to a root agent with subagents without starting over. Add subagents to an existing agent in Redpanda Agentic Data Plane, then gradually move tools and responsibilities to the new subagents. ## [](#root-agent-with-subagents-pattern)Root agent with subagents pattern A multi-agent architecture uses a root agent that delegates to specialized internal subagents. This pattern works for complex domains spanning multiple areas, multiple data sources with different access patterns, and tasks requiring specialized expertise within one deployment. > 📝 **NOTE** > > Subagents in Agentic Data Plane are internal specialists within a single agent. They share the parent agent’s provider credentials, budget, and policies, but each can have different names, descriptions, system prompts, and MCP tools. By default, a subagent uses the parent’s model and LLM provider, but each subagent can override either or both. ### [](#how-it-works)How it works The root agent interprets user requests and routes them to appropriate subagents. Each subagent owns a specific business area with focused expertise. Subagents access only the MCP tools they need. By default, subagents use the parent agent’s model and LLM provider, but each subagent can override either or both. All subagents share the parent agent’s provider credentials and budget. ### [](#example-e-commerce-platform)Example: E-commerce platform A typical e-commerce agent includes a root agent that interprets requests and delegates to specialists, an order subagent for processing, history, and status updates, an inventory subagent for stock checks and warehouse operations, and a customer subagent for profiles, preferences, and history. By default these subagents use the parent’s model, but each has its own system prompt and tool access, and can override the model or LLM provider when a specialist needs one. ### [](#why-choose-internal-subagents)Why choose internal subagents Internal subagents provide domain isolation, allowing you to update the order subagent without affecting inventory. Debugging is easier because each subagent has narrow scope and fewer potential failure points. All subagents share resources, reducing complexity and cost compared to separate deployments. Use internal subagents when you need domain separation within a single agent deployment. ## [](#external-agent-to-agent-pattern)External agent-to-agent pattern External A2A integration connects agents across organizational boundaries, platforms, or independent systems. > 📝 **NOTE** > > Cross-agent calling between separate Agentic Data Plane agents is not supported. This pattern only applies to connecting Agentic Data Plane agents with external agents you host elsewhere. ### [](#when-to-use-external-a2a)When to use external A2A Use external [Agent2Agent (A2A) protocol](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#agent2agent-a2a-protocol) for multi-organization workflows that coordinate agents across company boundaries, for platform integration connecting Agentic Data Plane agents with agents hosted elsewhere, and when agents require different deployment environments such as GPU clusters, air-gapped networks, or regional constraints. ### [](#how-it-works-2)How it works Agents communicate using the [A2A protocol](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/), a standard HTTP-based protocol for discovery and invocation. Each agent manages its own credentials and access control independently, and can deploy, scale, and update without coordinating with other agents. Agent cards define capabilities without exposing implementation details. ### [](#example-multi-platform-customer-service)Example: Multi-platform customer service A customer service workflow might span multiple platforms: - Agentic Data Plane agent accesses real-time order and inventory data - CRM agent hosted elsewhere manages customer profiles and support tickets - Payment agent from a third party handles transactions in a secure environment Each agent runs on its optimal infrastructure while coordinating through A2A. ### [](#why-choose-external-a2a)Why choose external A2A External A2A lets different teams own and deploy their agents independently, with each agent choosing its own LLM, tools, and infrastructure. Sensitive operations stay in controlled environments with security isolation, and you can add agents incrementally without rewriting existing systems. ### [](#trade-offs-2)Trade-offs External A2A adds network latency on every cross-agent call, and authentication complexity multiplies with each agent requiring credential management. Removing capabilities or changing contracts requires coordination across consuming systems, and debugging requires tracing requests across organizational boundaries. For implementation details on external A2A integration, see [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/). ## [](#common-anti-patterns)Common anti-patterns Avoid these architecture mistakes that lead to unmaintainable agent systems. ### [](#the-monolithic-prompt)The monolithic prompt A monolithic prompt is a single 3000+ word system prompt covering multiple domains. This pattern fails because: - LLM confusion increases with prompt length - Multiple teams modify the same prompt creating conflicts and unclear ownership - Changes to one domain risk breaking others Split into domain-specific subagents instead. Each subagent gets a focused prompt under 500 words. ### [](#the-tool-explosion)The tool explosion A tool explosion occurs when a single agent has too many tools from every MCP server in the environment. This pattern fails because: - The LLM struggles to choose correctly from large tool sets - Tool descriptions compete for limited prompt space - The agent invokes wrong tools with similar names, wasting iteration budget on selection mistakes Limit tools per agent to 10-15 for optimal performance. Agents with more than 20-25 tools often show degraded tool selection accuracy. Use subagents to partition tools by domain. For tool design patterns, see [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/). ### [](#premature-a2a-splitting)Premature A2A splitting Premature splitting creates three separate A2A agents when all logic could fit in one agent with internal subagents. This pattern fails because: - Network latency affects every cross-agent call - Authentication complexity multiplies with three sets of credentials - Debugging requires correlating logs across systems - You manage three deployments instead of one Start with internal subagents for domain separation. Split to external A2A only when you need organizational boundaries or different infrastructure. ### [](#unbounded-tool-chaining)Unbounded tool chaining Unbounded chaining sets max iterations to 100, returns hundreds of items from tools, and places no constraints on tool call frequency. This pattern fails because: - The context window fills with tool results - Requests time out before completion - Costs spiral with many iterations multiplied by large context - The agent loses track of the original goal For best results: - Design workflows to complete in 20-30 iterations - Return paginated results from tools - Add prompt constraints like "Never call the same tool more than 3 times per request" ## [](#model-selection-guide)Model selection guide Choose models based on task complexity, latency requirements, and cost constraints. Agentic Data Plane displays available models with descriptions when creating agents. ### [](#match-models-to-task-complexity)Match models to task complexity For simple queries, choose cost-effective models such as GPT-5 Mini. For balanced workloads, choose mid-tier models such as Claude Sonnet 4.5 or GPT-5.2. For complex reasoning, choose premium models such as Claude Opus 4.5 or GPT-5.2. ### [](#balance-latency-and-model-size)Balance latency and model size For real-time responses, choose smaller models. Use models optimized for speed, such as Mini or base tiers. For batch processing, optimize for accuracy over speed. Use larger models when users aren’t waiting for results. ### [](#optimize-for-cost-and-volume)Optimize for cost and volume For high volume, use cost-effective models. Smaller tiers reduce costs while maintaining acceptable quality. For critical accuracy, use premium models. Higher costs are justified when errors are costly. ### [](#model-provider-documentation)Model provider documentation For complete model specifications, capabilities, and pricing: - [OpenAI Models](https://platform.openai.com/docs/models) - [Anthropic Claude Models](https://docs.anthropic.com/claude/docs/models-overview) - [Google Gemini Models](https://ai.google.dev/gemini-api/docs/models) - [AWS Bedrock Models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) ## [](#design-principles)Design principles Follow these principles to create maintainable agent systems. ### [](#explicit-agent-boundaries)Explicit agent boundaries Each agent should have clear scope and responsibilities. Define scope explicitly in the system prompt, assign a specific tool set for the agent’s domain, and specify well-defined inputs and outputs. Do not create agents with overlapping responsibilities. Overlapping domains create confusion about which agent handles which requests. ### [](#tool-scoping-per-agent)Tool scoping per agent Assign tools to the agent that needs them. Don’t give all agents access to all tools. Limit tool access based on agent purpose. Tool scoping reduces misuse risk and makes debugging easier. ### [](#error-handling-and-fallbacks)Error handling and fallbacks Design agents to handle failures gracefully. Use retry logic for transient failures like network timeouts. Report permanent failures like invalid parameters immediately. Provide clear error messages to users. Log errors for debugging. ## [](#next-steps)Next steps - [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/) - [Connect Agents with A2A](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) --- # Page 8: Set Up Authentication **URL**: https://docs.redpanda.com/agentic-data-plane/connect/authentication.md --- # Set Up Authentication > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up Authentication latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: authentication page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: authentication.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/authentication.adoc description: Configure OAuth providers, user-delegated OAuth, and client connections so agents and tools authenticate securely. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-07-30" --- Agents and tools authenticate through OAuth. Enable user-delegated access so agents act as the signed-in user, register an OAuth provider when you want to manage one yourself, and connect remote MCP clients. - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) Register an OAuth provider in Agentic Data Plane so MCP servers can authenticate users (or service accounts) against an upstream system like Slack, Jira, GitHub, or Salesforce. - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) Have each end-user authenticate against the MCP server's upstream system with their own credentials. Redpanda stores their token in the vault and injects it at call time. - [Set Up GitHub OAuth Provider and MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/github-oauth-tutorial/) Complete walkthrough for configuring GitHub as an OAuth provider and creating a user-delegated GitHub MCP server. - [Connect Remote MCP Clients](https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients/) Connect external MCP clients to MCP servers in AI Gateway using client metadata documents, dynamic client registration, or manual registration. --- # Page 9: Run Claude Code and Codex through AI Gateway **URL**: https://docs.redpanda.com/agentic-data-plane/connect/coding-agents.md --- # Run Claude Code and Codex through AI Gateway > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Run Claude Code and Codex through AI Gateway latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: coding-agents page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: coding-agents.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/coding-agents.adoc description: Launch Claude Code or Codex with rpk ai run, routing their model traffic through a managed, enterprise-passthrough, or AWS Bedrock Agentic Data Plane provider. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Launch Claude Code through an Agentic Data Plane Anthropic or AWS Bedrock provider with rpk ai run claude learning-objective-2: Launch Codex through an Agentic Data Plane OpenAI provider with rpk ai run codex learning-objective-3: Choose between a managed (API-key) provider and an enterprise passthrough (subscription-seat) provider for Claude Code page-git-created-date: "2026-07-01" page-git-modified-date: "2026-08-10" --- Claude Code and Codex are command-line coding agents. The [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) command launches either one with its model traffic routed through an AI Gateway provider, reusing your `rpk ai` sign-in so you don’t set base URLs or mint tokens by hand. Your team’s coding-agent calls then flow through Agentic Data Plane: provider credentials stay in the dataplane secret store, usage rolls up on the **Cost & Usage** page, and no provider key or session token is written to disk. After completing this guide, you will be able to: - Launch Claude Code through an Agentic Data Plane Anthropic or AWS Bedrock provider with `rpk ai run claude` - Launch Codex through an Agentic Data Plane OpenAI provider with `rpk ai run codex` - Choose between a managed (API-key) provider and an enterprise passthrough (subscription-seat) provider for Claude Code ![Request flow from Claude Code or Codex through rpk ai run and AI Gateway to a compatible LLM provider. The coding agent receives model responses while provider credentials remain in the Agentic Data Plane secret store and usage flows to Cost and Usage.](https://docs.redpanda.com/agentic-data-plane/connect/_images/coding-agent-request-flow.svg) Figure 1. `rpk ai run` routes coding-agent model traffic through AI Gateway ## [](#prerequisites)Prerequisites - The `rpk ai` CLI, installed and signed in to the Agentic Data Plane environment whose gateway you want to target. See [Use the Agentic Data Plane CLI](https://docs.redpanda.com/agentic-data-plane/cli/). The `rpk ai run` command uses the credentials from `rpk ai auth login`, not a `rpk cloud` session. - An enabled [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) provider of the matching type. Claude Code needs an Anthropic or AWS Bedrock provider; Codex needs an OpenAI or OpenAI-compatible provider. To create one, see [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). - The coding agent installed on your workstation: - Claude Code. See [Anthropic’s Claude Code documentation](https://code.claude.com/docs). - Codex. See [OpenAI’s Codex documentation](https://github.com/openai/codex). The `rpk ai run codex` command needs a recent Codex version and tells you when yours is too old. - A Redpanda service account or user with permission to invoke the provider (`dataplane_adp_llmprovider_invoke`). See [LLM provider permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#llm-provider-permissions). To let `rpk ai run claude` read the provider and auto-detect its type and auth mode, also grant `dataplane_adp_llmprovider_get`; without read access, pass `--passthrough` for a passthrough provider or `--bedrock` for a Bedrock provider. ## [](#run-claude-code)Run Claude Code Launch Claude Code with its traffic routed through an Anthropic provider: ```bash rpk ai run claude -L -m ``` In this command, `` is the name of your Anthropic provider (the segment after `/providers/` in its **Proxy URL**), and `` is a model the provider allows. The flags work as follows: - `-L`, `--llmprovider`: The provider to route through. Required, and it must be an Anthropic or Bedrock provider. - `-m`, `--model`: The model to use. Omit it to let Claude Code pick its default. Flags before a literal `--` configure `rpk ai run` itself. Everything after `--` is passed straight through to Claude Code, so you can set any of its own flags. For example, to start Claude Code in plan mode: ```bash rpk ai run claude -L -m claude-sonnet-4-6 -- --permission-mode plan ``` Claude Code starts in your real configuration home, so your workspace trust, onboarding, theme, and Model Context Protocol (MCP) servers all apply, and nothing is written into `~/.claude`. ### [](#choose-managed-or-enterprise-passthrough)Choose managed or enterprise passthrough For an Anthropic provider, the `rpk ai run claude` command reads its **Auth passthrough** setting and picks one of two auth modes. The choice determines who pays for the tokens. Managed (API key) The provider stores an Anthropic API key in the dataplane secret store, shown as **API key reference** on its **Connection** card. The gateway authenticates each request with your Redpanda token, injects the stored key, and forwards the call to Anthropic. Tokens are billed as API usage against the account that owns that key, and per-model spend appears on the provider’s **Overview** tab. You need no Anthropic credential of your own. Enterprise passthrough (subscription) The provider has **Auth passthrough** enabled and stores no API key. Claude Code signs in with your own Claude subscription (Max, Team, or Enterprise), and `rpk ai run claude` forwards that subscription login to Anthropic unchanged. Tokens are billed against your subscription seat, not as per-token API usage. Agentic Data Plane authenticates your organization with a separate gateway credential that `rpk ai run` sets for you, so the call is still observed on the **Cost & Usage** page while the subscription bears the cost. > 📝 **NOTE** > > When `rpk ai run claude` cannot read the provider (invoke-only access), it cannot detect the provider’s type or auth mode, and it launches in managed Anthropic mode by default. Pass `--passthrough` to force passthrough for a subscription provider, or pass `--bedrock` for a Bedrock provider. When the provider is readable, its own type and setting win: the launch ignores a contradicting `--passthrough` on a managed Anthropic provider, or a contradicting `--bedrock` on an Anthropic provider, with a warning. ### [](#use-an-aws-bedrock-provider)Use an AWS Bedrock provider Claude Code can also route through a Bedrock provider, running in its native Bedrock mode: ```bash rpk ai run claude -L -m us.anthropic.claude-sonnet-4-6 ``` In this command, `` is the name of your Bedrock provider. For `-m`, pass an inference-profile ID the provider allows: a model identifier with a region prefix such as `us.`, `eu.`, `apac.`, or `global.`. Bedrock requires an inference profile for Claude 4.6 and later models; older models also accept bare foundation-model IDs. For selection guidance, see [AWS Bedrock: Inference profiles and IAM](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#bedrock-inference-profiles). AI Gateway signs each upstream call with the AWS credentials stored on the provider, so no AWS keys reach your machine. The session authenticates to the gateway with your Redpanda token, exactly like a managed Anthropic provider, and the token refreshes for the whole session. Enterprise passthrough does not apply to Bedrock: a Claude subscription has no Bedrock analog, so `--passthrough` fails with an error. In Bedrock mode, Claude Code runs background tasks on a Haiku-class model by default. If the provider’s allowlist does not include one, set `ANTHROPIC_SMALL_FAST_MODEL` to an allowlisted inference-profile ID before you launch. ## [](#run-codex)Run Codex Launch Codex with its traffic routed through an OpenAI or OpenAI-compatible provider: ```bash rpk ai run codex -L -m ``` In this command, `` is the name of your OpenAI or OpenAI-compatible provider, and `` is a model the provider allows. Only OpenAI and OpenAI-compatible providers work, because Codex speaks the OpenAI Responses API. Codex also takes a reasoning-effort flag, `-e` (`--effort`), which accepts `minimal`, `low`, `medium`, or `high`. Omit it to let Codex use its default. As with Claude Code, everything after a literal `--` is passed straight through to Codex. For example, to skip Codex’s approval prompts: ```bash rpk ai run codex -L -m gpt-5.3-codex -e high -- --ask-for-approval never ``` Codex runs against a throwaway configuration home, so your own `~/.codex` is never read or modified. The launch directory is auto-trusted under a `workspace-write` sandbox, so Codex does not prompt for trust on every run. Pass `--no-auto-trust` to keep Codex’s normal first-run trust prompt, for example, when you launch inside a freshly cloned repository. Codex has no passthrough mode. Every Codex session authenticates to the gateway with your Redpanda token, and the gateway injects the provider’s stored API key when one is configured. ## [](#make-it-a-shortcut)Make it a shortcut If you launch the same agent and provider often, define a shell alias so the launch becomes a short command. Anything after `--` is passed to the agent, so you can pin your own agent flags in the alias too. For example, to launch Claude Code through your provider with your preferred model every time: ### Bash or Zsh Add the alias to `~/.bashrc` or `~/.zshrc`: ```bash alias claude-gw='rpk ai run claude -L -m claude-sonnet-4-6' ``` ### fish Define a saved alias (fish persists it as a function): ```fish alias --save claude-gw 'rpk ai run claude -L -m claude-sonnet-4-6' ``` ## [](#attach-agentic-data-plane-hosted-mcp-servers-optional)Attach Agentic Data Plane-hosted MCP servers (optional) Claude Code can call MCP servers for tool access. To use the MCP servers you host in Agentic Data Plane (managed catalog types, self-managed proxied servers, or both), register each one with Claude Code: ```bash claude mcp add https://aigw..clusters.rdpa.co/mcp/v1/ ``` In this URL, `` is your dataplane’s cluster identifier, and `` is the MCP server’s name in Agentic Data Plane. For OAuth-protected MCP servers (most managed types), Claude Code prompts you to complete the consent flow on first use. Agentic Data Plane stores the resulting token in the per-user [token vault](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/), so later invocations reuse it. If a server exposes a large tool catalog, turn on [code mode](https://docs.redpanda.com/agentic-data-plane/gateway/code-mode/) and point Claude Code at the server’s code-mode endpoint to cut the token cost of loading every tool definition. ## [](#observe-usage)Observe usage Open **Cost & Usage** under **Governance** in the sidebar, then filter by your provider. Within a few seconds, the requests from your coding-agent session appear, and the provider’s **Overview** tab breaks spend down per model. Ad-hoc `rpk ai run` sessions are not registered agents, so they do not appear in the per-agent [Transcripts view](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). Transcripts cover agents you create in Agentic Data Plane. ## [](#configure-the-agent-by-hand)Configure the agent by hand The `rpk ai run` command is for interactive command-line work. For CI runners, headless jobs, and application code, authenticate with the OIDC `client_credentials` grant and set the agent’s base URL and token yourself. For the full flow, with the Proxy URL anatomy, the token mint, and SDK examples, see [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/). For a managed Anthropic provider, set the variables and launch the agent directly: ```bash export ANTHROPIC_BASE_URL="https://aigw..clusters.rdpa.co/llm/v1/providers/" export ANTHROPIC_AUTH_TOKEN="" claude ``` In this example, `` is an access token minted with the `client_credentials` grant. It has a short lifetime, and AI Gateway does not refresh it for you, so re-mint it when it expires. The `rpk ai run` command avoids this by refreshing the token for the whole session. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | --llmprovider is required | The rpk ai run command needs the provider to route through. Pass -L , using the provider’s exact name. | | Provider is the wrong type for the agent | Claude Code needs an Anthropic or Bedrock provider, and Codex needs an OpenAI or OpenAI-compatible provider. Confirm the Type on the provider’s Connection card, and pass a provider of the matching type to -L. An older rpk ai plugin also rejects Bedrock providers for Claude Code; update it with rpk ai upgrade. | | Provider is disabled | The provider’s Status is not Active. Enable it with rpk ai llm-provider update --enabled, or from the provider detail page. | | no usable gateway token | The cached rpk ai credentials are missing or expired. Run rpk ai auth login to refresh them, then relaunch. The error’s rpai auth login hint refers to the same sign-in. | | The session hangs, or the agent reports that the model is not allowed for the provider (HTTP 403) | The model you passed to -m is not in the provider’s allowlist. Claude Code retries the rejected requests silently, so the session can appear to hang; when rpk ai run claude can read the provider, the launch warns you and names the allowed models. Open the provider detail page, confirm the model row appears, and pass that exact identifier. For a Bedrock provider, that identifier is an inference-profile ID. | | Missing claude or codex binary on the PATH | The agent binary is not installed or not on your PATH. Install it (see Prerequisites), then relaunch. | | Sign-in required before rpk ai starts | Run rpk ai auth login to sign in and select an Agentic Data Plane environment. | | Unknown command error from rpk ai run | Your rpk ai plugin predates rpk ai run. Update it with rpk ai upgrade, then relaunch. | | Passthrough session ends with an auth error partway through | A passthrough gateway token cannot refresh mid-session. When rpk ai run claude warns that the token expires soon, run rpk ai auth login for a full-length session before relaunching. | | MCP tool calls return OAuthConnectionRequired | You have not completed the consent flow for that MCP server. See User-delegated OAuth; Claude Code surfaces the authorization URL in the error so you can finish the handshake. | ## [](#next-steps)Next steps - [Use the Agentic Data Plane CLI](https://docs.redpanda.com/agentic-data-plane/cli/) - [Configure an LLM Provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) - [Connect Your App to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) --- # Page 10: Understand Agent Concepts **URL**: https://docs.redpanda.com/agentic-data-plane/connect/concepts.md --- # Understand Agent Concepts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Understand Agent Concepts latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: concepts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: concepts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/concepts.adoc description: Understand how declaratively configured agents execute reasoning loops, manage context, invoke tools, and handle errors. page-topic-type: concepts personas: agent_builder learning-objective-1: Explain how agents execute reasoning loops and make tool invocation decisions learning-objective-2: Describe how agents manage context and state across interactions learning-objective-3: Identify error handling strategies for agent failures page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-24" --- When you configure an agent’s LLM, system prompt, and tools, Redpanda Agentic Data Plane manages execution through a reasoning loop: analyze context, invoke a tool, process the result, and repeat until the task completes. Each iteration adds tokens to the context window and runs against your iteration budget, so execution decisions directly shape cost, latency, and reliability. Understanding this model gives you the foundation to set effective iteration limits, design lean tool sets, and diagnose failures before they affect your users. After reading this page, you will be able to: - Explain how agents execute reasoning loops and make tool invocation decisions - Describe how agents manage context and state across interactions - Identify error handling strategies for agent failures ## [](#agent-execution-model)Agent execution model Every agent request follows a reasoning loop. The agent doesn’t execute all tool calls at once. Instead, it makes decisions iteratively. ### [](#the-reasoning-loop)The reasoning loop The following diagram shows how agents process requests through iterative reasoning: ![Diagram showing the agent reasoning loop: User Request flows to LLM Receives Context](https://docs.redpanda.com/agentic-data-plane/_images/agent-reasoning-loop.png) Figure 1. Agent reasoning loop with tool integration When an agent receives a request: 1. The LLM receives the context, including system prompt, conversation history, user request, and previous tool results. 2. The LLM chooses to invoke a tool, requests more information, or responds to user. 3. The tool runs and returns results if invoked. 4. The tool’s results are added to conversation history. 5. The LLM reasons again with an expanded context. The loop continues until one of these conditions is met: ![Diagram showing exit conditions: Task Complete returns response](https://docs.redpanda.com/agentic-data-plane/_images/agent-exit-conditions.png) Figure 2. Reasoning loop exit conditions - Agent completes the task and responds to the user - Agent reaches max iterations limit - Agent encounters an unrecoverable error > 📝 **NOTE** > > If the agent encounters an unrecoverable error on the first iteration, it returns an error immediately. Unrecoverable errors include authentication failures, invalid tool configurations, or LLM API failures. ### [](#why-iterations-matter)Why iterations matter Each iteration includes these phases: 1. **LLM reasoning**: The model processes the growing context to decide the next action. 2. **Tool invocation**: If the agent decides to call a tool, execution happens and waits for results. 3. **Context expansion**: Tool results are added to the conversation history for the next iteration. With higher iteration limits, agents can complete complex tasks but can cost more and take longer. With lower iteration limits, agents can respond faster and are cheaper but may fail on complex requests. #### [](#cost-calculation)Cost calculation Calculate the approximate cost per request by estimating average context tokens per iteration: Cost per request = (iterations x context tokens x model price per token) Example with 30 iterations at $0.000002 per token: Iteration 1: 500 tokens x $0.000002 = $0.001 Iteration 15: 2000 tokens x $0.000002 = $0.004 Iteration 30: 4000 tokens x $0.000002 = $0.008 Total: ~$0.013 per request Actual costs vary based on: - Tool result sizes (large results increase context) - Model pricing (varies by provider and model tier) - Task complexity (determines iteration count) Setting max iterations creates a cost/capability trade-off: | Limit | Range | Use Case | Cost | | --- | --- | --- | --- | | Low | 10-20 | Simple queries, single tool calls | Cost-effective | | Medium | 30-50 | Multi-step workflows, tool chaining | Balanced | | High | 50-100 | Complex analysis, exploratory tasks | Higher | Iteration limits prevent runaway costs when agents encounter complex or ambiguous requests. ## [](#mcp-tool-invocation-patterns)MCP tool invocation patterns MCP tools extend agent capabilities beyond text generation. Understanding when and how tools execute helps you design effective tool sets. ### [](#synchronous-tool-execution)Synchronous tool execution In Redpanda Agentic Data Plane, tool calls block the agent. When the agent decides to invoke a tool, it pauses and waits while the tool executes (querying a database, calling an API, or processing data). When the tool returns its result, the agent resumes reasoning. This synchronous model means latency adds up across multiple tool calls, the agent sees tool results sequentially rather than in parallel, and long-running tools can delay or fail agent requests due to timeouts. ### [](#tool-selection-decisions)Tool selection decisions The LLM decides which tool to invoke based on system prompt guidance (such as "Use get\_orders when customer asks about history"), tool descriptions from the MCP schema that define parameters and purpose, and conversation context where previous tool results influence the next tool choice. Agents can invoke the same tool multiple times with different parameters if the task requires it. ### [](#tool-chaining)Tool chaining Agents chain tools when one tool’s output feeds another tool’s input. For example, an agent might first call `get_customer_info(customer_id)` to retrieve details, then use that data to call `get_order_history(customer_email)`. Tool chaining requires sufficient max iterations because each step in the chain consumes one iteration. ### [](#tool-granularity-considerations)Tool granularity considerations Tool design affects agent behavior. Coarse-grained tools that do many things result in fewer tool calls but less flexibility and more complex implementation. Fine-grained tools that each do one thing require more tool calls but offer higher composability and simpler implementation. Choose granularity based on how often you’ll reuse tool logic across workflows, whether intermediate results help with debugging, and how much control you want over tool invocation order. For tool design guidance, see [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/). ## [](#context-and-state-management)Context and state management Agents handle two types of information: conversation context (what’s been discussed) and state (persistent data across sessions). ### [](#conversation-context)Conversation context The agent’s context includes the system prompt (always present), user messages, agent responses, tool invocation requests, and tool results. As the conversation progresses, context grows. Each tool result adds tokens to the context window, which the LLM uses for reasoning in subsequent iterations. ### [](#context-window-limits)Context window limits LLM context windows limit how much history fits. Small models support 8K-32K tokens, medium models support 32K-128K tokens, and large models support 128K-1M+ tokens. When context exceeds the limit, the oldest tool results get truncated, the agent loses access to early conversation details, and may ask for information it already retrieved. Design workflows to complete within context limits. Avoid unbounded tool chaining. ## [](#service-account-authorization)Service account authorization When you create an AI agent, Agentic Data Plane provisions a service account for it automatically. The service account is the agent’s own identity. You don’t create it, and you don’t manage it separately from the agent. - **Client ID**: Derived from the agent’s name, in the form `serviceaccounts/`. The principal is `@iam.serviceaccounts.`, where `` is your environment’s DNS zone. For example, an agent named `my-agent` gets the client ID `serviceaccounts/my-agent`. - **Authorized scope**: The credentials authenticate the agent’s LLM and MCP tool calls through AI Gateway, and they attribute spend, tokens, latency, and traces back to the agent. They also give you a single point of revocation, because revoking a leaked client secret ends the agent’s access with it. Use these credentials only inside the agent they belong to. A caller that invokes the agent from outside authenticates as its own principal, not as the agent. What the agent may do in Agentic Data Plane comes from [access policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/), not from the service account itself. For an agent, author those on its **Permissions** tab, where the principal is locked to the agent’s service account. For the permission names behind each action, see [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/). ### [](#manage-agent-credentials)Manage agent credentials On a self-managed agent, pair the client ID with a client secret to obtain access tokens. Manage secrets on the agent’s **Credentials** tab, or with the [`rpk ai agent credential`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-credential/) commands. To issue a secret, click **Create secret** and optionally name it. Copy the secret before you dismiss the dialog, because it’s shown once and can’t be retrieved again. Each secret in the list shows: | Column | Description | | --- | --- | | Name | The label you gave the secret when you created it. | | Created | When the secret was issued. | | Expires | 90 days after creation. | More than one secret can be active at a time, so you can roll credentials over without downtime: create the new secret, deploy it, then revoke the old one. To revoke a secret, click **Revoke** on its row. > 📝 **NOTE** > > Revoking a secret immediately breaks anything still using it. The agent’s service account and client ID stay in place, and only that secret stops working. ## [](#next-steps)Next steps - [Choose an Agent Architecture](https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns/) - [Agentic Data Plane Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) --- # Page 11: Create an Agent **URL**: https://docs.redpanda.com/agentic-data-plane/connect/create-agent.md --- # Create an Agent > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Create an Agent latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: create-agent page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: create-agent.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/create-agent.adoc description: Configure a managed Agentic Data Plane agent declaratively on a single create canvas. No Python or JavaScript code required. page-topic-type: how-to personas: agent_builder learning-objective-1: Create a managed Agentic Data Plane agent through the create canvas learning-objective-2: Configure an LLM provider, model, and tools for an agent learning-objective-3: Connect to and test a running agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- Create a new [AI agent](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent) declaratively through Redpanda Agentic Data Plane. No Python or JavaScript code required. This guide walks through the managed create canvas section by section, then covers how to connect to and test the running agent. After reading this page, you will be able to: - Create a managed Agentic Data Plane agent through the create canvas - Configure an LLM provider, model, and tools for an agent - Connect to and test a running agent ## [](#prerequisites)Prerequisites - At least one [LLM provider configured](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) in Agentic Data Plane. - Optional: One or more [MCP servers](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) registered in Agentic Data Plane if you want the agent to call tools. - Optional: A system prompt prepared in advance (see [System prompt best practices](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/)). ## [](#open-the-create-canvas)Open the create canvas 1. Open **Agents** in the sidebar. 2. Click **Create agent**. 3. Choose how the agent runs. Click **Redpanda manages it**, so Redpanda deploys, runs, and observes the agent for you. (To run an agent you host yourself, see [Set Up a Self-Managed Agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/).) ![The runtime choice in the create-agent flow, with a Redpanda manages it card for the managed runtime and an I host it myself card for self-managed agents](https://docs.redpanda.com/agentic-data-plane/shared/_images/create-agent-runtime-choice.png) Agentic Data Plane opens the **New managed agent** canvas: a single page with sections for **Identity**, **Model**, **Instructions**, and **Tools**, plus a collapsible **Advanced** area for subagents and tags. Everything on the canvas is editable later from the agent’s **Overview** tab. Required fields are marked with an asterisk; the **Create agent** button stays disabled until you provide them, with helper text explaining what’s still missing. ## [](#identity)Identity In the **Identity** section, name the agent and optionally describe it. 1. Enter a **Name**. Required. Shown in the agent registry list and the detail header (for example, `Customer Support Agent`). Agentic Data Plane derives the agent’s resource ID from this name when you create the agent; you don’t enter the ID directly. The derived ID is a lowercase slug that starts with a letter, ends with a letter or digit, and can contain letters, digits, and hyphens (up to 63 characters). The ID is used in URLs and CLI commands and can’t be changed after creation. > 💡 **TIP** > > Click **Suggest a name** to generate a placeholder you can edit. 2. Optionally, add a **Description**. An internal note shown on the agent’s detail page. This is not the system prompt; write that under **Instructions**. ## [](#model)Model In the **Model** section, choose the [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) the agent uses to interpret user intent and decide which tools to invoke. The picker is a single searchable list of every model your configured LLM providers expose (for example, `us.anthropic.claude-haiku-4-5-20251001-v1:0`): search by name, select a model, and Agentic Data Plane routes to the provider that serves it. When the same model is served by more than one provider, the picker shows the provider next to the model so you can choose which one to use. To use a model the picker doesn’t list, select a provider’s **Custom model ID** entry and type the identifier. Calls to the model are proxied through the [AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/overview/). To add a new provider, see [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). For model selection guidance, see [Model selection guide](https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns/#model-selection-guide). For external model specifications and pricing: - [OpenAI models](https://platform.openai.com/docs/models) - [Anthropic Claude models](https://docs.anthropic.com/claude/docs/models-overview) - [Google Gemini models](https://ai.google.dev/gemini-api/docs/models) - [AWS Bedrock models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) > 📝 **NOTE** > > **Max iterations** is not on the create canvas. It’s the upper bound on agent reasoning loops per request, and you set it after creation from the agent’s **Overview** tab, in the **Model & runtime** section. Leave it unset (or `0`) to use the runtime default of 100. Each iteration consumes tokens and adds latency, so tune it to task complexity: simple queries (10-20), balanced workflows (20-40), complex analysis (40-100). For an AWS Bedrock provider, Agentic Data Plane validates the model when you create or update the agent. A model ID that isn’t a valid Bedrock model for the provider’s region is rejected at save time, with an error that points at the offending model field, instead of being accepted and then failing later when the agent deploys. A common mistake is the bare foundation-model ID: Claude 4.6 and later require the full inference-profile ID (for example, `us.anthropic.claude-opus-4-7` rather than `claude-opus-4-7`). To use a model the picker doesn’t list, select the provider’s **Custom model ID** entry and enter the full inference-profile ID. The same check applies to each subagent’s model override. See [AWS Bedrock: Inference profiles and IAM](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#bedrock-inference-profiles). ## [](#instructions)Instructions In the **Instructions** section, write the system prompt that defines the agent’s role, responsibilities, constraints, and output format. The system prompt is optional but strongly recommended; when set, it’s sent on every invocation as the first message. To start from a template, select one of the starter cards above the editor. Each card fills in a complete system prompt that you can then edit. The field is a Markdown editor with three modes: - **Live**: The default. Styles the formatted Markdown as you type and reveals the raw syntax for the construct you’re editing. - **Markdown**: Always shows the exact source. - **Preview**: Renders the formatted result. In either editing mode, type the prompt directly and use the formatting toolbar, or select the upload icon to import a Markdown file. To edit in a larger surface, click **Expand editor**. A typical prompt defines: - Agent role and responsibilities - Available tools (matches the servers you select in the **Tools** section) - Safety rules and constraints - Expected output format Example structure: ```text You are an [agent role]. Responsibilities: - [Task 1] - [Task 2] Available tools: - [tool_name]: [description] Never: - [Constraint 1] - [Constraint 2] Response format: - [Format guideline] ``` For full guidance on writing effective prompts, see [System prompt best practices](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/). ## [](#tools)Tools In the **Tools** section, grant the agent access to the Model Context Protocol (MCP) servers that expose tools to it. This section is optional, and you can change the list at any time after creation. 1. In the server list, select the checkbox of each server you want the agent to call. If your organization runs many MCP servers, a **Search MCP servers** box appears above the list to help you find them. To create a new server instead of attaching an existing one, click **Add MCP server**, which opens the server-creation flow in a new tab. 2. The agent discovers each selected server’s tools at runtime; you don’t pre-select individual tools. The agent calls them based on the LLM’s interpretation of the system prompt and the user request. Keep the system prompt aligned with the servers you select so the model knows what’s available. > 💡 **TIP** > > Apply the principle of least privilege. Connect only the servers the agent needs. An agent can reference at most 32 MCP servers in total, counting the servers on the root agent plus every subagent’s servers. The limit counts references, not distinct servers, so a server referenced by both the root agent and a subagent counts twice. If the total exceeds 32, you can’t save the agent. For registering MCP servers, see [Create an MCP server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/). ## [](#advanced-subagents-and-tags)Advanced: subagents and tags Expand the **Advanced** area to add subagents and tags. Both are optional. ### [](#subagents)Subagents Subagents are internal specialists within the same agent. Each subagent has its own name, delegation hint, system prompt, and MCP tools. By default, a subagent inherits the parent agent’s model and LLM provider, but you can override both per subagent. All subagents share the parent agent’s provider credentials, gateway endpoint, and execution settings. The root agent orchestrates and delegates work to the appropriate subagent based on the request. To add a specialist, click **Add subagent** (or **Add a subagent**) and configure it in the panel that opens: - **Name**: Required. Must start with a lowercase letter, contain only lowercase letters, digits, and hyphens, and end with a letter or digit (for example, `researcher`). The name must be unique among the agent’s subagents. - **When should the parent use it?**: Required. The delegation hint the parent reads to decide when to route a task to this subagent. Be specific about inputs, outputs, and use cases. - **Model**: Optional. Defaults to the parent agent’s model. A subagent always uses the parent agent’s LLM provider, so you override only the model, choosing from the models that provider serves. Use **Reset to inherited** to go back to the parent’s model. - **Instructions**: Required. Domain-specific system prompt for what the subagent does. - **Tools**: Optional. Select the MCP servers this subagent can access. Click **Add subagent** to save it, or **Cancel** to discard. To keep the agent flat, leave the list empty. For multi-agent design patterns, see [Agent architecture patterns](https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns/). ### [](#tags)Tags Add **Tags** as key/value pairs to organize and filter agents in the registry and to break down [Cost & Usage](https://docs.redpanda.com/agentic-data-plane/control/budgets/) by team, environment, and more. ## [](#create-the-agent)Create the agent 1. Review the canvas. The **Create agent** button is disabled until you enter a name and pick a model; the helper text next to it tells you which is still missing. 2. Click **Create agent**. 3. Wait for the agent to reach the **Running** state. When the agent is running, open its detail page. A managed agent has these tabs: - **Overview**: The agent’s full configuration, organized into sections that you edit and save one at a time. The **Agent** section holds the display name, description, and tags (the agent ID is fixed). The **Model & runtime** section holds the LLM provider, model, and max iterations, along with the runtime status and endpoint. The **System prompt**, **MCP servers**, and **Subagents** sections hold the instructions, attached servers, and subagents. - **Triggers**: Schedules and event sources that invoke the agent. See [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/). - **Playground**: An interactive test surface. See [Test your agent](#test-your-agent). - **Cost & Usage**: Spend, tokens, and latency for the agent. - **Transcripts**: A record of every conversation against the agent. - **Permissions**: The access policies that apply to the agent as a principal. See [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). ## [](#connect-to-the-agent)Connect to the agent A running managed agent exposes an [A2A](https://a2a-protocol.org/) endpoint that you can call programmatically or integrate with external systems. - **A2A endpoint**: The URL where the agent receives messages, provisioned after the agent starts. It is published in the agent’s card at `https:///.well-known/agent-card.json`, and you can fetch it with the [`rpk ai agent a2a card`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-card/) command. - **Authentication**: The agent card is public, but every other request to the endpoint requires an access token, sent as an `Authorization: Bearer` header. The caller also needs permission to invoke the agent. See [A2A runtime permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#a2a-runtime-permissions). - **Code examples**: Call the agent with an official A2A client SDK (Go, Node.js, Python, and Java), with `curl`, or with the [`rpk ai agent a2a send`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-send/) command. For more about calling agents and integrating them with other systems, see [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/). ## [](#update-a-managed-agent)Update a managed agent Change a managed agent’s configuration at any time after you create it, either in Agentic Data Plane or with the [`rpk ai agent update`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-update/) command. You can edit settings such as the model, system prompt, tools, and subagents. The agent ID is fixed and can’t change. Updates roll out without interrupting the agent. Redpanda starts the new version and waits for it to become ready before it stops the old one, so in-flight and new requests keep reaching a running instance throughout the change. ## [](#test-your-agent)Test your agent Two tabs on the agent detail page support testing and observability: - **Playground**: Send test prompts interactively and inspect each step of the agent’s reasoning, tool calls, and responses. Use Playground during development and tuning. The composer shows the agent’s provider and model as read-only, so a test run always uses the agent’s saved configuration. - **Transcripts**: Records every conversation against the agent (prompt, tool calls, tool outputs, final response). Use transcripts to audit production behavior. To test the agent in Playground: 1. Open the agent’s **Playground** tab. 2. Enter a test prompt. 3. Verify the agent: - Selects appropriate tools - Follows the system prompt’s constraints - Returns the expected output format 4. Iterate on the system prompt, MCP tool selection, or model choice as needed. For detailed observability strategies, see [Monitor Agent Activity](https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents/). ## [](#example-configurations)Example configurations ### [](#simple-query-agent)Simple query agent - **Model**: GPT-5 Mini (fast, cost-effective) - **Tools**: Single MCP server exposing a `get_orders` tool - **Use case**: Customer order lookups ### [](#complex-analytics-agent)Complex analytics agent - **Model**: Claude Sonnet 4.5 (balanced) - **Tools**: Multiple servers for data query, aggregation, and formatting - **Use case**: Multi-step data analysis ### [](#multi-agent-orchestrator)Multi-agent orchestrator - **Model**: Claude Opus 4.5 (advanced reasoning) - **Tools**: Tools the root agent delegates to subagents - **Subagents**: `order-agent`, `inventory-agent`, `customer-agent` - **Use case**: E-commerce operations ## [](#next-steps)Next steps - [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/) - [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/) - [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) --- # Page 12: Create an MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/create-server.md --- # Create an MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Create an MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: create-server page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: create-server.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/create-server.adoc description: Create an MCP server in the Agentic Data Plane by picking a managed type from the marketplace or registering a self-managed server you host yourself, then configure authentication, code mode, and access. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Create a managed or self-managed MCP server in Agentic Data Plane learning-objective-2: Configure each authentication mode and pick the right one for your use case learning-objective-3: Save the server, verify the server URL, and confirm tools were discovered page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- Create MCP servers to give agents structured access to your databases, queues, CRMs, and other business systems. Each server exposes [tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool), [resources](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#resource), and [prompts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#prompt) that an agent can discover and invoke. Choose managed types that Redpanda hosts for you, or register self-managed servers you host yourself. After completing this guide, you will be able to: - Create a managed or self-managed MCP server in Agentic Data Plane - Configure each authentication mode and pick the right one for your use case - Save the server, verify the server URL, and confirm tools were discovered ## [](#prerequisites)Prerequisites - For any authentication mode that uses upstream credentials: the credentials in hand and a secret already created in the Redpanda Agentic Data Plane secret store. Secret references must be `UPPER_SNAKE_CASE` (proto regex `^[A-Z][A-Z0-9_]*$`). For example: `MCP_API_KEY`, `OPENAI_API_KEY`. - For user-delegated OAuth on a managed server, or to attach an existing provider on a self-managed server: an [OAuth provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) resource already configured. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). - For automatic OAuth setup on a self-managed server: nothing, unless the server doesn’t support dynamic client registration. In that case, an OAuth app registered with the upstream vendor, its client ID, and a client-secret reference for a confidential app. ## [](#open-the-mcp-servers-page)Open the MCP Servers page 1. Open **MCP Servers** in the sidebar. 2. Click **Add MCP server**. ## [](#pick-a-backend)Pick a backend The marketplace picker lists every managed type as a card and includes a **Remote (Proxied)** option for self-managed servers. - **Managed**: Pick a card. Redpanda hosts the server in-process. The configuration form is rendered from the type’s protobuf schema; field labels and help text come straight from the proto. - **Self-managed**: Pick **Remote (Proxied)**. You provide a URL and a transport, and Redpanda proxies requests to your server. For a tour of every managed type and which one fits your use case, see [Managed catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/). To go deep on the self-managed path (transport choices, TLS, multi-server aggregation), see [Register a self-managed server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/). ## [](#name-and-basic-fields)Name and basic fields Every server has the same identity fields. | Field | Required | Notes | | --- | --- | --- | | Name | Yes | Lowercase letters, numbers, and hyphens only (proto regex ^[a-z][a-z0-9-]*$); 1–63 characters; must start with a letter. Immutable after create. Used as the URL path segment. | | Description | No | Up to 256 characters. Shown in list and detail views. | | Enabled | Yes (toggle) | Disabled servers reject every tool call. | The `name` is the server’s identifier and is immutable. The update API preserves it: you can’t rename a server after create. To use a different name, delete the server and create a new one. ## [](#configure-the-managed-flow-managed-types-only)Configure the managed flow (managed types only) Each managed type ships its own configuration schema. The form on this page is rendered from the type’s `_config.proto`, so field labels and help text come directly from the proto definition. No per-type hand-written form code is required. For per-type fields, see the [Managed catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/): a reference of every managed MCP type Redpanda hosts, grouped by category, with a description and a link to its setup guide where one exists. > 📝 **NOTE** > > MCP enforces a 64-character limit on tool names. For managed MCP types whose generated names exceed that limit, Agentic Data Plane truncates the prefix and replaces it with a hash, so the long-form name becomes something like `64ghux5adn_github_read_v1_GitHubReadService_GetAuthenticatedUser`. Agentic Data Plane always preserves the version, service, and method suffix, so the short tool name an agent sees (for example, `get_authenticated_user`) stays stable across truncations. > > You don’t configure the truncation. This detail matters only when you correlate tool calls in logs or transcripts against the generated proto names. ## [](#configure-the-self-managed-flow-remoteproxied-only)Configure the self-managed flow (Remote/Proxied only) Two fields on top of the identity fields: | Field | Required | Notes | | --- | --- | --- | | URL | Yes | Endpoint URL of your server. Must start with http:// or https://. HTTPS is required if you choose user-delegated OAuth (proto rule remote_mcp.user_oauth_requires_https). | | Transport | Yes | SSE (server-sent events) or Streamable HTTP (newer bidirectional protocol). Pick whichever your server speaks. See Register a self-managed server for how to test which transport your server uses. | ## [](#configure-authentication)Configure authentication Both managed and self-managed servers offer the same set of authentication modes. Managed types only show the modes that make sense for that type; for example, SQL never offers user-delegated OAuth. | Mode | Use when | | --- | --- | | None | The upstream server doesn’t need authentication, or the managed type wraps an in-process resource that needs no creds. | | Static key | Single shared API key. Provide a key_ref (secret reference, UPPER_SNAKE_CASE) and an optional header_name (defaults to Authorization). | | Token passthrough | Forward the caller’s Authorization header to the upstream server as-is. Use this mode when the upstream already validates upstream tokens. | | Service-account OAuth | 2-legged OAuth client credentials. One shared upstream identity for every caller. Provide client_id, client_secret_ref, token_url, and any required scopes. | | User-delegated OAuth | Each end-user authenticates against the upstream system with their own credentials, and Redpanda injects the user’s token at call time. For a self-managed server, Redpanda can set up OAuth automatically from the server URL, or you can attach an OAuth provider your organization already registered. A managed type that supports this mode attaches an OAuth provider in the type’s configuration. The first time a user calls a tool that needs this server, Redpanda surfaces a consent prompt; Redpanda stores the resulting connection in the token vault, where it shows up under My Connections. See User-delegated OAuth for the full flow. | > 📝 **NOTE** > > Choosing between `Service-account OAuth` and `User-delegated OAuth` is the credential-mode decision. Service-account authentication gives every caller the same identity at the upstream; user-delegated authentication gives each caller their own. ## [](#code-mode-optional)Code mode (optional) Toggle **Enable Code Mode** to add `{name}_search` and `{name}_execute` tools alongside the server’s own tools. Agents can use `_search` to discover available tools and `_execute` to run sandboxed Python or JavaScript that orchestrates them. This is useful when you’d rather have the agent generate a small program than call tools one at a time. When code mode is enabled, the **Connection** tab surfaces a second URL, the `Code Mode URL`, that clients can connect to instead of the standard one. > 📝 **NOTE** > > Defer advanced code-mode patterns (sandboxing limits, runtime selection, dependency management) to follow-up content; this page covers only enabling it. ## [](#output-format-optional)Output format (optional) By default, a server forwards each tool’s results to the agent as JSON. To reduce the number of tokens an agent spends reading large, tabular results, set the server’s **Output format** to a token-optimized encoding. In the **Output format** section, set **Encoding** to one of these values: | Encoding | Result | | --- | --- | | JSON (default) | Forwards results to the agent unchanged. | | TOON (token-optimized) | Re-encodes tabular and nested-object results into a denser TOON representation that uses fewer tokens than JSON. | | JTON (token-optimized) | Re-encodes tabular results into JTON, a compact superset of JSON. | The gateway re-encodes a result only when it can do so without losing data. When a result isn’t tabular, or can’t be converted losslessly, the gateway forwards the original JSON unchanged, so changing the encoding never drops or corrupts a result. To change the encoding later, edit the server. The selected encoding takes effect after token-optimized output is enabled for your gateway. Until then, results are forwarded as JSON. > 📝 **NOTE** > > Output format is a preview capability. ## [](#save-and-verify)Save and verify 1. Click **Create server**. The server appears in the list with a **Type** badge: **Managed** or **Self-managed**. 2. Open the detail page. The **Overview** tab summarizes the server, including its discovered tools, attached agents, and server metadata. 3. If your organization has the **Activity** tab enabled, open it to review how agents have used this server. The **Recent calls** table lists recent tool calls across the agents attached to this server over the last 7 days, showing each call’s time, tool, agent, caller, duration, and status. Click **Refresh** to reload the feed. If no traffic has run yet, the tab is empty. 4. Open the **Connection** tab to get the `Server URL` that clients use to connect. Copy it for use later. The tab also gives you ready-to-paste connection snippets, pre-filled with this server’s URL, for common MCP clients such as Claude Code, Claude Desktop, and Cursor, plus code examples in several languages. The code examples authenticate with the OAuth authorization-code flow and surface the OAuth client ID to use. 5. Open the **Inspector** tab. Redpanda performs a live `tools/list` against the server and lists every tool it discovered. See [Test a server’s tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) for how to call them. A populated tools list confirms that the connection works and credentials resolve correctly. If the list is empty or the tab shows an error, see [Troubleshooting](#troubleshooting). To mask, drop, or filter the data this server’s tools return before an agent sees it, open the **Data Policies** tab. See [Shape Tool Data with Data Policies](https://docs.redpanda.com/agentic-data-plane/connect/data-policies/). > 📝 **NOTE** > > Data policies are a preview capability. ## [](#create-from-the-cli)Create from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) for a non-UI path through the same create flow, useful for scripting and CI. ```bash # Managed type (Workday example) rpk ai mcp-server create --name workday-hr \ --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.workday.v1.WorkdayMCPConfig", "tenant": "acme", "host": "wd2-impl-services1.workday.com", "oauth_refresh_token": { "username": "isu_user@acme", "password_secret_ref": "${secrets.WORKDAY_PASSWORD}", "refresh_token_secret_ref": "${secrets.WORKDAY_REFRESH_TOKEN}" } }' # Managed type with user-delegated OAuth inside the managed config # (NetSuite example) rpk ai mcp-server create --name netsuite \ --description "NetSuite ERP with user OAuth" \ --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.netsuite.v1.NetSuiteMCPConfig", "accountId": "1234567_SB1", "userOauth": { "providerName": "netsuite", "requiredScopes": ["rest_webservices"] } }' # Self-managed server with user-delegated OAuth rpk ai mcp-server create --name github-proxy \ --url https://mcp.example.com/sse \ --user-oauth-provider github \ --user-oauth-scopes repo,read:org # Update a self-managed server's user-delegated OAuth scopes rpk ai mcp-server update github-proxy \ --user-oauth-provider github \ --user-oauth-scopes repo,read:org,workflow ``` | Flag | Notes | | --- | --- | | --managed-config | JSON blob carrying the managed type’s _config.proto shape, including a @type URL. | | --user-oauth-provider | Self-managed servers only. Name of an OAuth provider already registered on the Outbound providers tab of Integrations setup. This flag is the attach-a-provider path; it doesn’t request automatic OAuth setup. See Configure an OAuth Provider. The principal needs dataplane_aigateway_oauthprovider_attach on the named provider. Managed servers that support user-delegated OAuth carry their auth shape inside --managed-config. | | --user-oauth-scopes | Self-managed servers only. Comma-separated scopes the server requires. Provide every scope any tool may need; user re-consent is required if scopes change later. | The command targets the AI Gateway of your selected Agentic Data Plane environment, using credentials from `rpk ai auth login`. ## [](#list-and-filter-servers-with-the-api)List and filter servers with the API To enumerate MCP servers programmatically, call `MCPServerService.ListMCPServers`. Beyond the `name_contains` substring match, the request accepts an AIP-160 `filter` expression and an AIP-132 `order_by`, so you can scope and sort the list on the server (for example, to page through only the managed servers, newest first) instead of filtering the loaded page. The list endpoint uses the same OIDC client-credentials authentication as the rest of AI Gateway. Mint a service-account access token using the flow in [Authenticate with OIDC client credentials](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/#authenticate-with-oidc-client-credentials), then pass it as a bearer token. The service account needs `dataplane_adp_mcpserver_list`. The request’s `filter` object carries two independent fields: | Field | Meaning | | --- | --- | | name_contains | Case-insensitive substring match on the server name. | | filter | AIP-160 expression (maximum 1024 characters) that the server ANDs with name_contains and the tenant scope. Filterable columns: name, type, enabled, description, created_at, and updated_at. Use it for multi-value OR semantics, for example type = "MCP_SERVER_TYPE_REMOTE" OR type = "MCP_SERVER_TYPE_MANAGED". The type column accepts the MCPServerType enum values MCP_SERVER_TYPE_REMOTE and MCP_SERVER_TYPE_MANAGED; enabled accepts true or false. | Set `order_by` (maximum 256 characters) to a comma-separated list of fields, each optionally suffixed with \` desc\` (ascending by default), for example `created_at desc`. Orderable fields are the keyset columns `name` and `created_at`. An unsupported field returns `InvalidArgument`. Leave `order_by` empty to use the server default (`created_at desc`). The order is encoded into the page token, so keep it stable across a paged walk. ```bash ACCESS_TOKEN="" # from the client_credentials flow DATAPLANE_BASE="https://aigw..clusters.rdpa.co" curl -s --request POST \ --url "${DATAPLANE_BASE}/redpanda.api.adp.v1alpha1.MCPServerService/ListMCPServers" \ --header "Authorization: Bearer ${ACCESS_TOKEN}" \ --header 'Content-Type: application/json' \ --data '{ "filter": { "filter": "type = \"MCP_SERVER_TYPE_MANAGED\"" }, "order_by": "created_at desc", "page_size": 50 }' | jq ``` The response carries the matching `mcp_servers`, a `next_page_token` for the next page, and an estimated `total_size` (AIP-158) of all servers matching the filter, ignoring pagination. ## [](#edit-disable-and-delete-a-server)Edit, disable, and delete a server - **Edit:** most fields can change. The `name` and `type` are immutable after create. - **Disable:** toggle **Enabled** off. The server stays in the list, but every tool call returns an error until you re-enable it. - **Delete:** permanently removes the server record. User OAuth connections are scoped to the OAuth Provider, not the server, so they stay in the token vault and keep working for other servers that use the same provider. If you re-create a server against the same provider, users don’t re-consent unless the new server requires additional scopes. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | Secret not found | The secret reference is case-sensitive and must be UPPER_SNAKE_CASE matching the proto regex ^[A-Z][A-Z0-9_]*$. Confirm the secret exists in the Agentic Data Plane secret store and is spelled identically. | | "HTTPS required" error on save (self-managed) | You picked user-delegated OAuth on a server with an http:// URL. Switch to https:// (proto rule remote_mcp.user_oauth_requires_https). | | Inspector / tools list empty after create | The connection or authentication might be failing. Check the Connection tab for upstream errors. For self-managed, confirm the transport (SSE vs. Streamable HTTP) matches what your server actually speaks. | | Tool calls return errors after disable/enable | Disabled servers reject all tool calls. Confirm the Enabled toggle is on. | ## [](#next-steps)Next steps - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Shape Tool Data with Data Policies](https://docs.redpanda.com/agentic-data-plane/connect/data-policies/) - [Plug in an App, Database, or Tool](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) --- # Page 13: Shape Tool Data with Data Policies **URL**: https://docs.redpanda.com/agentic-data-plane/connect/data-policies.md --- # Shape Tool Data with Data Policies > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Shape Tool Data with Data Policies latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-policies page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: data-policies.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/data-policies.adoc description: Add data policies to an MCP server to mask, drop, or filter the data its tools send and return, restrict argument values, and preview the effect before saving. page-topic-type: how-to personas: platform_engineer, agent_builder learning-objective-1: Create a data policy that masks, drops, or filters the data a tool returns learning-objective-2: Restrict the argument values that callers can send to a tool learning-objective-3: Preview the composed effect of a server's data policies before saving page-git-created-date: "2026-07-29" page-git-modified-date: "2026-08-10" --- Data policies shape the data an MCP server exposes. Use them to mask, drop, or hash sensitive fields in the results a [tool](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool) returns, restrict the argument values a caller can send, and filter whole records out of list results, all before the model sees the data. Sensitive data stays protected without turning tools off. After reading this page, you will be able to: - Create a data policy that masks, drops, or filters the data a tool returns - Restrict the argument values that callers can send to a tool - Preview the composed effect of a server’s data policies before saving ## [](#how-data-policies-work)How data policies work A data policy belongs to one MCP server. It binds a set of transforms to the tools they shape and the people they apply to: - **Tools**: The tools on this server the policy shapes. An empty list means every tool. - **Principals**: Who the policy applies to, as `User:` entries. An empty list means every caller. - **Transforms**: What the policy does to tool-call arguments on the way to the server and to results on the way back. Data policies complement permissions. Permissions decide _whether_ a caller can invoke a tool. A server’s data policies decide _what the data looks like_ for the callers they match. See [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/). The [AI Gateway](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-gateway) enforces data policies on live traffic in both directions. A server can carry several data policies, and every policy that matches a call applies. Policies compose most-restrictively: adding a policy can only narrow what a caller sees, never widen it. Data policies fail closed. If a rule cannot be enforced, matching calls are denied rather than served unshaped data. ## [](#prerequisites)Prerequisites You need an MCP server with discovered tools. To create one, see [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/). ## [](#open-the-data-policies-tab)Open the Data Policies tab 1. Open **MCP Servers** in the sidebar. 2. Select a server. 3. Open the **Data Policies** tab. The **Tools** list shows every tool the server exposes. A colored dot marks each tool’s status: green when at least one data policy shapes it, yellow when the tool handles data but no policy covers it, and gray when there is nothing to protect. A legend at the bottom of the list counts the tools in each status. When a server exposes many tools, use the filter box to narrow the list by tool name, title, or description; the count next to **Tools** shows how many tools match. ## [](#create-a-data-policy)Create a data policy 1. In the **Tools** list, select the tool you want to shape. 2. Click **+** to start a new policy. The policy gets a suggested name, which you can edit. 3. On the **Configuration** tab, choose a treatment for each request or response field: - **Keep**: Preserve the field. In allowlist mode, only kept fields survive. - **Mask**: Replace the field’s value but keep the field present. Choose one of the [mask methods](#mask-methods). - **Drop**: Remove the field entirely. 4. Save the policy. A new policy applies only to the tool you created it from until you widen its targeting. See [Target tools and people](#target-tools-and-people). You can also edit the policy as YAML. The form and the YAML editor are two views of the same draft: switch between them at any time. ### [](#mask-methods)Mask methods | Method | What it does | | --- | --- | | Redact | Replaces the value with a fixed placeholder. The default placeholder is [REDACTED]. | | Partial | Keeps the first or last characters and masks the rest. For example, keeping the last four characters turns 123-45-6789 into *6789. | | Hash | Replaces the value with a salted SHA-256 digest. Equal inputs produce equal digests, so values stay correlatable without exposing the plaintext. The salt comes from a secret you reference by name, never from the policy itself. | | Pattern | Rewrites the value with an RE2 regular-expression substitution. The replacement text can reference capture groups, such as $1. | ### [](#select-fields)Select fields A rule selects fields with a JSONPath expression, such as `$.user.email` (you can omit the leading `$.`). To match a field wherever it appears, use a descendant selector such as `$..ssn` with mask and drop rules. Wildcard, index, and filter expressions are not supported: you can’t save a policy that uses them. For list results, selectors evaluate against each record. For single-object results, they evaluate against the whole payload. By default, a mask or drop rule whose selector matches nothing denies the call. This protects you when an upstream API renames a field: the renamed field fails the call instead of silently leaking. If a field is legitimately optional, mark its rule absence-safe so a missing field is acceptable. ### [](#allowlist-mode)Allowlist mode By default, fields you don’t select pass through unchanged, and only your mask and drop rules take effect. Switch a policy’s field rules to allowlist mode to invert this: only fields with an explicit **Keep** survive, and everything else is dropped. ## [](#restrict-argument-values)Restrict argument values A policy can tighten the allowed values of a tool-call argument. The AI Gateway rejects a call whose argument falls outside the limits, with an error that names the violated argument. Address the argument with a JSONPath into the tool’s arguments, such as `$.query.limit`, then apply any of these limits: | Limit | Applies to | | --- | --- | | Minimum and maximum value | Numbers | | Minimum and maximum length | Strings | | Pattern (RE2 regular expression) | Strings | | Format, such as email, date, or uuid | Strings | | Allowed values | Strings | | Minimum and maximum item count | Arrays | The gateway also merges these limits into the input schema the tool advertises, so agents see the tightened contract up front instead of discovering it through failed calls. ## [](#filter-records-out-of-results)Filter records out of results A row filter drops whole records from a list-shaped result. It names the record array and the condition a record must satisfy to survive: - **Path**: Where the record array is in the result, such as `$.body` for an OpenAPI server’s response envelope, `$.result` for a ServiceNow-style wrapper, or `$` when the result is a bare array. If the path is absent or not an array, the filter does nothing. - **Condition**: A comparison of a record field against a literal value, such as `@.priority >= 8`. Supported operators are `==`, `!=`, `<`, `<=`, `>`, and `>=`. The literal can be a string, number, boolean, or null. A record that doesn’t have the compared field never survives the filter. Conditions on the same array combine across rules and policies, so a record must satisfy all of them. ## [](#target-tools-and-people)Target tools and people By default, a saved policy applies to the tool you created it from. Widen or narrow its targeting on the **Configuration** tab: - **Tools**: List the tools the policy shapes. Clear the list to shape every tool on the server. - **Principals**: List the people the policy applies to, as `User:` entries with the caller’s full email address, for example `User:alice@example.com`. Clear the list to apply the policy to every caller. ## [](#preview-the-effect)Preview the effect Before you save, check what a policy actually does: - The **Configuration** tab shows the composed effect for the selected tool across all of the server’s policies, including your unsaved edits, with each field’s winning treatment and which policy contributed it. - The **Preview** tab runs sample arguments and a sample result through the same code that shapes live traffic and shows the shaped data an agent would receive. If matching calls would be denied, a banner explains why, for example an unenforceable rule or a strict selector that matches nothing. ## [](#limitations)Limitations Data policies are a preview capability with these limitations: - Principals accept individual users only. Group-based targeting is not supported. - Field selectors support named paths and descendant selectors only. Wildcard, index, and filter expressions are rejected. - On a self-managed server that uses the legacy Server-Sent Events (SSE) transport, results stream past response shaping. Calls that match a policy with response rules are denied on those servers. Request rules still work. - If your gateway does not enforce data policies, the tool header shows a **Not enforced here** badge: rules save but don’t apply to live traffic. To read a server’s data policies through the API, get the individual server. List responses omit data policies to keep payloads small. ## [](#next-steps)Next steps - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) --- # Page 14: Draw Charts from an Agent **URL**: https://docs.redpanda.com/agentic-data-plane/connect/draw-charts.md --- # Draw Charts from an Agent > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Draw Charts from an Agent latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: draw-charts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: draw-charts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/draw-charts.adoc description: Make an agent render inline bar and line charts in the Playground by emitting a chart code block that follows the Agentic Data Plane chart contract. page-topic-type: how-to personas: agent_builder learning-objective-1: Instruct an agent to draw a bar or line chart by emitting a chart code block learning-objective-2: Migrate an earlier chart prompt to the Agentic Data Plane chart contract learning-objective-3: Diagnose a chart that renders as an error or stays a placeholder page-git-created-date: "2026-06-30" page-git-modified-date: "2026-08-10" --- An agent draws a chart by emitting a fenced code block tagged `chart` whose body is strict JSON that follows the Agentic Data Plane chart contract. The Playground renders the data with the Agentic Data Plane chart components. The fence body contains data, not Recharts component code. The agent’s **Playground** tab renders that block with Chart, Data, and Code views. This is a rendering convention: the agent decides when a chart helps and writes the data, and the Playground draws it. No tool call or application setup is required. > ❗ **IMPORTANT** > > The chart contract supports `bar` and `line` charts. The renderer uses shadcn/Recharts, not Chart.js. Existing `bar` and `line` blocks written for the earlier Chart.js format still render. When a block has no top-level `title`, the renderer falls back to `options.plugins.title.text`, but it ignores every other legacy presentation option. Other legacy chart types show an error. Convert them to `bar` or `line`. See [Migrate from Chart.js](#migrate-chart-js-prompt) to update an existing prompt. After reading this page, you will be able to: - Instruct an agent to draw a bar or line chart by emitting a chart code block - Migrate an earlier chart prompt to the Agentic Data Plane chart contract - Diagnose a chart that renders as an error or stays a placeholder ## [](#prerequisites)Prerequisites - A deployed agent in Agentic Data Plane. To create one, see [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). - An understanding of how to write an agent system prompt. See [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/). ## [](#how-chart-rendering-works)How chart rendering works When the agent’s response contains a fenced code block tagged `chart`, the Playground parses the block body as a chart configuration and draws the chart in place of the code. Every other code block renders as plain code, so a `chart` block is the agent’s only departure from ordinary output. The renderer supplies the visual presentation. A valid configuration renders as a responsive SVG chart with the Redpanda theme, a tooltip, and a legend. The agent supplies the chart type, optional title, labels, series names, and numeric values. Agentic Data Plane draws a `chart` block wherever it shows the agent’s response: the agent’s **Playground** tab and the **Transcripts** tab. When an external application calls the agent, whether that application draws the chart depends on how it renders the agent’s output. ![Chart rendering flow. An agent response contains a strict JSON chart code block. The Playground validates the chart contract and produces Chart, Data, and Code views from the same configuration. Invalid configurations produce a visible inline error.](https://docs.redpanda.com/agentic-data-plane/connect/_images/agent-chart-rendering.svg) Figure 1. The Playground turns a chart code block into three synchronized views ## [](#write-the-chart-block)Write the chart block Use the fields below. Unsupported chart types or invalid values show an error. - Tag the fence `chart`. The opening fence is three backticks followed by the word `chart`, with no other language tag. - Write the body as strict JSON. The Playground parses the body with a JSON parser, not a JavaScript evaluator, so use double-quoted keys and strings, unquoted numbers, no trailing commas, no comments, and no JavaScript expressions, functions, or callbacks. - Set the top-level `type` to `bar` or `line`. - Add a top-level `data` object with `labels` and `datasets` arrays. Each dataset contains a `data` array and can include a `label`. Without a `label`, the Playground names the series `Series 1`, `Series 2`, and so on. Values in a dataset must be finite numbers or `null`. - To display a title, add a top-level `title` string. - Do not add renderer-specific `options`, plugins, scales, animations, or color properties. The Playground controls those presentation details. A bar chart: ````text ```chart { "type": "bar", "title": "Monthly orders", "data": { "labels": ["Jan", "Feb", "Mar"], "datasets": [ { "label": "Orders", "data": [120, 190, 140] } ] } } ``` ```` ### [](#supported-chart-types)Supported chart types | Type | Use for | | --- | --- | | bar | Comparisons across categories. | | line | Trends across an ordered axis, such as time. | ## [](#instruct-the-agent-through-its-system-prompt)Instruct the agent through its system prompt To render a chart, tell the agent to emit a valid `chart` block. Add the convention to the agent’s [system prompt](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/): name the fence tag, state the strict JSON rules, list the supported types, and include one worked example. The following snippet works as a standalone prompt or as a section added to an existing one: ````text You can draw charts inline. When a bar or line chart communicates better than text, emit one fenced code block tagged exactly `chart` whose body follows the chart contract as strict JSON. Rules for the chart block: - The fence language tag is exactly: chart - The body is valid JSON: double-quoted keys and strings, unquoted numbers, no trailing commas, no comments, and no JavaScript, functions, or callbacks. - The top level contains "type", optional "title", and "data". - Allowed "type" values: "bar" and "line". - "data" contains "labels" and "datasets". Each dataset contains a numeric "data" array and can include a "label". - Do not emit renderer-specific options, plugins, scales, animations, or colors. - Put a brief plain-text explanation (1 or 2 sentences) before the chart block. Do not also paste a data table; the Playground has a built-in Data view. Example response to "show last quarter's orders by month": Orders peaked in February: ```chart { "type": "bar", "title": "Last quarter's orders", "data": { "labels": ["Jan", "Feb", "Mar"], "datasets": [ { "label": "Orders", "data": [120, 190, 140] } ] } } ``` ```` For prompt-writing patterns that make this output reliable, see [Output formatting](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/#output-formatting). ## [](#test-the-chart-in-the-playground)Test the chart in the Playground 1. Open the agent and switch to the **Playground** tab. 2. Enter a prompt that calls for a chart, such as `Show the broker count for the demo cluster over the last 3 months as a bar chart`. 3. Wait for the agent to finish its response. The chart appears in place of the `chart` block. ![The Playground rendering a bar chart titled Broker count / demo cluster / 3 months, with a Chart, Data, and Code view selector above bars for January, February, and March.](https://docs.redpanda.com/agentic-data-plane/connect/_images/agent-chart-rendered.png) Use the view selector to switch between: - **Chart**: The responsive bar or line chart. Hover over the chart to inspect values. - **Data**: The chart’s values as a table, reconstructed from the configuration. - **Code**: The JSON configuration the Playground parsed. Zoom, pan, and PNG export are not available. ## [](#migrate-chart-js-prompt)Migrate from Chart.js Existing `bar` and `line` configurations keep their type, labels, datasets, and values. Move the title to the top level and remove legacy presentation options. Convert other chart types to `bar` or `line`. | Earlier Chart.js configuration | Current chart contract | | --- | --- | | "type": "bar" or "type": "line" | Keep the type. | | pie, doughnut, polarArea, or radar | Use bar to compare categories. | | scatter or bubble | Choose line for an ordered trend or bar for category comparisons, and convert each point to a numeric dataset value. | | options.plugins.title.text | Move the string to the top-level title field. | | options, plugins, scales, animations, or colors | Remove them. The Playground supplies the presentation. | | Zoom, pan, or PNG export instructions | Remove them. These controls are not available. | For example, replace: ```json { "type": "bar", "data": { "labels": ["Jan", "Feb", "Mar"], "datasets": [ { "label": "Orders", "data": [120, 190, 140], "backgroundColor": "#ea580c" } ] }, "options": { "plugins": { "title": { "display": true, "text": "Monthly orders" } } } } ``` With: ```json { "type": "bar", "title": "Monthly orders", "data": { "labels": ["Jan", "Feb", "Mar"], "datasets": [ { "label": "Orders", "data": [120, 190, 140] } ] } } ``` ## [](#troubleshooting)Troubleshooting A block that does not satisfy the contract shows an inline error that names the problem. Click **Error details** to inspect the body the agent sent. ![An inline Failed to render chart error that explains the JSON problem and provides an Error details button.](https://docs.redpanda.com/agentic-data-plane/connect/_images/agent-chart-error.png) | Symptom | Cause and fix | | --- | --- | | A Failed to render chart error appears in place of the chart | The block body does not satisfy the contract. The error description names the problem. Click Error details to inspect the body the agent sent. Common causes include invalid JSON, a missing type field, a missing data object, a type other than bar or line, or a non-numeric dataset value. | | The error description reads Unsupported chart type | Change the type to bar or line. For migration guidance, see Migrate from Chart.js. | | A Building chart… placeholder remains visible | The placeholder shows while the chart block streams in, because the Playground cannot parse the configuration until the closing fence arrives. The chart replaces the placeholder when the agent finishes the response. If the response finishes and the body cannot be parsed, the Failed to render chart error replaces the placeholder. If the placeholder remains after the agent stops responding, instruct the agent to emit one complete chart block with a closing fence. | | The agent pastes a table or raw JSON instead of a chart | The agent either did not tag the fence chart or wrote prose instead of a block. Add the fence rule and worked example to the system prompt. | ## [](#next-steps)Next steps - [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) --- # Page 15: Set Up GitHub OAuth Provider and MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/github-oauth-tutorial.md --- # Set Up GitHub OAuth Provider and MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up GitHub OAuth Provider and MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: github-oauth-tutorial page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: github-oauth-tutorial.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/github-oauth-tutorial.adoc description: Complete walkthrough for configuring GitHub as an OAuth provider and creating a user-delegated GitHub MCP server. page-topic-type: tutorial personas: agent_builder, platform_engineer learning-objective-1: Create a GitHub OAuth app and register it as an OAuth provider in Agentic Data Plane learning-objective-2: Test the OAuth connection proactively on the My Connections page learning-objective-3: Configure a GitHub managed MCP server with user-delegated OAuth page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-13" --- This tutorial shows how to connect Redpanda Agentic Data Plane to GitHub using OAuth, so users can authenticate with their own GitHub credentials when calling GitHub MCP tools. After completing this tutorial, you will be able to: - Create a GitHub OAuth app and register it as an OAuth provider in Agentic Data Plane - Test the OAuth connection proactively on the My Connections page - Configure a GitHub managed MCP server with user-delegated OAuth ## [](#before-you-begin)Before you begin - You need a GitHub account with permission to create OAuth apps (personal account or organization with appropriate role). - You need Agentic Data Plane administrator permissions to create OAuth providers and MCP servers. - You should understand the difference between user-delegated OAuth (each user authenticates as themselves) and service-account OAuth (all users share one identity). This tutorial covers user-delegated OAuth. > 💡 **TIP** > > GitHub recommends considering GitHub Apps instead of OAuth Apps for production use, as they offer fine-grained permissions and short-lived tokens. However, OAuth Apps work well for getting started with Redpanda Agentic Data Plane’s AI Gateway integration. ## [](#create-a-github-oauth-app)Create a GitHub OAuth app Register an OAuth application in GitHub to obtain the client ID and client secret. > 📝 **NOTE** > > Throughout this tutorial, fields marked with \* are required. 1. Sign in to GitHub and navigate to **Settings → Developer settings → OAuth Apps**. 2. Click **New OAuth App** (or **Register a new application** if this is your first OAuth app). 3. Fill in the application details: | Field | Value | | --- | --- | | Application name | Agentic Data Plane AI Gateway (or your preferred name) | | Homepage URL | https://aigw..clusters.rdpa.co+ This must match exactly. Replace with your actual cluster ID, such as https://aigw.d6kjl4h19241bg3ek3t0.cluster.rdpa.co. | | Application description | Optional. For example: "Redpanda AI Gateway GitHub integration". | | Authorization callback URL | https://aigw..clusters.rdpa.co/oauth/v1/callbackThis must match exactly. Replace with your actual cluster ID, such as https://aigw.d6kjl4h19241bg3ek3t0.cluster.rdpa.co/oauth/v1/callback. | 4. Click **Register application**. 5. On the application page, note your **Client ID** (for example, `Iv1.abc123…​`). Copy it for later. 6. Click **Generate a new client secret** and copy it immediately (you won’t see it again). > ❗ **IMPORTANT** > > - GitHub OAuth apps support only **one** callback URL (unlike GitHub apps which support multiple). > > - Store your client secret securely. GitHub will not show it again after you navigate away. ## [](#create-the-github-oauth-provider-in-agentic-data-plane)Create the GitHub OAuth provider in Agentic Data Plane Register GitHub as an OAuth provider in Agentic Data Plane. 1. Open **Integrations setup** in the sidebar and select the **Outbound providers** tab. 2. Click **Create provider** and select **GitHub** from the catalog. 3. Fill in the provider details: ### [](#provider-details)Provider details | Field | Value | Notes | | --- | --- | --- | | Name * | A name for this OAuth provider, for example, github. | Lowercase letters, numbers, and hyphens only. This is the machine identifier used when attaching the provider to an MCP server. Immutable after creation. | | Client ID * | The client ID from your GitHub OAuth app, for example, Iv1.abc123…​ | This is found in GitHub Settings → Developer settings → OAuth Apps → [Your App]. | | Display name * | GitHub | Human-readable name shown in Agentic Data Plane. | | Authorization endpoint * | https://github.com/login/oauth/authorize | GitHub’s OAuth authorization URL where users are redirected to grant permission. | | Token endpoint * | https://github.com/login/oauth/access_token | GitHub’s OAuth token exchange endpoint where authorization codes are exchanged for access tokens. | | Revocation Endpoint | Leave empty | GitHub doesn’t provide a standard RFC 7009 token revocation endpoint. Agentic Data Plane will handle token lifecycle without this. | | Client secret ref * | GITHUB_CLIENT_SECRET | Click New. Enter GITHUB_CLIENT_SECRET as the key name, paste in your client secret from GitHub, and then click Create secret. This creates the secret in the Secrets Store and links it to this provider in Agentic Data Plane. If you already have a secret in Secrets Store, click Existing instead. | ### [](#scopes)Scopes Define the GitHub scopes your MCP servers will need. Click in the **Scopes** field and add the following scopes: - `repo` - Full control of private repositories (read and write) - `read:user` - Read user profile information - `read:org` - Read organization membership - `workflow` - Update GitHub Actions workflows > 💡 **TIP** > > **Common GitHub OAuth scopes:** > > - `repo` - Access to private repositories > > - `public_repo` - Access to public repositories only > > - `read:user`, `user:email` - Read user profile and email > > - `read:org` - Read organization membership > > - `workflow` - Manage GitHub Actions workflows > > - `gist` - Create and read gists > > > See the [GitHub OAuth scopes documentation](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/scopes-for-oauth-apps) for the complete list. > > **Scope management**: Include every scope any MCP server attached to this provider might need. Users must re-consent when scopes are added later, so it’s better to include all necessary scopes upfront. ### [](#grant-types)Grant types - **Browser consent** ✓ (checked) - OAuth 2.0 Authorization Code flow. This is the standard flow for user-delegated OAuth. - **Token exchange** ☐ (unchecked) - RFC 8693 token exchange. Leave unchecked unless you’re implementing advanced token exchange scenarios. ### [](#token-endpoint-auth-method)Token endpoint auth method Select **Client Secret (Basic)** This sends credentials as `client_id:client_secret` in the HTTP Basic Authorization header, which is GitHub’s expected authentication method. | Method | When to use | | --- | --- | | Client Secret (Basic) | Most common. GitHub expects this method. Credentials sent in Authorization header. | | Client Secret (POST) | Credentials sent in the POST body. Use only if the provider requires it. | | None (PKCE only) | For public clients with no client secret. Not applicable for GitHub OAuth Apps. | ### [](#require-pkce)Require PKCE **Enabled** (toggle on) Proof Key for Code Exchange (PKCE) adds an extra layer of security to the OAuth flow. > 📝 **NOTE** > > PKCE (RFC 7636) protects against authorization code interception attacks. When enabled, the gateway generates a code challenge before redirecting the user to GitHub and verifies it when exchanging the authorization code for a token. ### [](#complete-creation)Complete creation 1. Review all fields to ensure they match the values above 2. Click **Create provider** The provider appears in the **Outbound providers** list with status "Enabled". ## [](#test-the-connection-on-the-my-connections-page)Test the connection on the My Connections page Before creating an MCP server, proactively test the OAuth flow to verify your provider configuration is correct. 1. Open **My Connections** in the sidebar. 2. Find **GitHub** in the connections rail, under **Available to connect**. 3. Click **Connect**. 4. You’ll be redirected to GitHub’s authorization page. 5. Review the requested permissions (scopes) and click **Authorize \[Your App Name\]**. 6. GitHub redirects you back to Agentic Data Plane. 7. GitHub now shows status "Connected". > 💡 **TIP** > > If the connection fails: > > - Check that the **Homepage URL** and **Authorization callback URL** in your GitHub OAuth app exactly match the expected URLs with your cluster ID. > > - Verify the **Client ID** and **Client secret ref** in the Agentic Data Plane OAuth provider are correct. > > - Check that the secret `GITHUB_CLIENT_SECRET` exists and contains the correct value. > > > See [Troubleshooting](#troubleshooting) for more debugging tips. ## [](#create-the-github-mcp-server)Create the GitHub MCP server Now create an MCP server that uses the GitHub OAuth provider for user-delegated authentication. 1. Open **MCP Servers** in the sidebar. 2. Click **Create server**. 3. The catalog shows managed MCP servers organized by category (AI, Databases, Streaming, Collaboration, AWS, Google, Utilities). 4. Use the search bar or browse to find and select **GitHub**. > 📝 **NOTE** > > GitHub is a managed MCP server, meaning Redpanda provides and maintains the implementation. You only need to configure authentication and select which tools to expose. ### [](#configure-the-github-mcp-server)Configure the GitHub MCP server Fill in the basic server details: | Field | Example value | Notes | | --- | --- | --- | | Name * | my-gh | Unique identifier. Lowercase letters, numbers, and hyphens only. | | Description | What this server does (shown in detail views) | Optional human-readable description. | | Code mode | Toggle on/off | Enable to add search and run tools that let agents run sandboxed code against this server’s tools. Typically enabled for development/testing. | | Restrict to org | redpanda-data | GitHub-specific field. If set, confines this MCP to the named GitHub org (for example, "redpanda-data"). Leave empty for unrestricted access to all repositories the user can access. | ### [](#configure-authentication)Configure authentication 1. In the **Auth** dropdown, select **User OAuth**. This enables user-delegated authentication where each user authenticates with their own GitHub credentials. 2. In the **User OAuth** section that appears: | Field | Value | Notes | | --- | --- | --- | | Provider name * | GitHub | Select the GitHub OAuth provider you created earlier from the dropdown. If "Not set" appears, you haven’t created a provider yet or it’s not visible. | | Required scopes | Click + Add required scopes | Add the same scopes you defined in the OAuth provider, adding each scope to a new line. If a user’s stored connection has fewer scopes than specified here, they’ll be prompted to re-consent with upgraded scopes. | ### [](#token-injection-advanced)Token injection (advanced) The **Injection** section controls how the resolved OAuth token is placed on upstream requests: | Field | Default value | Notes | | --- | --- | --- | | Header name | Authorization | HTTP header name for the token. GitHub expects Authorization, so keep the default. | | Header prefix | Bearer | Value prefix before the token. GitHub expects Bearer , so keep the default. Set empty to omit the prefix entirely (for upstreams that expect a bare API key as the token). | > 💡 **TIP** > > **How to authenticate to GitHub** link explains GitHub’s expected authentication format. For standard GitHub OAuth, the defaults (`Authorization: Bearer `) are correct. ### [](#complete-creation-2)Complete creation 1. Review all fields. 2. Click **Submit**. The GitHub MCP server appears in the **MCP Servers** list showing: - **Name**: `my-gh` (or your chosen name) - **Type**: Managed (badge) - **Status**: Enabled (badge) - **Tools**: A list of available tools like `get_authenticated_user`, `list_my_orgs`, `list_org_repos`, and more > ❗ **IMPORTANT** > > **User OAuth vs service account OAuth:** > > - **User OAuth** (what we’re configuring): Each end-user authenticates with their own GitHub account. The upstream GitHub API sees requests as coming from that individual user and applies their permissions. > > - **Service account OAuth**: All users share one GitHub identity. The upstream sees every request as coming from the same service account. > > > For most scenarios where users need to access their own repositories or respect GitHub’s per-user permissions, choose **User OAuth**. ## [](#verify-the-mcp-server-works)Verify the MCP server works Test that the MCP server is correctly configured and can authenticate users. 1. In Agentic Data Plane, find your GitHub MCP server in the **MCP Servers** list. 2. Click it to view details. 3. If you tested the connection in Connections, you’re already connected and can test immediately. 4. If you skipped the Connections test, the first time you call a tool from this server, you’ll be prompted to authorize GitHub. The flow is: 1. Call a tool (for example, through a Claude Desktop integration pointing to this MCP server). 2. Agentic Data Plane returns `OAuthConnectionRequired` with an authorization URL. 3. The UI prompts you to connect to GitHub. 4. You authorize in GitHub. 5. The tool call automatically retries and succeeds. 5. After successful authorization, subsequent tool calls use your stored GitHub token with no additional prompts (until the token expires or scopes change). ## [](#what-you-accomplished)What you accomplished You have now: - Created a GitHub OAuth app with the correct callback URL - Registered GitHub as an OAuth provider in Agentic Data Plane - Tested the OAuth connection proactively - Created a GitHub managed MCP server with user-delegated OAuth - Verified that users can authenticate and call GitHub tools You could next create an agent using this GitHub MCP server. Each user who calls tools from this MCP server will authenticate with their own GitHub account. User tokens are stored securely in Agentic Data Plane’s token vault and automatically refreshed before expiry. ## [](#troubleshooting)Troubleshooting | Symptom | Solution | | --- | --- | | redirect_uri_mismatch error during authorization | The callback URL in your GitHub OAuth app doesn’t match Agentic Data Plane’s callback. Go to GitHub Settings → Developer settings → OAuth Apps → [Your App] and verify the Authorization callback URL is exactly https://aigw..clusters.rdpa.co/oauth/v1/callback. | | invalid_client error during token exchange | The Client ID or Client secret is incorrect, or the authentication method doesn’t match. Verify the Client ID in the OAuth provider matches GitHub, and that GITHUB_CLIENT_SECRET contains the correct client secret. Ensure Token Endpoint Auth Method is set to "Client Secret (Basic)". | | invalid_scope error during consent | One of the scopes you requested isn’t valid for GitHub OAuth apps. Check the GitHub OAuth scopes documentation and remove or rename invalid scopes. | | "OAuth provider not found" when creating MCP server | The provider name doesn’t match. Verify you created the provider with name github (lowercase) and are selecting the correct provider in the MCP server configuration. | | Connection works in Connections but fails in MCP server | The MCP server’s required scopes might be different from what you consented to in Connections. Edit the OAuth provider to include all necessary scopes, then re-consent in Connections. | | Token expired or OAuthTokenExpired error | GitHub tokens eventually expire or can be revoked. The user needs to re-authorize in Connections or by triggering the consent flow again on the next tool call. | | Permission denied when attaching provider to MCP server | You need the dataplane_aigateway_oauthprovider_attach permission on the GitHub provider. Contact your Agentic Data Plane administrator to grant this permission. | ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 16: Choose an Integration Pattern **URL**: https://docs.redpanda.com/agentic-data-plane/connect/integration-overview.md --- # Choose an Integration Pattern > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Choose an Integration Pattern latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: integration-overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: integration-overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/integration-overview.adoc description: Choose the right integration pattern for connecting agents, pipelines, and external applications. page-topic-type: best-practices personas: agent_builder, platform_engineer learning-objective-1: Choose the integration pattern that fits your use case learning-objective-2: Apply appropriate authentication for internal versus external integration learning-objective-3: Select the right communication protocol for your integration scenario page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-06" --- Choose the right integration pattern for how your agents, pipelines, and external applications communicate. The pattern you pick determines who initiates the call, how data flows, and whether the interaction is synchronous or event-driven. After reading this page, you will be able to: - Choose the integration pattern that fits your use case - Apply appropriate authentication for internal versus external integration - Select the right communication protocol for your integration scenario ## [](#integration-scenarios)Integration scenarios Agentic Data Plane supports three primary integration scenarios based on who initiates the call and where the caller is located: | Scenario | Description | When to Use | Guide | | --- | --- | --- | --- | | Agent needs capabilities | Your agent invokes MCP tools to fetch data, call APIs, or access external systems on-demand | Agent-initiated, synchronous, interactive workflows | How MCP Servers Work | | Pipeline processes events | Your Redpanda Connect pipeline invokes agents for each event in a stream using the a2a_message processor | Event-driven, automated, high-volume stream processing | Integrate with Redpanda Pipelines | | External system calls agent | Your application or agent (hosted outside Agentic Data Plane) calls Agentic Data Plane agents using the A2A protocol | Backend services, CLI tools, custom UIs, multi-platform agent workflows | Connect Agents with A2A | ## [](#common-use-cases-by-pattern)Common use cases by pattern Each integration pattern serves different scenarios based on how data flows and who initiates the interaction. ### [](#agent-needs-capabilities)Agent needs capabilities (MCP tools) Use MCP tools when your agent needs on-demand access to data or capabilities. The agent decides when to invoke tools as part of its reasoning process. It waits for responses before continuing. This pattern works well for interactive workflows: customer support lookups, approval flows, or context-aware chatbots. Avoid MCP tools for high-volume stream processing or automated workflows without user interaction. Use pipeline-initiated integration instead. For implementation details, see [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/). ### [](#pipeline-processes-events)Pipeline processes events (`a2a_message`) Use the `a2a_message` processor when your pipeline needs to invoke agents for every event in a stream. The pipeline controls when agents execute. This pattern is ideal for automated, high-volume processing where each event requires AI reasoning. Common scenarios include real-time fraud detection, sentiment scoring for customer reviews, and content moderation that classifies and routes content. For implementation details, see [Integrate with Redpanda Pipelines](https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns/). ### [](#external-system-calls-agent)External system calls agent Use external integration when your applications, services, or agents hosted outside Agentic Data Plane need to call Agentic Data Plane agents. External systems send requests using the A2A protocol and receive responses synchronously. This works for backend services, CLI tools, custom UIs, and agents hosted on other platforms. Common scenarios include backend services analyzing data as part of workflows, CLI tools invoking agents for batch tasks, custom UIs displaying agent responses, CRM agents coordinating with Redpanda agents, and multi-platform workflows spanning different infrastructure. To learn how the A2A protocol enables this integration, see [Connect Agents with A2A](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/). ## [](#pattern-comparison)Pattern comparison The following table compares the two primary internal integration patterns: | Criterion | Agents Invoking MCP Tools | Pipelines Calling Agents | | --- | --- | --- | | Trigger | User question or agent decision | Event arrival in topic | | Frequency | Ad-hoc, irregular, as needed | Continuous, every event | | Latency | Low (agent waits for response) | Higher (async acceptable) | | Control Flow | Agent decides when to invoke | Pipeline decides when to invoke | | Use Case | "Fetch me data", "Run this query" | "Process this stream", "Enrich all events" | | Human in Loop | Often yes (user-driven) | Often no (automated) | ## [](#security-considerations-for-external-integration)Security considerations for external integration When integrating external applications with Agentic Data Plane agents, protect credentials and tokens. ### [](#protect-service-account-credentials)Protect service account credentials Store the client ID and secret in secure credential stores, not in code. Use environment variables or your organization’s secrets manager. Rotate credentials if compromised and restrict access based on the principle of least privilege. ### [](#protect-access-tokens)Protect access tokens Access tokens grant full access to the agent. Anyone with a valid token can send requests, receive responses, and consume agent resources (subject to rate limits). Treat access tokens like passwords and never log them or include them in error messages. ## [](#next-steps)Next steps - [Connect Agents with A2A](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - [Integrate with Redpanda Pipelines](https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns/) --- # Page 17: Integrate with Your Stack **URL**: https://docs.redpanda.com/agentic-data-plane/connect/integrations.md --- # Integrate with Your Stack > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Integrate with Your Stack latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: integrations page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: integrations.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/integrations.adoc description: Choose an integration pattern and connect the Agentic Data Plane to your existing stack, Redpanda pipelines, and developer tools. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-07-02" --- Connect the Agentic Data Plane to your existing systems. Choose an integration pattern, wire in Redpanda pipelines, and use Redpanda Agentic Data Plane from developer tools like Claude Code and Codex. - [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/) Choose the right integration pattern for connecting agents, pipelines, and external applications. - [Integrate with Redpanda Pipelines](https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns/) Build Redpanda Connect pipelines that invoke agents for event-driven processing and streaming enrichment. - [Run Claude Code and Codex through AI Gateway](https://docs.redpanda.com/agentic-data-plane/connect/coding-agents/) Launch Claude Code or Codex with rpk ai run, routing their model traffic through a managed, enterprise-passthrough, or AWS Bedrock Agentic Data Plane provider. --- # Page 18: BambooHR Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/bamboohr.md --- # BambooHR Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: BambooHR Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/bamboohr page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/bamboohr.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/bamboohr.adoc description: Let agents read BambooHR employee, time-off, and performance data with each caller's BambooHR OAuth identity. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the BambooHR managed MCP server with a BambooHR subdomain and user-delegated OAuth learning-objective-2: Identify the BambooHR tools available to agents learning-objective-3: Test BambooHR tool calls with an authorized user's connection page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-02" --- The **BambooHR** managed MCP server lets agents read BambooHR employee directory, time-off, and performance data with the calling user’s OAuth identity. The BambooHR configuration stores the company subdomain and the name of the BambooHR [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) to authenticate against. OAuth credentials and user tokens come from that provider and the [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault). After reading this page, you will be able to: - Configure the BambooHR managed MCP server with a BambooHR subdomain and user-delegated OAuth - Identify the BambooHR tools available to agents - Test BambooHR tool calls with an authorized user’s connection ## [](#what-this-mcp-server-does)What this MCP server does Use BambooHR when an agent needs HR context from BambooHR without sharing one upstream API key across every caller. Each caller authorizes with BambooHR, and tool calls run with the BambooHR permissions available to that caller. | Tool | What it does | | --- | --- | | get_employee | Fetches a single employee by ID. Use employee ID 0 to return the authenticated user’s own record. You can pass a comma-separated fields list, or leave it empty to return all fields the caller can access. | | get_employee_directory | Lists all employee directory entries visible to the authenticated user. | | get_time_off_requests | Queries time-off requests by date range. You can filter by employee ID, status, or time-off type. Supported statuses are approved, denied, superceded, requested, and canceled. | | whos_out | Returns absences and holidays in a date range. If you omit dates, the server starts with today and ends 14 days later. | | get_performance_feedback | Returns peer or manager feedback for an employee. You can filter by completed, pending, or all feedback. | ## [](#prerequisites)Prerequisites - A BambooHR workspace where you can create an OAuth 2.0 application. - A BambooHR OAuth Provider in Redpanda Agentic Data Plane. BambooHR OAuth authorize and token endpoints include your company subdomain. - A BambooHR company subdomain, such as `mycompany` for `mycompany.bamboohr.com`. The managed config accepts letters, numbers, and hyphens. - BambooHR users with the permissions required to read the employee, time-off, or performance data your agent needs. ## [](#configure-bamboohr-oauth)Configure BambooHR OAuth In BambooHR, create an OAuth 2.0 application in the BambooHR developer settings. Use the OAuth callback URL for your AI Gateway deployment. Register a BambooHR OAuth Provider in Agentic Data Plane with tenant-specific endpoints: ```text Authorization endpoint: https://api.bamboohr.com/api/gateway.php//v1/oauth2/authorize Token endpoint: https://api.bamboohr.com/api/gateway.php//v1/oauth2/token ``` Store the BambooHR client secret in the Agentic Data Plane secret store, then reference that secret from the OAuth Provider. ## [](#create-the-managed-mcp-server)Create the managed MCP server Create a managed MCP server with the BambooHR type. Set the `subdomain` field and reference the BambooHR OAuth Provider under the `userOauth` auth variant. BambooHR supports per-user OAuth only, so the `userOauth` block is required. The BambooHR managed config holds the company subdomain and the OAuth provider to authenticate against: ```json { "@type": "type.googleapis.com/redpanda.mcps.bamboohr.v1.BambooHRMCPConfig", "subdomain": "mycompany", "userOauth": { "providerName": "bamboohr" } } ``` `userOauth.providerName` is the resource name of the BambooHR OAuth Provider you registered. The provider’s stored per-user token is injected on each tool call. ## [](#authorize-and-test)Authorize and test Before a caller can use BambooHR tools, the caller must authorize the BambooHR OAuth Provider. After authorization, Redpanda stores the user’s access token in the token vault and subsequent tool calls use that token automatically. Test the server in the Inspector before you connect it to an agent. Start with `get_employee_directory` or `get_employee` with employee ID `0`, because both calls confirm that the user’s OAuth connection resolves and BambooHR returns data visible to that user. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | Tool calls return permission errors | Confirm the BambooHR user can access the requested HR data. Restricted BambooHR roles can return permission errors, especially for performance data. | | OAuth succeeds, but tool calls fail | Confirm the OAuth Provider endpoints use the same BambooHR subdomain as the MCP server configuration. | | The server cannot find the BambooHR company | Confirm subdomain contains only the part before .bamboohr.com, such as mycompany. | ## [](#next-steps)Next steps - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 19: Freshservice Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/freshservice.md --- # Freshservice Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Freshservice Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/freshservice page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/freshservice.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/freshservice.adoc description: Manage Freshworks Freshservice ITSM tickets, assets, change requests, and agents from an agent, with API-key or per-user OAuth authentication. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Freshservice managed MCP server in API-key or User-OAuth mode learning-objective-2: Find the Freshservice credentials each mode needs learning-objective-3: List, create, and update tickets, assets, and changes from the Inspector or an agent page-git-created-date: "2026-06-09" page-git-modified-date: "2026-07-29" --- The Freshservice managed MCP server lets agents work with your Freshworks Freshservice ITSM instance: list and search tickets, read full ticket detail, create and update tickets, browse CMDB assets, open change requests, and look up support agents. After reading this page, you will be able to: - Configure the Freshservice managed MCP server in API-key or User-OAuth mode - Find the Freshservice credentials each mode needs - List, create, and update tickets, assets, and changes from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Wraps the Freshservice REST API v2. The following authentication modes are supported: - API key (Basic authentication): A long-lived Freshservice API key sent as the HTTP Basic authentication username, with the literal string `X` as the password. Best for service-account-style use. - User OAuth: Per-user Freshservice OAuth tokens resolved from the gateway’s [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault). Best when you want each agent action attributed to the calling end-user. Responses are curated for token efficiency: high-volume fields (routing headers, internal SLA sub-deadlines, spam and email-config flags, agent scoreboards) are dropped before reaching the LLM, and ticket and change descriptions are returned as plain text rather than HTML. Freshservice error response bodies are never forwarded to the caller; only the HTTP status code is surfaced, so API keys and internal request IDs cannot leak into tool error text. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Freshservice instance and its domain (for example, `mycompany.freshservice.com`). - For API-key mode: the API key from your Freshservice profile (see [Get Freshservice credentials](#get-freshservice-credentials)). - For User-OAuth mode: a Freshservice OAuth app and an [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). ## [](#get-freshservice-credentials)Get Freshservice credentials ### [](#option-1-api-key-recommended-for-service-accounts)Option 1: API key (recommended for service accounts) 1. In Freshservice, open **Profile Settings**. 2. Copy the value from the **API Key** section. 3. Store the key in the Agentic Data Plane secret store under a name like `FRESHSERVICE_API_KEY`. The MCP sends the key as the HTTP Basic authentication username with `X` as the password (`base64(:X)`), which is the Freshservice convention. The API key inherits the permissions of the Freshservice user it belongs to, so use an account with the roles your workflows need (agent, asset, and change permissions). ### [](#option-2-user-oauth)Option 2: User OAuth For per-user authentication, register an OAuth app on Freshservice and a matching OAuth Provider in Agentic Data Plane: 1. Register a Freshservice OAuth app at the Freshworks developer portal to obtain a client ID and client secret. 2. Register a matching OAuth Provider in Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). The `freshservice` preset pre-fills the authorization and token endpoints; Agentic Data Plane substitutes your Freshservice subdomain into the `{domain}` placeholders. 3. Each end-user authenticates once through the OAuth flow; tokens are stored in the gateway’s token vault. The OAuth scopes are `read` and `write`. Drop `write` if the MCP only needs to read. ## [](#configure)Configure Create a new Freshservice MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Freshservice` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Freshservice configuration form: | Field | Notes | | --- | --- | | domain | Your Freshservice host, without the https:// scheme or a trailing slash (for example, mycompany.freshservice.com). | | auth | api_key for service-account mode, or user_oauth for per-user mode. | | api_key (API-key mode) | Secret-store reference holding the API key (for example, FRESHSERVICE_API_KEY). UPPER_SNAKE_CASE. | | user_oauth (User-OAuth mode) | The OAuth Provider you configured, and the minimum scopes a user’s connection must have. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI For a managed server, set the auth method inside the `--managed-config` JSON. The `auth` field is required. #### API-key mode ```bash rpk ai mcp-server create --name acme-freshservice --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.freshservice.v1.FreshServiceMCPConfig", "domain": "mycompany.freshservice.com", "api_key": { "key_secret_ref": "FRESHSERVICE_API_KEY" } }' ``` #### User-OAuth mode ```bash rpk ai mcp-server create --name acme-freshservice-oauth --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.freshservice.v1.FreshServiceMCPConfig", "domain": "mycompany.freshservice.com", "user_oauth": { "provider_name": "freshservice-prod", "required_scopes": ["read", "write"] } }' ``` Replace `freshservice-prod` with the name of the OAuth Provider you configured. ## [](#tools)Tools The Freshservice MCP exposes tools across tickets, assets, changes, and agents: | Tool | Description | | --- | --- | | list_tickets | List tickets, with optional status, priority, and page filters. | | get_ticket | Fetch one ticket with its full description text and attachments. | | create_ticket | Open a new incident or service-request ticket. | | update_ticket | Update ticket fields such as status, priority, and assignee. | | list_assets | List CMDB assets, with optional type and page filters. | | get_asset | Fetch one asset with its dynamic type_fields. | | create_change | Open a new change request with planning fields. | | list_agents | List support agents, with optional active and email filters. | ### [](#example-list-open-tickets)Example: List open tickets ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-freshservice \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "list_tickets", "arguments": { "status": "open", "priority": "urgent" } } }' ``` Replace `` with your cluster ID and `$TOKEN` with a gateway access token. ### [](#example-create-a-ticket)Example: Create a ticket ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-freshservice \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "create_ticket", "arguments": { "subject": "Laptop will not boot", "description": "Reported by the Dresden office; powers on but no display.", "priority": 2, "status": 2 } } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | freshservice API error (status 401) | Confirm FRESHSERVICE_API_KEY matches the value from Profile Settings, and that the key’s Freshservice user has the roles the operation needs. | | freshservice API error (status 403) | The API key’s Freshservice user lacks permission for the resource (for example, assets or changes). Grant the role or use an account that has it. | | OAuthConnectionRequired (User-OAuth mode) | First call from a user with no stored token. The user completes the Freshservice OAuth consent flow, the token lands in the vault, and later calls reuse it. See User-delegated OAuth. | | scope_upgrade_required (User-OAuth mode) | The server’s required scopes were extended after users consented. Users re-consent with the higher scope. | > 📝 **NOTE** > > Freshservice error bodies are not forwarded to the caller, so tool errors carry only the HTTP status code. Reproduce the call against the Freshservice API directly to see the full error detail. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 20: Grafana Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/grafana.md --- # Grafana Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Grafana Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/grafana page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/grafana.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/grafana.adoc description: Query Grafana dashboards, datasources, Prometheus metrics, and Loki logs from an agent, and create or update dashboards, using a service-account token. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Grafana managed MCP server with a service-account token learning-objective-2: Pick the right service-account role for read versus write workflows learning-objective-3: Search dashboards, query Prometheus and Loki, and upsert dashboards from the Inspector or an agent page-git-created-date: "2026-06-09" page-git-modified-date: "2026-07-29" --- The Grafana managed MCP server lets agents work with a Grafana instance: search dashboards and folders, list datasources, run PromQL queries against Prometheus datasources, run LogQL queries against Loki datasources, and create or update dashboards. After reading this page, you will be able to: - Configure the Grafana managed MCP server with a service-account token - Pick the right service-account role for read versus write workflows - Search dashboards, query Prometheus and Loki, and upsert dashboards from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Wraps Grafana’s HTTP API and proxies Prometheus and Loki queries through Grafana’s datasource proxy (`/api/datasources/proxy/uid/{uid}/…​`). A single base URL plus one service-account token covers both the core API and datasource queries, so the gateway makes no separate network connection to the metric or log backends. The tool surface is read-heavy. The one write tool is `create_or_update_dashboard`; there are no tools to mutate alerts or annotations. Authentication is a single shared service-account token, so every caller of this MCP server acts as that service account, and there is no per-user audit trail. Grafana exposes no OAuth client-registration flow for per-user delegation against an arbitrary instance, so the Grafana MCP server does not offer a User-OAuth mode. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Grafana instance and its base URL (for example, `[https://myorg.grafana.net](https://myorg.grafana.net)`). - Permission to create a service account and a service-account token in Grafana. ## [](#get-grafana-credentials)Get Grafana credentials 1. In Grafana, go to **Administration > Users and access > Service accounts**. 2. Create a service account. Assign the Viewer role to use every read tool. To use `create_or_update_dashboard`, assign Editor (or a custom role with dashboard write permission) instead. 3. On the service account, select **Add service account token**, then copy the generated token (`glsa_…​`). It is shown only once. 4. Store the token in the Agentic Data Plane secret store under a name like `GRAFANA_SERVICE_ACCOUNT_TOKEN`. Note your instance base URL and, if your instance is multi-org, the numeric org ID. ## [](#configure)Configure Create a new Grafana MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Grafana` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Grafana configuration form: | Field | Notes | | --- | --- | | base_url | Base URL of your Grafana instance (for example, https://myorg.grafana.net). | | org_id | Optional. The numeric organization ID for a multi-org instance. Omit it to use the token’s default organization. | | service_account_token | Secret-store reference holding the service-account token (for example, GRAFANA_SERVICE_ACCOUNT_TOKEN). UPPER_SNAKE_CASE. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI ```bash rpk ai mcp-server create --name acme-grafana --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.grafana.v1.GrafanaMCPConfig", "base_url": "https://myorg.grafana.net", "org_id": "1", "service_account_token": { "key_secret_ref": "GRAFANA_SERVICE_ACCOUNT_TOKEN" } }' ``` ## [](#tools)Tools The Grafana MCP exposes tools across dashboards, datasources, Prometheus, and Loki: | Tool | Description | | --- | --- | | search_dashboards | Search dashboards and folders by query (title substring), tags, and type (dash-db or dash-folder). Returns curated hits (uid, title, type, tags, url, folder). | | get_dashboard_summary | Fetch a curated summary of one dashboard by uid (title, tags, folder, and a compact per-panel list with each panel’s id, title, type, and datasource). | | list_datasources | List configured datasources (uid, name, type, is_default). Use the uid with the query tools. | | query_prometheus | Run an instant or range PromQL query against a Prometheus datasource (datasource_uid, expr, query_type, and time arguments). Returns the raw Prometheus result JSON. | | list_prometheus_metric_names | List metric names in a Prometheus datasource (datasource_uid, optional limit). | | list_prometheus_label_names | List label names in a Prometheus datasource (datasource_uid). | | list_prometheus_label_values | List the values of one label in a Prometheus datasource (datasource_uid, label). | | query_loki_logs | Run a LogQL query against a Loki datasource over a time range (datasource_uid, query, start, end, limit, direction). Returns the raw Loki result JSON. | | list_loki_label_names | List label names in a Loki datasource (datasource_uid, optional start and end). | | list_loki_label_values | List the values of one label in a Loki datasource (datasource_uid, label, optional start and end). | | create_or_update_dashboard | Create a dashboard or update an existing one (upsert on the model’s uid). Arguments: dashboard_json, folder_uid, overwrite, message. Requires an Editor service account. | ### [](#example-list-datasources)Example: List datasources ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-grafana \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "list_datasources", "arguments": {} } }' ``` Replace `` with your cluster ID and `$TOKEN` with a gateway access token. ### [](#example-run-an-instant-promql-query)Example: Run an instant PromQL query Find the Prometheus datasource `uid` with `list_datasources` first, then query it: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-grafana \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "query_prometheus", "arguments": { "datasource_uid": "PBFA97CFB590B2093", "expr": "up", "query_type": "PROM_QUERY_TYPE_INSTANT" } } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | 401 Unauthorized | Confirm GRAFANA_SERVICE_ACCOUNT_TOKEN matches the token Grafana showed at creation, and that the service account is enabled. | | 403 Forbidden on create_or_update_dashboard | The service account has the Viewer role. Assign Editor (or a role with dashboard write permission). | | Wrong organization’s data | The token resolves to its default organization. Set the org_id field to target a specific organization on a multi-org instance. | ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 21: Ironclad Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/ironclad.md --- # Ironclad Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Ironclad Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/ironclad page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/ironclad.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/ironclad.adoc description: Read and manage contracts in Ironclad CLM. Per-user OAuth so each agent action runs as the calling end-user with their own Ironclad permissions. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Ironclad managed MCP server with per-user OAuth learning-objective-2: Pick the right region and scopes for your tenant learning-objective-3: List, fetch, and launch contract workflows from an agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-29" --- The **Ironclad** managed MCP server gives an LLM read and write access to [Ironclad](https://ironcladapp.com/), a contract lifecycle management (CLM) platform. Useful for agents that need to find contracts, check signature status, launch new contracts from templates, or retrieve executed documents. After reading this page, you will be able to: - Configure the Ironclad managed MCP server with per-user OAuth - Pick the right region and scopes for your tenant - List, fetch, and launch contract workflows from an agent ## [](#what-this-mcp-server-does)What this MCP server does Per-user OAuth 2.0 (Authorization Code grant). No static API key is stored in the MCP config; each user authorizes their own Ironclad account through AI Gateway’s OAuth flow. It is **not** a replacement for the Ironclad web UI for complex workflow management or template authoring. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - An Ironclad tenant where you can register an OAuth app. - An [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane for Ironclad. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). - Familiarity with [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#get-ironclad-credentials)Get Ironclad credentials Set up the OAuth app on Ironclad and the matching OAuth Provider in Agentic Data Plane: 1. Log in to your Ironclad account and go to **Settings > API > OAuth Apps**. 2. Create a new OAuth app. Set the redirect URI to your AI Gateway callback URL (typically `https://aigw..clusters.rdpa.co/oauth/v1/callback`). 3. Select the following scopes: - `public.workflows.readWorkflows` - `public.workflows.readSchemas` - `public.workflows.createWorkflows` - `public.workflows.readDocuments` 4. Copy the `Client ID` and `Client Secret`. 5. In Agentic Data Plane, register an OAuth Provider with: - **Authorization endpoint**: `https://na1.ironcladapp.com/oauth/authorize` (use `eu1` for EU-hosted accounts; `demo` for sandbox) - **Token endpoint**: `https://na1.ironcladapp.com/oauth/token` (adjust region accordingly) - The Client ID and a secret-store reference for the Client Secret ## [](#configure)Configure Create a new Ironclad MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Ironclad` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Ironclad configuration form: | Field | Notes | | --- | --- | | region | IRONCLAD_REGION_NA (default), IRONCLAD_REGION_EU for EU-hosted accounts, or IRONCLAD_REGION_DEMO for sandbox testing. | | OAuth Provider | The Ironclad OAuth Provider you configured. | | Required scopes | public.workflows.readWorkflows, public.workflows.readSchemas, public.workflows.createWorkflows, public.workflows.readDocuments. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI Ironclad authenticates per-user through the Ironclad OAuth Provider you configured. Reference that provider in the managed config’s `userOauth.providerName` field. Ironclad supports per-user OAuth only, so the `userOauth` block is required. The `--user-oauth-provider` and `--user-oauth-scopes` flags apply to remote (`--url`) servers only. ```bash rpk ai mcp-server create --name my-ironclad --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.ironclad.v1.IroncladMCPConfig", "region": "IRONCLAD_REGION_NA", "userOauth": { "providerName": "ironclad" } }' ``` For EU-hosted accounts, use `"IRONCLAD_REGION_EU"`. For sandbox testing, use `"IRONCLAD_REGION_DEMO"`. `userOauth.providerName` is the resource name of the Ironclad OAuth Provider you registered. ## [](#tools)Tools The Ironclad MCP exposes the following tools: | Tool | Description | | --- | --- | | list_workflows | List contracts with optional status filter and page/per_page pagination. | | get_workflow | Get full details of a contract by workflow_id. | | list_workflow_schemas | List available contract templates with their field IDs. | | create_workflow | Launch a new contract from a template (schema_id + attributes_json). | | list_workflow_documents | List documents attached to a contract. | ### [](#example-find-all-contracts-awaiting-signature)Example: Find all contracts awaiting signature ```bash curl -X POST https://aigw..clusters.rdpa.co/mcp/v1/my-ironclad \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "list_workflows", "arguments": {"status": "signing", "per_page": 10} } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | OAuthConnectionRequired | First call from a user with no stored token. The user completes Ironclad’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. | | scope_upgrade_required | Server’s required_scopes was extended after users had already consented. Users re-consent with the higher scope. | | Wrong region results | Confirm the Region field matches your Ironclad tenant. EU-hosted accounts use IRONCLAD_REGION_EU; sandbox tenants use IRONCLAD_REGION_DEMO. | | schema_id not found in create_workflow | Run list_workflow_schemas first to get valid IDs for your tenant. | ## [](#limitations)Limitations This page does not cover: - **Template authoring**: Define templates in the Ironclad web UI, then reference them by `schema_id`. - **eSignature flows**: Handled inside Ironclad; this MCP launches and reads workflow state. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 22: Jira Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/jira.md --- # Jira Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Jira Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/jira page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/jira.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/jira.adoc description: Let agents search, read, and manage Jira issues using either a shared API token or each end-user's own Atlassian identity through user-delegated OAuth. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Jira managed MCP server with Basic authentication or Atlassian's OAuth flow learning-objective-2: Pick the right scopes for the tools your agents use learning-objective-3: Walk a user through the consent flow and verify the connection page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Jira** managed MCP server lets agents search, read, and manage Jira issues. It authenticates with either a shared API token (Basic authentication) or, for per-user identity, user-delegated OAuth. With OAuth it’s the enterprise counterpart to [the Slack setup guide](https://docs.redpanda.com/agentic-data-plane/connect/managed/slack/), and Atlassian’s flow has its own scope model and quirks worth calling out. After reading this page, you will be able to: - Configure the Jira managed MCP server with Basic authentication or Atlassian’s OAuth flow - Pick the right scopes for the tools your agents use - Walk a user through the consent flow and verify the connection ## [](#what-this-mcp-server-does)What this MCP server does The Jira managed type exposes the following read tools: - `query`: Search for issues using a JQL query. Returns the matching issues as JSON. - `get_issue`: Retrieve a single issue by its key, for example `PROJ-123`. - `list_projects`: Return visible Jira projects, optionally filtered by a search query. - `get_transitions`: Return the available workflow transitions for an issue. It also exposes the following write tools: - `create_issue`: Create a new issue. - `update_issue`: Update fields on an existing issue. - `transition_issue`: Move an issue to a new workflow state. - `add_comment`: Add a comment to an issue. - `add_worklog`: Log time spent on an issue. - `create_issue_link`: Create a directional link between two issues. ## [](#prerequisites)Prerequisites - A Jira (Atlassian Cloud) site and its base URL, for example `[https://mycompany.atlassian.net](https://mycompany.atlassian.net)`. - Credentials for one of the following authentication modes: - `Basic auth`: An Atlassian account email and an API token from `id.atlassian.com`, with the token stored in the Redpanda Agentic Data Plane secret store. - `User-delegated OAuth`: An Atlassian OAuth 2.0 (3LO) app registered against `[https://api.atlassian.com](https://api.atlassian.com)`, plus an [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) in Redpanda Agentic Data Plane configured for Atlassian’s authorize/token URLs and carrying the app’s client credentials. See [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#atlassians-scope-model-user-delegated-oauth)Atlassian’s scope model (user-delegated OAuth) When you use user-delegated OAuth, Atlassian uses a granular, prefixed scope namespace. The tools need: | Scope | Allows | | --- | --- | | read:jira-user | Read user profile. | | read:jira-work | Read issues, projects, sprints, and so on. Required by the read tools. | | write:jira-work | Create and update issues, transitions, comments, and worklogs. Required by the write tools. | | offline_access | Issue a refresh token so Redpanda can refresh expired access tokens. Required for any long-lived MCP server: without it, tokens expire after one hour and users re-consent every time. | > 📝 **NOTE** > > Always include `offline_access` in `required_scopes`. Without it, `OAuthTokenExpired` will hit users every hour. ## [](#configure)Configure 1. Open **MCP Servers > Create Server**. 2. Pick **Jira** from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Jira configuration form: - `Base URL`: Your Jira instance URL, for example `[https://mycompany.atlassian.net](https://mycompany.atlassian.net)`. - `Max results per page` (optional): Page size for paginated queries. Defaults to `50`. - `Auth`: Choose `Basic auth` or `User-delegated OAuth`. - For `Basic auth`: Provide the account email and the API-token secret reference. - For `User-delegated OAuth`: Pick the Atlassian OAuth Provider you configured, and set `Required scopes` to at least `read:jira-user`, `read:jira-work`, and `offline_access`. Add `write:jira-work` if your agents use the write tools. 5. Click **Create**. ## [](#test-the-consent-flow)Test the consent flow 1. Open the **Inspector** tab. 2. Run an issue-search tool with a small JQL filter. 3. The first call returns `OAuthConnectionRequired` with an Atlassian `authorize_url`. The Inspector surfaces it as a consent prompt. 4. Click **Authorize**. Atlassian asks you to pick a site (Cloud instance) and approve scopes. 5. Atlassian redirects back. Your connection appears under **My Connections** with a site label. 6. Re-run the search; results come back. ## [](#use-with-agents)Use with agents Point an agent at the `Server URL` on the server’s **Connection** tab. Each user calling the agent will trigger their own consent flow on first call. With user-delegated OAuth, define the server’s `required_scopes` to include every scope the tools need: Atlassian doesn’t allow per-tool scope upgrades, so the user consents once with the full set. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | OAuthTokenExpired after about an hour | offline_access wasn’t in required_scopes at consent time. Update the server config and have users re-consent. | | "Resource not found" for a project the user has access to | Atlassian’s OAuth grants are site-scoped. The user authorized for one Cloud instance; the project lives on another. They need to re-consent with the second site. | | scope_upgrade_required after widening scopes | You added a scope to required_scopes after users had already consented. Users re-consent with the higher scope. | | invalid_grant during refresh | Refresh tokens expire if unused for ~90 days. The user re-consents. | ## [](#limitations)Limitations - **Atlassian app management**: The OAuth app and its callback URLs are managed in `developer.atlassian.com`, not in Agentic Data Plane. - **Jira Server / Data Center** (self-hosted): This MCP type targets Atlassian Cloud. Self-hosted Jira may need a self-managed MCP server instead. See [Register a self-managed MCP server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/). - **Confluence access**: Separate scope namespace; not exposed by this MCP server. ## [](#next-steps)Next steps - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 23: Kafka Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/kafka.md --- # Kafka Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Kafka Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/kafka page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/kafka.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/kafka.adoc description: Produce messages to topics on Kafka or Redpanda brokers through a managed MCP server hosted in the Agentic Data Plane. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Kafka managed MCP server against a Kafka or Redpanda cluster learning-objective-2: Produce a test message through the Inspector learning-objective-3: Pick the right SASL mechanism for your broker (PLAIN or SCRAM-SHA-256/512) page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Kafka** managed MCP server lets agents produce messages to topics on either an Apache Kafka cluster or a Redpanda cluster. Despite the name, it works against any Kafka-compatible broker. After reading this page, you will be able to: - Configure the Kafka managed MCP server against a Kafka or Redpanda cluster - Produce a test message through the Inspector - Pick the right SASL mechanism for your broker (PLAIN or SCRAM-SHA-256/512) ## [](#what-this-mcp-server-does)What this MCP server does The Kafka managed type proxies a managed Kafka producer. It exposes the following tool: - `produce`: Send a message to a Kafka topic, with an optional key, an optional partition, and optional headers. Consuming messages, listing topics, and inspecting metadata are not currently exposed by this managed type. To read from topics, use a [self-managed MCP server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/) or Redpanda Connect. ## [](#prerequisites)Prerequisites - A Kafka or Redpanda cluster reachable from the Agentic Data Plane. - The cluster’s bootstrap servers and SASL/TLS settings. - For SCRAM or PLAIN: Secrets in the Redpanda Agentic Data Plane secret store for the username and password (`UPPER_SNAKE_CASE`, for example `KAFKA_SASL_USER` and `KAFKA_SASL_PASSWORD`). ## [](#configure)Configure 1. Open **MCP Servers > Create Server**. 2. Pick **Kafka** from the marketplace picker. 3. Fill in the identity fields (`name`, `description`). 4. In the Kafka configuration form, provide: - `Seed brokers`: Bootstrap broker addresses for the cluster. - `TLS`: Enable for production. You can optionally skip certificate verification for local development only. - `SASL mechanism`: `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`. Leave empty to disable SASL. - `Username / password`: `UPPER_SNAKE_CASE` secret references for the SASL credentials. - `Compression` (optional): Defaults to `lz4`. - `Max in-flight` (optional): Maximum number of in-flight produce requests. Defaults to `10`. - `Timeout` (optional): Produce request timeout. Defaults to `10s`. 5. Click **Create**. ## [](#test)Test 1. Open the **Inspector** tab on the server’s detail page. 2. In **Tools**, select the `produce` tool. 3. Produce a test message to a sandbox topic and confirm a successful response. 4. Verify the message landed by consuming the topic with `rpk topic consume` or another Kafka client. See [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) for general Inspector usage. ## [](#authentication)Authentication The Kafka managed type’s authentication is part of its config, not the generic MCP authentication modes: it uses Kafka protocol authentication (SASL over TLS), not MCP authentication. | Mechanism | Use when | | --- | --- | | PLAIN | Username and password over TLS. Common for managed Kafka services. | | SCRAM-SHA-256 / SCRAM-SHA-512 | Salted challenge-response. Default for Redpanda. | TLS controls transport encryption: enable it, and optionally skip certificate verification for development. Client-certificate (mTLS) and `OAUTHBEARER` authentication are not currently supported by this managed type. ## [](#use-with-agents)Use with agents After the Kafka server is created, point an agent at the `Server URL` on the server’s **Connection** tab. The agent can then produce messages to topics through the exposed `produce` tool. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | connection refused or dial timeout | Brokers aren’t reachable from Agentic Data Plane egress. Confirm bootstrap addresses and any private-network requirements. | | SASL authentication failed | Check username/password reference content and the SASL mechanism. | | TLS handshake error | Certificate chain isn’t trusted, or you’ve enabled TLS against a plaintext broker. Confirm broker config. | | unknown topic or partition | The target topic doesn’t exist on the broker. Create it first, or confirm the topic name. | ## [](#limitations)Limitations - **Reading and administration**: This server only produces messages. Consuming, listing topics, and managing topics or ACLs aren’t exposed. Use rpk, a Kafka client, or Redpanda Connect. - **Schema registry**: Not exposed by this MCP server. - **Streaming joins or processing**: For stream processing, use Redpanda Connect. --- # Page 24: Plug in an App, Database, or Tool **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog.md --- # Plug in an App, Database, or Tool > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Plug in an App, Database, or Tool latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/managed-catalog page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/managed-catalog.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/managed-catalog.adoc description: Reference of every managed MCP server type Redpanda hosts in-process, grouped by category, with display name, description, and a link to a setup guide where one exists. page-topic-type: reference personas: agent_builder, platform_engineer page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- Managed MCP servers are in-process implementations Redpanda hosts for you. Each type has a fixed set of [tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool) and a type-specific configuration schema. To create one, open **MCP Servers > Create Server** in Redpanda Agentic Data Plane and pick the type from the marketplace picker. See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) for the full create flow. This catalog lists the managed MCP server types Redpanda registers by default, grouped by category. ## [](#choosing-managed-vs-self-managed)Choosing managed vs. self-managed | Question | Choose managed when… | | --- | --- | | Who hosts and operates the server? | You want Redpanda to host it. | | Where do upstream credentials live? | You’re happy storing them in the Agentic Data Plane secret store. | | Do you need custom tool logic? | The fixed tool set of the managed type covers your use case. | | Do you need per-user identities at the upstream? | The managed type supports user-delegated OAuth for that system. | If any of these answers are "no," prefer [a self-managed server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/) instead. ## [](#ai)AI | Display name | Description | Setup guide | | --- | --- | --- | | AWS Bedrock (Preview) | Invoke foundation models (Claude, Llama, Titan) and generate embeddings on AWS Bedrock. | | | Cohere (Preview) | Generate embeddings and chat completions with the Cohere API. | | | OpenAI (Preview) | Chat completions, embeddings, and tools with the OpenAI API. | | ## [](#aws)AWS | Display name | Description | Setup guide | | --- | --- | --- | | AWS S3 (Preview) | Read, write, and manage objects in Amazon S3 buckets. | | | AWS SNS (Preview) | Publish and manage topics on AWS Simple Notification Service. | | | AWS SQS (Preview) | Send, receive, and manage messages on AWS Simple Queue Service. | | ## [](#communication)Communication | Display name | Description | Setup guide | | --- | --- | --- | | Discord (Preview) | Post messages, read channels, and manage servers on Discord. | | | Freshservice (Preview) | Manage ITSM tickets, assets, and change requests in Freshworks Freshservice. | Setup guide | | GitHub (Read) | Read-only access to GitHub repositories, pull requests, commits, and code. | Setup guide | | Jira (Preview) | Manage Jira issues, projects, and workflows. | Setup guide | | Pylon (Preview) | Search and manage Pylon issues, accounts, contacts, and knowledge-base articles. | | | ServiceNow (Preview) | Create and manage ServiceNow incidents, knowledge articles, and records. | Setup guide | | Slack (Preview) | Post messages and read channels on Slack. | Setup guide | | Zendesk (Preview) | Search and manage Zendesk Support tickets, users, and Help Center articles. | Setup guide | ## [](#database)Database | Display name | Description | Setup guide | | --- | --- | --- | | Elasticsearch (Preview) | Query and index documents in an Elasticsearch cluster. | | | Metabase (Preview) | Search tables and metrics, run queries, and inspect saved questions in Metabase. | Setup guide | | MongoDB (Preview) | Query collections and documents in MongoDB. | | | Qdrant (Preview) | Vector search over a Qdrant collection. | | | Redis (Preview) | Read, write, and query keys in Redis. | | | SQL (Preview) | Query SQL databases (Postgres, MySQL, ClickHouse, MSSQL, SQLite) with MCP. | Setup guide | ## [](#google)Google | Display name | Description | Setup guide | | --- | --- | --- | | GCP Pub/Sub (Preview) | Publish and subscribe to topics on Google Cloud Pub/Sub. | | | Gmail (Preview) | Read and send emails through Gmail with a service account or per-user OAuth. | | | Google Calendar | Read and manage Google Calendar events and schedules. | | | Google Drive | Read and search files in Google Drive. | | ## [](#streaming)Streaming | Display name | Description | Setup guide | | --- | --- | --- | | Kafka (Preview) | Produce, consume, and inspect topics on Kafka or Redpanda brokers. | Setup guide | | NATS (Preview) | Publish and subscribe on NATS and NATS JetStream. | | ## [](#utility)Utility | Display name | Description | Setup guide | | --- | --- | --- | | Azure AD (Preview) | Manage Azure Active Directory users and groups with Microsoft Graph API. | | | BambooHR | Access employee directory, time-off, and performance data from BambooHR. | Setup guide | | BILL (bill.com) (Preview) | Manage accounts-payable bills, vendors, AR invoices, payments, and customers in BILL. | | | DocuSign (Preview) | Send, track, and manage DocuSign signing envelopes. | | | Grafana (Preview) | Query Grafana dashboards, datasources, Prometheus metrics and Loki logs, and create or update dashboards. | Setup guide | | Greenhouse (Preview) | Manage jobs, candidates, and applications in Greenhouse ATS. | | | Ironclad (Preview) | Read and manage contracts in Ironclad CLM. | Setup guide | | Morningstar Portfolio Analytics (Preview) | Run X-Ray, performance, risk-score, hypothetical, and ESG analytics on caller-supplied portfolios with Morningstar Direct Web Services. | | | Morningstar Securities (Preview) | Search, retrieve, screen, and pull research on funds and equities with the Morningstar Direct Web Services Investment Analysis API. | | | NetSuite (Preview) | Query Oracle NetSuite records and run SuiteQL with the SuiteTalk REST API. | Setup guide | | Okta (Preview) | Manage Okta users and groups. | | | OpenAPI (Preview) | Expose any OpenAPI/Swagger HTTP API as MCP tools. | Setup guide | | Ramp (Preview) | Manage Ramp corporate cards, transactions, spend limits, and reimbursements. | Setup guide | | Salesforce | Query, create, update, and delete Salesforce CRM records using SOQL and the REST API, and run and inspect saved Salesforce reports. | Setup guide | | Sentry (Preview) | Read-only access to Sentry organizations, projects, issues, events, and releases for diagnosing and fixing errors. | Setup guide | | SharePoint (Preview) | Access SharePoint sites, document libraries, lists, and files through the Microsoft Graph API. | Setup guide | | Text Chunker (Preview) | Split and chunk text for RAG and LLM ingestion pipelines. | | | Workday (Preview) | Drive Workday Human Resources business processes with SOAP. | Setup guide | ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) --- # Page 25: Metabase Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/metabase.md --- # Metabase Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Metabase Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/metabase page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/metabase.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/metabase.adoc description: Give an LLM agent read-and-query access to a Metabase analytics instance through the semantic layer (search, get_table, query metrics) and SQL escape hatches (run_native_query, run_card). page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Metabase managed MCP server with a v0.49+ API key bound to the right Metabase group learning-objective-2: Identify when to use semantic-layer tools (search, get_metric, query) versus the native-SQL escape hatches (run_native_query, run_card) learning-objective-3: Run a metric-first analysis against your Metabase instance from the Inspector or an agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Metabase** managed MCP server lets agents read and query a Metabase analytics instance. The tool surface is a hybrid of Metabase’s semantic-layer primitives (tables, metrics, fields, dimensions) and SQL-first escape hatches (native queries and saved questions), wrapping the Metabase REST API with a long-lived API key. After reading this page, you will be able to: - Configure the Metabase managed MCP server with a v0.49+ API key bound to the right Metabase group - Identify when to use semantic-layer tools (`search`, `get_metric`, `query`) versus the native-SQL escape hatches (`run_native_query`, `run_card`) - Run a metric-first analysis against your Metabase instance from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Metabase is a self-service analytics platform that exposes data through a **semantic layer** (tables, metrics, fields) plus a SQL editor for ad-hoc native queries. This MCP wraps the Metabase REST API so an LLM can: - Search the semantic layer for relevant tables, metrics, saved questions, and dashboards. - Retrieve a table or metric’s structure (fields, dimensions, related tables). - Run a structured query against a metric or table with filters, group-by, and limits. - Run a saved question (card) or a hand-written native SQL query. The agent’s preferred path is **metric-first**: discover what already exists, fetch the metric’s definition, and call `query` against it so the aggregation logic stays human-vetted. The SQL escape hatches (`run_native_query`, `run_card`) are available when the semantic layer cannot express the question, but a metric is strictly safer because it forecloses common analytics mistakes (averaging averages, double-counting joined rows). It is **not** a Metabase admin tool. Creating dashboards, editing cards, configuring data sources, and managing users live in the Metabase admin UI. ## [](#authentication-model)Authentication model Metabase API keys (v0.49+) inherit the **permissions of the group they’re bound to**. The MCP sends the resolved key in the `X-API-Key` header on every upstream request. The Metabase MCP uses one API key per MCP instance, applied to every tool call, with no per-user delegation. To give different agents different access, mount multiple MCP instances bound to different Metabase groups. Auth is implemented as `oneof auth { StaticKeyAuth api_key }` (single variant today, future-proof for adding session or OAuth modes without a wire-format break). ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Metabase instance on v0.49 or later (API keys are not available on older versions). - Admin access to the Metabase instance to mint an API key. - A Metabase **group** with the right permissions for the data you want the agent to access (typically `Native query editing` on the relevant databases plus `View` on the relevant collections). - One Redpanda Agentic Data Plane secret-store entry for the API key. Secret references must be `UPPER_SNAKE_CASE`, for example `METABASE_API_KEY`. ## [](#get-a-metabase-api-key)Get a Metabase API key Mint the key in the Metabase admin UI: 1. Sign in to your Metabase instance as an admin. 2. Open **Settings > Authentication > API keys**. 3. Click **Create API key**. 4. Give the key a descriptive name, for example `redpanda-mcp`. 5. Bind the key to a group whose permissions match the access you want the agent to have. For an analytics agent, an `Analytics` group with `Native query editing` on the relevant databases and `View` on the relevant collections is typical. 6. Copy the key. Metabase shows it only once. 7. Save the key in your Agentic Data Plane secret store under a name like `METABASE_API_KEY`. > 📝 **NOTE** > > Permission errors surface from the upstream Metabase API as `metabase API error (status 4xx)`. Fix by adjusting the bound group’s permissions in Metabase, not by re-issuing a different key. ## [](#configure)Configure Create a new Metabase MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick **Metabase** from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Metabase configuration form: | Field | Notes | | --- | --- | | Base URL | Your Metabase instance URL, for example https://metabase.mycompany.com. No trailing slash. | | API key ref | Secret-store reference for the Metabase API key (UPPER_SNAKE_CASE). Example: METABASE_API_KEY. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI Run `rpk ai mcp-server create` with the Metabase managed-config payload: ```bash rpk ai mcp-server create --name metabase --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.metabase.v1.MetabaseMCPConfig", "base_url": "https://metabase.mycompany.com", "api_key": { "key_secret_ref": "METABASE_API_KEY" } }' ``` The `api_key` field wraps the shared `StaticKeyAuth`; `key_secret_ref` is the bare secret-store key name (`UPPER_SNAKE_CASE`). ## [](#tools)Tools The Metabase MCP exposes tools across a semantic-layer surface and a SQL-first surface. The agent’s preferred path is the semantic-layer surface; the SQL-first surface is an escape hatch. ### [](#semantic-layer-tools)Semantic-layer tools These six tools cover discovery, schema lookup, and structured queries against pre-vetted metrics. | Tool | Description | | --- | --- | | search | Run Metabase’s universal search across tables, metrics, saved questions, dashboards, and collections. Use this first to discover existing analyses before constructing a fresh query. | | get_table | Retrieve a table’s fields, related tables, and metrics defined on it. The field list is what the agent uses to write correct filters and group-bys. | | get_table_field_values | Sample values plus a small statistical summary for one field on a table. Useful before writing a filter; the agent can check actual cardinality and value shape rather than guessing. | | get_metric | Retrieve a metric definition with its queryable dimensions. Metrics are pre-vetted aggregations; calling query against a metric is strictly preferred over hand-rolled SQL when the measure already exists. | | get_metric_field_values | Sample values for a dimension of a metric. Same role as get_table_field_values, scoped to a metric’s queryable dimensions. | | query | One-shot structured query against a table OR a metric, with optional group_by_field_ids, filters (MBQL JSON snippets), and limit. Returns typed columns and rows. | ### [](#sql-first-escape-hatches)SQL-first escape hatches These five tools let an agent enumerate databases, run native SQL, and execute saved questions when the semantic layer cannot express the question. | Tool | Description | | --- | --- | | list_databases | Enumerate the databases Metabase is connected to. The id returned here is what get_table, list_tables, run_native_query, and query take as the database reference. | | list_tables | List tables in a database, optionally filtered by schema. Use this when browsing schema by name; search is the right tool when looking by keyword. | | run_native_query | Execute a native (raw SQL) query against a database. Escape hatch for cases the semantic layer cannot express; prefer query against a metric or table when possible. | | get_card | Retrieve a saved question’s definition (name, description, database, display type, native SQL when applicable). Use this to inspect an analysis a human curated. | | run_card | Execute a saved question and return its result rows. The card carries its own database, query, and parameters; this RPC just runs it as-is. | ## [](#examples)Examples The examples in this section assume `[https://aigw..clusters.rdpa.co/mcp/v1/metabase](https://aigw.\.clusters.rdpa.co/mcp/v1/metabase)` is your MCP server URL and that you’ve already authenticated to the gateway. ### [](#discover-what-exists-then-query-a-metric)Discover what exists, then query a metric Search for revenue-related objects: ```bash curl -sS -X POST https://aigw..clusters.rdpa.co/mcp/v1/metabase \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "search", "arguments": { "q": "revenue", "models": ["metric", "card"], "page_size": 10 } } }' | jq ``` If `search` returns a metric like `{ "model": "metric", "id": 200, "name": "Monthly Revenue", …​ }`, fetch its dimensions: ```bash curl -sS -X POST https://aigw..clusters.rdpa.co/mcp/v1/metabase \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "get_metric", "arguments": { "metric_id": 200 } } }' | jq ``` Then run the metric grouped by one of its dimensions (the `field_id` values come from the previous response’s `dimensions[]`): ```bash curl -sS -X POST https://aigw..clusters.rdpa.co/mcp/v1/metabase \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "query", "arguments": { "database_id": 5, "metric_id": 200, "group_by_field_ids": [12], "limit": 1000 } } }' | jq ``` ### [](#native-sql-escape-hatch)Native SQL escape hatch When a question genuinely needs raw SQL (for example, multi-CTE windowed analysis Metabase’s MBQL doesn’t express), drop down to `run_native_query`: ```bash curl -sS -X POST https://aigw..clusters.rdpa.co/mcp/v1/metabase \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ "jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": { "name": "run_native_query", "arguments": { "database_id": 5, "query": "SELECT date_trunc(\"month\", created_at) AS m, count(*) FROM orders GROUP BY 1 ORDER BY 1 DESC LIMIT 12" } } }' | jq ``` The bound group’s permissions on the target database determine whether the call succeeds. A key bound to a group without `Native query editing` on the database returns a 403. ## [](#notes-and-limits)Notes and limits Five behaviors are worth knowing before you point an agent at the server: - **Permissions follow the API key’s group.** The MCP doesn’t bypass Metabase’s permission system; permission errors surface as `metabase API error (status 4xx)`. Fix the group, not the MCP. - **Metric vs. saved question.** In Metabase v0.50+ both metrics and saved questions are stored as cards (`/api/card/:id`). `get_metric` validates that the card’s `type == "metric"` and surfaces queryable dimensions; `get_card` works on any card. `run_card` runs whatever is saved. - **Sample database.** The `list_databases` tool exposes Metabase’s bundled Sample Database with an `is_sample` flag set. Agents should usually exclude it from real analyses. - **Body cap.** Query results are capped at 25 MiB on the gateway-side wrapper. Queries that exceed the cap return an error rather than blowing up the agent’s context. Lower the `limit` argument on `query` or `run_native_query` if you hit it. - **Response curation.** The MCP drops Metabase’s internal indexing metadata (search ranking signals, fingerprint blobs, embedded user objects, per-column metadata blobs) so each response is 60–85% smaller than the raw REST API would return. Agents see clean, action-relevant fields. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | metabase API error (status 401) | API key is invalid or has been revoked. Mint a new key in Settings > Authentication > API keys and update the secret in the Agentic Data Plane secret store under the same name. | | metabase API error (status 403) | The API key’s group lacks the required permission. Check the group’s database and collection permissions in Admin > Permissions. | | metabase API error (status 404) on get_table or get_metric | The table_id or metric_id doesn’t exist in the bound database. Confirm the ID with search or list_tables first. | | Query returns an error about exceeding the body cap | Result set exceeds 25 MiB. Lower limit on the query, or pre-aggregate by calling query against a metric. | | Agent keeps calling run_native_query instead of query | Confirm the relevant metric exists and has queryable dimensions. The agent prefers metrics when search or get_table surfaces them. | ## [](#limitations)Limitations This page does not cover: - **Metabase administration.** Creating dashboards, editing cards, configuring data sources, and managing users live in the Metabase admin UI or its `/api/setup` endpoints. - **Per-user delegation.** The MCP uses one shared API key per instance. To give different agents different access, mount multiple MCP instances bound to different Metabase groups. - **Metabase MBQL reference.** See Metabase’s own documentation for the full MBQL filter shape; the MCP passes `filters` arguments through verbatim. ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test a server’s tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Plug in an App, Database, or Tool](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) --- # Page 26: NetSuite Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/netsuite.md --- # NetSuite Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: NetSuite Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/netsuite page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/netsuite.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/netsuite.adoc description: Let agents read NetSuite records and run SuiteQL with each caller's NetSuite OAuth identity. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the NetSuite managed MCP server with account ID and user-delegated OAuth learning-objective-2: Identify the NetSuite read-only tools available to agents learning-objective-3: Test NetSuite tool calls with an authorized user's connection page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-29" --- The **NetSuite** managed MCP server lets agents query Oracle NetSuite records and run SuiteQL with the calling user’s OAuth identity. The server wraps SuiteTalk REST APIs and exposes read-only tools for records, customers, sales orders, and items. After reading this page, you will be able to: - Configure the NetSuite managed MCP server with account ID and user-delegated OAuth - Identify the NetSuite read-only tools available to agents - Test NetSuite tool calls with an authorized user’s connection ## [](#what-this-mcp-server-does)What this MCP server does Use NetSuite when an agent needs ERP, finance, or accounting context from records that a NetSuite user can already read. Every tool is read-only. | Tool | What it does | | --- | --- | | suite_ql_query | Runs a SuiteQL query and returns rows as JSON. Use this tool for joins, aggregates, and cross-record analysis. | | get_record | Fetches one record by record type and internal ID. You can expand sub-resources, such as line items and addresses. | | list_records | Lists records of a type with optional NetSuite filter expression and paging. Use typed helpers when they fit. | | get_customer | Fetches a customer record by internal ID. | | list_customers | Lists customers with optional company name, email, modified-since, inactive, and paging filters. | | list_sales_orders | Lists sales orders with optional customer ID, date range, status, modified-since, and paging filters. | | search_items | Searches inventory and service items by SKU, display name, item type, inactive flag, and paging filters. | The managed server does not expose write tools, saved-search execution, or report execution. ## [](#prerequisites)Prerequisites - A NetSuite Integration Record that uses OAuth 2.0 Authorization Code grant. - A NetSuite [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) in Redpanda Agentic Data Plane for the same account. - Your NetSuite account ID. Production accounts use a numeric ID such as `1234567`. Sandbox accounts include a suffix such as `1234567_SB1`; release-preview accounts can use a suffix such as `1234567_RP1`. - NetSuite users with the **REST Web Services** permission on at least one role. ## [](#configure-netsuite-oauth)Configure NetSuite OAuth In NetSuite, create an Integration Record with OAuth 2.0 enabled. Set the redirect URI to the OAuth callback URL for your AI Gateway deployment. Select the **REST Web Services** scope. Find your account ID under **Setup > Company > Company Information > ACCOUNT ID**. The managed server normalizes underscores to hyphens when it builds the SuiteTalk host. Register a NetSuite OAuth Provider in Agentic Data Plane with account-scoped endpoints: ```text Authorization endpoint: https://.app.netsuite.com/app/login/oauth2/authorize.nl Token endpoint: https://.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token Scopes: rest_webservices ``` Store the NetSuite client secret in the Agentic Data Plane secret store, then reference that secret from the OAuth Provider. ## [](#create-the-managed-mcp-server)Create the managed MCP server Create a managed MCP server with the NetSuite type. NetSuite authentication is part of the managed config, so include `user_oauth` inside the JSON. ```json { "@type": "type.googleapis.com/redpanda.mcps.netsuite.v1.NetSuiteMCPConfig", "account_id": "1234567_SB1", "user_oauth": { "provider_name": "netsuite", "required_scopes": ["rest_webservices"] } } ``` For a non-UI path, use `rpk ai mcp-server create` with `--managed-config`. The top-level `--user-oauth-provider` and `--user-oauth-scopes` flags apply to self-managed servers only. ```bash rpk ai mcp-server create --name netsuite \ --description "NetSuite ERP with user OAuth" \ --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.netsuite.v1.NetSuiteMCPConfig", "account_id": "1234567_SB1", "user_oauth": { "provider_name": "netsuite", "required_scopes": ["rest_webservices"] } }' ``` ## [](#authorize-and-test)Authorize and test Before a caller can use NetSuite tools, the caller must authorize the NetSuite OAuth Provider. After authorization, Redpanda stores the user’s access token in the token vault and subsequent tool calls use that token automatically. Test the server in the Inspector before you connect it to an agent. Start with a narrow `suite_ql_query`, `get_customer`, or `list_customers` call to confirm that NetSuite returns records visible to the authorized user. ## [](#netsuite-query-notes)NetSuite query notes NetSuite record filters use NetSuite-specific forms: - Use `CONTAIN` for substring filters, not `CONTAINS`. - Use `M/D/YYYY` date literals for REST collection filters. - Match reference fields by internal ID. For example, sales-order customer filters use `entity ANY_OF [12345]`. - Use NetSuite status IDs such as `_pendingFulfillment` for sales-order status filters, not display labels. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | Tool calls return 403 | Confirm the authorizing NetSuite user has the REST Web Services permission. | | OAuth succeeds, but tool calls fail | Confirm the OAuth Provider endpoints and the MCP server account_id refer to the same NetSuite account. | | A sales-order status filter returns no rows | Use the NetSuite internal status ID, such as _pendingFulfillment, instead of the display label. | ## [](#next-steps)Next steps - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 27: OpenAPI Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/openapi.md --- # OpenAPI Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: OpenAPI Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/openapi page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/openapi.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/openapi.adoc description: Expose any OpenAPI 3.x-described HTTP API as MCP tools. Point Redpanda at a spec URL and get a fully-typed tool surface for agents, with no custom code. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the OpenAPI managed MCP server against an OpenAPI 3 spec learning-objective-2: Pick the right authentication mode for the upstream API learning-objective-3: Verify generated tools through the Inspector page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **OpenAPI** managed MCP server is the "bring your own API" escape hatch. Hand it an OpenAPI 3.x spec, and it generates one MCP tool per operation in the spec. No custom code, no per-API managed type: useful when the API you want to expose is not in the catalog. After reading this page, you will be able to: - Configure the OpenAPI managed MCP server against an OpenAPI 3 spec - Pick the right authentication mode for the upstream API - Verify generated tools through the Inspector ## [](#what-this-mcp-server-does)What this MCP server does The OpenAPI managed type: - Loads an OpenAPI 3.x spec (3.0 or 3.1) from a URL or pasted JSON/YAML. Swagger 2.0 specs are not supported. - Generates one MCP tool per operation, with input schemas derived from the spec’s parameter and request-body schemas. `oneOf`, `anyOf`, and `discriminator` are supported; `callbacks` are not. - Forwards calls to the upstream API, applying the configured authentication. By default the generator walks schemas up to a depth of 5 (`max_schema_depth`); schemas nested deeper than that are truncated. ## [](#prerequisites)Prerequisites - An HTTP API with an OpenAPI 3.x spec. - The spec URL or the spec content itself. - Credentials for the API, if it requires them. ## [](#configure)Configure 1. Open **MCP Servers > Create Server**. 2. Pick **OpenAPI** from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the OpenAPI configuration form: - `Spec`: A URL to the OpenAPI 3.x spec (JSON or YAML), or paste the spec inline. Set exactly one. Inline specs are capped at 3 MiB; host larger specs and use the URL instead. - `Base URL` (optional): Useful when the spec’s `servers` block doesn’t match your environment. - `Operation filter` (optional): Include or exclude operations by tag, `operationId` glob, path glob, or HTTP method. - `Max schema depth` (optional): Maximum depth for schema conversion. Defaults to 5. - `Extract headers` (optional): Response header names to surface to the agent, such as `X-CSRF-Token` or `Set-Cookie`. Listed headers are appended to the tool result; all other response headers are dropped. Matching is case-insensitive. 5. Configure authentication (see [Authentication](#authentication)). 6. Click **Create**. ## [](#authentication)Authentication OpenAPI is the most flexible managed type for authentication: the upstream API can need anything. All the standard authentication modes apply: | Mode | Use when | | --- | --- | | No Auth | Public APIs (rare in practice). | | Bearer | API expects a bearer token. The handler sends Authorization: Bearer . | | API Key | API expects a key in a header or query parameter. Choose the placement and the parameter name. | | Basic | API expects HTTP Basic authentication (a username and password). | | User OAuth | API supports OAuth on behalf of users and you want per-user identities. Requires an OAuth Provider configured for that API. See Configure User-Delegated OAuth. | | Service-account OAuth | API supports the OAuth client-credentials grant and you want a single shared service-account identity for every call, rather than per-user tokens. | For `Service-account OAuth`, the gateway runs the client-credentials exchange once per server and shares the resulting token across all callers. Supply the client ID, a secret-store reference for the client secret, the token URL, and any scopes the API requires. Store the client secret in the Agentic Data Plane secret store under an `UPPER_SNAKE_CASE` key, such as `OAUTH_CLIENT_SECRET`, and reference it from the form. Use an `https` token endpoint. ## [](#test)Test 1. Open the **Inspector** tab. 2. The Tools panel lists every operation from your spec, named per the `operationId` (or per the path if `operationId` is missing). Tool names longer than 64 characters are truncated and given a short hash suffix to keep them unique, because LLM providers cap tool-name length. Parameter names that start with `$` (common in OData APIs, such as `$top` and `$filter`) appear without the `$` prefix in the tool schema; the server still sends the original parameter name to the API. 3. Pick one and run it. The Inspector renders a form from the operation’s parameter and request-body schemas. See [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) for general Inspector usage. ## [](#use-with-agents)Use with agents After tools generate cleanly, point an agent at the `Server URL` on the server’s **Connection** tab. The agent sees one tool per OpenAPI operation, named accordingly. Each tool returns the upstream response as a structured result with the fields `status_code`, `headers`, and `body`. `body` is the parsed JSON payload when the response is JSON, or the raw response text otherwise. `headers` holds the response headers you selected with `Extract headers`, and is empty when none are configured. Agents that read structured output can address these fields directly, and a plain-text copy of the same result is also included for clients that read only text. A 4xx or 5xx response surfaces as a tool error rather than a structured result. Every generated tool also accepts an optional `headers` input, which lets the agent set request headers the spec doesn’t declare. Agent-supplied headers can’t override the configured authentication headers or the spec’s own header parameters. Combined with `Extract headers`, this supports APIs that round-trip headers, such as a CSRF token fetched from one response and echoed on the next request. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | "Failed to load spec" | Confirm the spec URL is reachable from Redpanda Agentic Data Plane and serves valid JSON or YAML. CORS doesn’t matter (Redpanda fetches server-side). | | Tools list doesn’t include an expected operation | The operation may be missing an operationId, or the operation include/exclude filters might be excluding it. If two operations resolve to the same tool name, the server keeps one, drops the other, and logs a warning naming both. | | Tool input schema looks wrong | oneOf, anyOf, and discriminator are supported, but callbacks are not, and schemas deeper than max_schema_depth (default 5) are truncated. Adjust the spec or raise the depth. | | Calls return 401 | Authentication mode or credentials are wrong. Confirm secret content and the API’s expected authentication header. | | Calls return 404 with the right operation | servers block in the spec doesn’t match your real API endpoint. Use the Base URL field. | ## [](#limitations)Limitations - **Custom tool logic**: The OpenAPI type is purely a spec-to-tools generator. For business logic on top of the API, use a [self-managed MCP server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/). - **GraphQL APIs**: OpenAPI doesn’t describe GraphQL. For GraphQL APIs, use a self-managed server. - **gRPC services**: Same as GraphQL: use a self-managed server. --- # Page 28: Ramp Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/ramp.md --- # Ramp Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Ramp Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/ramp page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/ramp.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/ramp.adoc description: Manage Ramp corporate cards, transactions, spend limits, and reimbursements from an LLM agent. Per-user OAuth so each agent action runs as the calling end-user. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Ramp managed MCP server with per-user OAuth learning-objective-2: Pick the right scopes and environment for production vs sandbox learning-objective-3: List transactions, manage cards, and adjust spend limits from an agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-29" --- The **Ramp** managed MCP server lets an LLM read and act on your company’s Ramp spend data: listing and inspecting transactions, browsing cards, managing spend limits, querying users and departments, looking up vendors, and reviewing reimbursements. After reading this page, you will be able to: - Configure the Ramp managed MCP server with per-user OAuth - Pick the right scopes and environment for production vs sandbox - List transactions, manage cards, and adjust spend limits from an agent ## [](#what-this-mcp-server-does)What this MCP server does Wraps the [Ramp Developer API v1](https://docs.ramp.com/developer-api/v1) using per-user OAuth tokens, so each user’s Ramp permissions are enforced automatically and no shared API key is stored. It is suitable for expense analysis, spend-policy enforcement, and corporate card management workflows. It is **not** intended for accounting system integrations or bulk data exports; use Ramp’s native accounting sync or data export features for those tasks. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Ramp account with admin access to the Ramp Developer Portal. - An [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane for Ramp. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). - Familiarity with [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#get-ramp-credentials)Get Ramp credentials Set up the OAuth app on Ramp and the matching OAuth Provider in Agentic Data Plane: 1. Sign in to the [Ramp Developer Portal](https://app.ramp.com/developer). 2. Go to **Developer Settings > Applications** and click **Create Application**. 3. Set the redirect URI to your AI Gateway OAuth callback (typically `https://aigw..clusters.rdpa.co/oauth/v1/callback`). 4. Note the **Client ID** and **Client Secret**. 5. Required scopes: - `transactions:read` - `cards:read` - `cards:write` - `users:read` - `departments:read` - `vendors:read` - `reimbursements:read` - `limits:read` - `limits:write` 6. In Agentic Data Plane, register an OAuth Provider with: - **Authorization endpoint**: `https://app.ramp.com/v1/authorize` - **Token endpoint**: `https://api.ramp.com/developer/v1/token` - The Client ID and a secret-store reference for the Client Secret ## [](#configure)Configure Create a new Ramp MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Ramp` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Ramp configuration form: | Field | Notes | | --- | --- | | environment | production for the live Ramp API. demo for Ramp’s sandbox environment. Omit (or leave empty) for production. | | OAuth Provider | The Ramp OAuth Provider you configured. | | Required scopes | All the scopes listed above. Drop write scopes (cards:write, limits:write) if the MCP only needs to read. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI Ramp authenticates per-user through the Ramp OAuth Provider you configured. Reference that provider in the managed config’s `userOauth.providerName` field. Ramp supports per-user OAuth only, so the `userOauth` block is required. The `--user-oauth-provider` and `--user-oauth-scopes` flags apply to remote (`--url`) servers only. ```bash rpk ai mcp-server create --name my-ramp --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.ramp.v1.RampMCPConfig", "environment": "production", "userOauth": { "providerName": "ramp" } }' ``` Set `environment` to `"demo"` to target Ramp’s sandbox. `userOauth.providerName` is the resource name of the Ramp OAuth Provider you registered. ## [](#tools)Tools The Ramp MCP exposes the following tools: | Tool | Description | | --- | --- | | list_transactions | List transactions with optional filters. Supports pagination through the start cursor. Returns up to page_size results (max 100). | | get_transaction | Retrieve a single transaction by ID, including line items, accounting selections, and policy violations. | | list_cards | List corporate cards. Supports pagination. | | create_card | Issue a new virtual card. Returns a deferred task ID, since Ramp creates cards asynchronously. | | suspend_card | Suspend an active card by ID. Returns a deferred task ID. | | list_users | List Ramp users in your organization. Supports pagination. | | list_departments | List departments. Supports pagination. | | list_vendors | List vendors. Supports pagination. | | list_reimbursements | List out-of-pocket reimbursement requests. Supports pagination. | | list_limits | List spend limits. Supports pagination. | | create_limit | Create a new spend limit. Returns a deferred task ID, since Ramp creates limits asynchronously. | | update_limit | Update an existing spend limit’s display name or spending restrictions synchronously. | ### [](#example-list-recent-transactions)Example: List recent transactions ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/my-ramp \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "list_transactions", "arguments": { "page_size": 25 } } }' ``` ### [](#example-create-a-virtual-card-for-a-vendor)Example: Create a virtual card for a vendor ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/my-ramp \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "create_card", "arguments": { "display_name": "AWS Services", "user_id": "usr_abc123", "idempotency_key": "create-aws-card-2026", "spending_limit_amount": 5000.0, "spending_limit_interval": "MONTHLY", "spending_limit_currency": "USD" } } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | OAuthConnectionRequired | First call from a user with no stored token. The user completes Ramp’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. | | scope_upgrade_required | Server’s required_scopes was extended after users had already consented. Users re-consent with the higher scope. | | create_card / create_limit returns a task ID with no card / limit details | These operations are asynchronous on Ramp’s side. The MCP returns a task ID that you can poll against Ramp’s API; the actual card or limit appears after the task completes. | | 403 Forbidden reading or writing | The calling user’s Ramp role doesn’t grant the action. Ramp’s role-based access control runs end-to-end: per-user OAuth means each user only sees what their Ramp account permits. | ## [](#limitations)Limitations This page does not cover: - **Bulk data export**: Use Ramp’s native data export. - **Accounting system integration**: Use Ramp’s accounting sync. - **Receipt management and approvals**: Handled in the Ramp web UI. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 29: Salesforce Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/salesforce.md --- # Salesforce Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Salesforce Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/salesforce page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/salesforce.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/salesforce.adoc description: Query, create, update, and delete Salesforce CRM records with SOQL and the REST API, and run saved reports, from an LLM agent. Choose a shared service-account identity or per-user OAuth. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Choose between service-account OAuth and per-user OAuth for the Salesforce managed MCP server learning-objective-2: Configure the Salesforce managed MCP server against your org learning-objective-3: Query, modify, and report on Salesforce records from an agent page-git-created-date: "2026-06-10" page-git-modified-date: "2026-07-29" --- The **Salesforce** managed MCP server lets agents work with [Salesforce](https://www.salesforce.com/) CRM data: running SOQL queries and SOSL searches, reading and writing records, inspecting object metadata, and running saved Salesforce reports through the REST and Analytics APIs. After reading this page, you will be able to: - Choose between service-account OAuth and per-user OAuth for the Salesforce managed MCP server - Configure the Salesforce managed MCP server against your org - Query, modify, and report on Salesforce records from an agent ## [](#what-this-mcp-server-does)What this MCP server does The Salesforce MCP exposes the following tools: | Tool | What it does | | --- | --- | | query | Execute a SOQL query. | | query_more | Fetch the next page of a large query result, using the nextRecordsUrl returned by query. | | search | Execute a SOSL full-text search. | | list_objects | List all available sObject types in the org. | | describe_object | Get field and schema metadata for an sObject. | | get_record | Fetch a record by ID, with optional field selection. | | create_record | Create a new record. | | update_record | Update an existing record (PATCH). | | delete_record | Delete a record. | | describe_report | Get a report’s metadata: columns, groupings, and filters. | | run_report | Run a saved report synchronously and return its results. | `run_report` executes a saved Salesforce report through the Analytics REST API, applying the report’s built-in groupings, aggregates, and filters that raw SOQL cannot reproduce. Use `describe_report` first to discover a report’s columns and filters. To list reports, use `query` with `SELECT Id, Name, DeveloperName FROM Report`. ## [](#choose-an-authentication-mode)Choose an authentication mode The Salesforce MCP supports two authentication modes. Set exactly one in the managed config: | Mode | When to use | | --- | --- | | Service-account OAuth (serviceAccountOauth) | OAuth 2.0 client-credentials grant through a Connected App. Service-to-service: no browser, no redirect. Every caller shares one Salesforce identity. Use this when a single shared identity is acceptable. | | Per-user OAuth (userOauth) | Each caller’s Salesforce OAuth token is resolved from the token vault on every request, so tool calls respect Salesforce row-level security for the calling user. Requires a Salesforce OAuth Provider. Use this when each user must act as themselves. | ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Salesforce org where you can create a Connected App. A free Developer Edition org or a Sandbox works. - Your Salesforce instance URL (your **My Domain** URL), such as `https://mycompany.my.salesforce.com`. Use the full `https://` base domain with no path. - For **service-account OAuth**: a Connected App configured for the client-credentials flow, and its consumer key and consumer secret. - For **per-user OAuth**: a Salesforce [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) and [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#create-a-connected-app-for-service-account-oauth)Create a Connected App for service-account OAuth Skip this section if you are using per-user OAuth. The client-credentials flow is a service-to-service flow: `client_id` + `client_secret` exchange for an `access_token`, with no browser interaction. 1. In Salesforce **Setup**, search for **App Manager** and click **New Connected App**. 2. Enter a Connected App Name and Contact Email. 3. Under **API (Enable OAuth Settings)**: - Check **Enable OAuth Settings**. - Set a Callback URL (required by the form, unused for client credentials). - Add the Manage user data via APIs (api) and Perform requests at any time (refresh\_token, offline\_access) OAuth scopes. - Check **Enable Client Credentials Flow**. 4. Click **Save**, then **Continue**. 5. Open the app from **App Manager**, click **Manage Consumer Details**, and copy the Consumer Key (your `client_id`) and Consumer Secret (your `client_secret`). 6. Set a Run As user: in **App Manager**, open the app, click **Manage > Edit Policies**, and under Client Credentials Flow set Run As to a Salesforce user with API access. Click **Save**. Store the consumer secret in the Redpanda Agentic Data Plane secret store under an `UPPER_SNAKE_CASE` key, such as `SALESFORCE_CLIENT_SECRET`. ## [](#configure)Configure Create a new Salesforce MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Salesforce` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Salesforce configuration form, set the Salesforce instance URL (`orgUrl`) and pick an Auth Method: service-account OAuth or per-user OAuth. For per-user OAuth, select the Salesforce OAuth Provider. 5. Optionally pin a Salesforce REST API version. The default is `v65.0`. 6. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) to create the server with a managed config. Set exactly one auth variant. For service-account OAuth, supply the Connected App’s `clientId`, a secret-store reference for the consumer secret (`clientSecretRef`), and the token URL: ```bash rpk ai mcp-server create --name my-salesforce --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.salesforce.v1.SalesforceMCPConfig", "orgUrl": "https://mycompany.my.salesforce.com", "serviceAccountOauth": { "clientId": "3MVG9...", "clientSecretRef": "SALESFORCE_CLIENT_SECRET", "tokenUrl": "https://mycompany.my.salesforce.com/services/oauth2/token" } }' ``` For per-user OAuth, reference the Salesforce OAuth Provider by name. The per-user token is resolved from the token vault on each call: ```bash rpk ai mcp-server create --name my-salesforce --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.salesforce.v1.SalesforceMCPConfig", "orgUrl": "https://mycompany.my.salesforce.com", "userOauth": { "providerName": "salesforce" } }' ``` To pin a specific API version, add `"apiVersion": "v62.0"` to the config. The default is `v65.0`. ## [](#tool-examples)Tool examples Run a SOQL query: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/my-salesforce \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "query", "arguments": { "soql": "SELECT Id, Name, Industry, AnnualRevenue FROM Account ORDER BY AnnualRevenue DESC LIMIT 10" } } }' ``` Fetch a record by ID, restricting the returned fields: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/my-salesforce \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "get_record", "arguments": { "sobject": "Account", "record_id": "001xx000003GYkZAAW", "fields": "Id,Name,Industry,AnnualRevenue,BillingCity" } } }' ``` ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | invalid_client_credentials from OAuth (service-account mode) | Confirm the Connected App has a Run As user set and the api scope, and that you are using the Consumer Key (not the app name) as clientId. | | OAuthConnectionRequired (per-user mode) | First call from a user with no stored token. The user completes Salesforce’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. | | INSUFFICIENT_ACCESS errors | The acting identity (the Run As user for service-account mode, or the calling user for per-user mode) lacks permission on that object or record. Grant the appropriate profile or permission set. | | INVALID_FIELD in a SOQL query | The field does not exist or field-level security hides it from the acting identity. Use describe_object to see which fields are visible. | | nextRecordsUrl rejected by query_more | The URL must start with /services/data/. Pass the exact value returned by the query response. | ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test MCP tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 30: Sentry Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/sentry.md --- # Sentry Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Sentry Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/sentry page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/sentry.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/sentry.adoc description: Give an agent read-only access to Sentry organizations, projects, issues, events, and releases through per-user OAuth, so it can read a stack trace, scope an error's impact, and localize the fix. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Sentry managed MCP server with per-user OAuth learning-objective-2: Confine the server to a single Sentry organization (optional) learning-objective-3: Diagnose an issue from the Inspector or an agent, from discovery through to the suspect commit page-git-created-date: "2026-06-26" page-git-modified-date: "2026-07-29" --- The **Sentry** managed MCP server gives agents read-only access to a Sentry instance: discover organizations and projects, list and read issues, read the events (with stack traces) behind them, scope an error’s impact across releases and tags, and surface the commits most likely responsible. The intended use is an agent that reads a Sentry issue and its stack trace to diagnose a bug, then fixes it in the codebase. After reading this page, you will be able to: - Configure the Sentry managed MCP server with per-user OAuth - Confine the server to a single Sentry organization (optional) - Diagnose an issue from the Inspector or an agent, from discovery through to the suspect commit ## [](#what-this-mcp-server-does)What this MCP server does Wraps the Sentry REST API and exposes read-only tools grouped by the diagnose-and-fix flow: discover organizations and projects, read the error and its stack trace, scope the impact across tags and releases, and localize the fix to a suspect commit. Authentication is per-user OAuth only. Every tool call runs as the authenticated user against a Sentry OAuth token resolved from the gateway’s [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault), so there is no shared service account and each action is attributed to the calling end-user. The server requests only read scopes (`org:read`, `project:read`, `event:read`, `project:releases`). The server is read-only by design: resolving or ignoring issues, commenting, and any other write operation is deliberately absent. Use the Sentry UI or its API directly for those. > 📝 **NOTE** > > Tool output is developer-facing observability data and is not guaranteed to be free of personally identifiable information (PII). The curated event shape omits Sentry’s `user` and `request` context, but event `tags` are forwarded verbatim and can carry user-supplied values (for example `user`, `url`, `server_name`, or custom tags), and suspect-commit and release-commit output includes commit author names and email addresses. This MCP server connects to `sentry.io` only. Self-hosted Sentry is not supported. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Sentry account on `sentry.io` with access to the organizations and projects you want the agent to read. - A Sentry OAuth application and a matching [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). ## [](#get-sentry-credentials)Get Sentry credentials Sentry MCP uses per-user OAuth, so you register an OAuth application on Sentry and a matching OAuth Provider in Agentic Data Plane: 1. In Sentry, go to **Settings > Account > API > Applications** and create a new application. Set the authorized redirect URI to your Agentic Data Plane gateway’s OAuth callback. 2. Copy the Client ID and Client Secret, and store the client secret in the Agentic Data Plane secret store. 3. Register a matching OAuth Provider in Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). Use Sentry’s authorize endpoint (`[https://sentry.io/oauth/authorize/](https://sentry.io/oauth/authorize/)`) and token endpoint (`[https://sentry.io/oauth/token/](https://sentry.io/oauth/token/)`). 4. Each end-user authenticates once through the OAuth flow; tokens are stored in the gateway’s token vault. Required scopes: `org:read`, `project:read`, `event:read`, and `project:releases` cover all tools. `org:read` lists organizations and projects and resolves event IDs; `project:read` lists issues and reads project events and suspect commits; `event:read` reads issue events, tags, and hashes (the stack trace); and `project:releases` reads releases and their commits. All scopes are read-only. ## [](#configure)Configure Create a new Sentry MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Sentry` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Sentry configuration form: | Field | Notes | | --- | --- | | organization_slug | Optional. Confines the server to a single Sentry organization. When set, every tool that takes an organization_slug is forced to this value and a request targeting a different organization is rejected; list_organizations stays unrestricted so it can still be used for discovery. Leave empty for unrestricted access, bounded only by the OAuth token. The slug is the URL segment at sentry.io/organizations/, not the display name. | | user_oauth | provider_name (the Sentry OAuth Provider you configured) and the minimum required scopes (org:read, project:read, event:read, project:releases covers all tools). | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) to create the server with a managed config. Set the OAuth provider and scopes inside the `--managed-config` JSON. Per-user OAuth is the only supported authentication method. #### Unrestricted The caller supplies the organization on each call, bounded only by the OAuth token’s access. ```bash rpk ai mcp-server create --name acme-sentry --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.sentry.v1.SentryMCPConfig", "user_oauth": { "provider_name": "sentry-prod", "required_scopes": ["org:read", "project:read", "event:read", "project:releases"] } }' ``` #### Confined to one organization Pin the server to a single organization with `organization_slug`. ```bash rpk ai mcp-server create --name acme-sentry --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.sentry.v1.SentryMCPConfig", "organization_slug": "my-org", "user_oauth": { "provider_name": "sentry-prod", "required_scopes": ["org:read", "project:read", "event:read", "project:releases"] } }' ``` ## [](#tools)Tools The Sentry MCP exposes read-only tools, grouped by the diagnose-and-fix flow: | Tool | Description | | --- | --- | | list_organizations | List the Sentry organizations the token can access. Used for discovery; never restricted by organization_slug. | | list_projects | List the projects within an organization. | | list_issues | List a project’s issues, filtered by a Sentry search query (for example, is:unresolved). | | get_issue | Fetch issue detail by numeric ID or short ID (for example, WEB-1). | | get_issue_latest_event | Return the most recent event for an issue, including its exception stack trace. | | list_issue_events | List the individual occurrences of an issue (compare a failing against a passing event). | | get_issue_event | Return one event of an issue with a full stack trace, by ID or the alias latest, oldest, or recommended (oldest is the regression boundary). | | get_event | Fetch an event by ID within a project, plus the issue ID it belongs to. | | resolve_event_id | Resolve a bare event ID to its issue and project at organization scope. | | get_issue_tag | Return the distribution of one tag (for example, release, environment, browser, or OS) for an issue. | | list_issue_tag_values | List the full set of values behind a tag’s capped top values. | | list_issue_hashes | List the issue’s grouping hashes. More than one hash means several crash signatures. | | get_event_committers | Return the commits and authors Sentry deems most likely responsible (suspect commits). Requires a source-code integration; returns empty when none is configured. | | list_releases | List an organization’s releases. Line an issue’s first-seen date up against a deploy. | | get_release | Return a single release’s detail by version. | | list_release_commits | List the commits that shipped in a release (the candidate changes behind a regression). | ## [](#tool-examples)Tool examples List the organizations you can read: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-sentry \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "list_organizations", "arguments": {} } }' ``` Replace `` with your cluster ID and `$TOKEN` with a gateway access token. Find unresolved issues in a project: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-sentry \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "list_issues", "arguments": { "organization_slug": "my-org", "project_slug": "web", "query": "is:unresolved", "stats_period": "24h" } } }' ``` Read an issue’s latest event and stack trace. This is the diagnostic step: the latest event carries the exception stack trace the agent reasons over. ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-sentry \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "get_issue_latest_event", "arguments": { "organization_slug": "my-org", "issue_id": "WEB-1" } } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | OAuthConnectionRequired | First call from a user with no stored token. The user completes Sentry’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. See User-delegated OAuth. | | scope_upgrade_required | The server’s required_scopes was extended after users had already consented. Users re-consent with the higher scope. | | A request is rejected naming two organization slugs | The server is confined with organization_slug and the call targeted a different organization. Target the pinned organization, or use an unrestricted server. | | get_event_committers returns empty | Sentry has no source-code integration configured for the project, so it cannot attribute suspect commits. Configure a source-code integration in Sentry. | ## [](#limitations)Limitations This page does not cover: - **Write operations**: Resolving or ignoring issues, commenting, and managing alerts are deliberately absent. The server is read-only. - **Self-hosted Sentry**: The server connects to `sentry.io` only. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 31: ServiceNow Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/servicenow.md --- # ServiceNow Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: ServiceNow Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/servicenow page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/servicenow.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/servicenow.adoc description: "Read and write a ServiceNow instance from an agent: incidents, knowledge articles, and allow-listed records, with service-account or per-user authentication." page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the ServiceNow managed MCP server with Basic authentication, OAuth client credentials, or User OAuth learning-objective-2: Use projections and hierarchies to shape responses and gate table reads learning-objective-3: Search, create, and update incidents and read allow-listed records from the Inspector or an agent page-git-created-date: "2026-06-09" page-git-modified-date: "2026-07-29" --- The ServiceNow managed MCP server lets agents work with a ServiceNow instance through the REST Table API: search, create, and update incidents, append work notes, search the knowledge base, look up users and assignment groups, search the configuration management database (CMDB) for configuration items, discover table schema and field choices, and read allow-listed tables. After reading this page, you will be able to: - Configure the ServiceNow managed MCP server with Basic authentication, OAuth client credentials, or User OAuth - Use projections and hierarchies to shape responses and gate table reads - Search, create, and update incidents and read allow-listed records from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Wraps the ServiceNow REST Table API. The MCP is vendor-generic: ServiceNow records are returned as opaque JSON objects (their field set varies by instance), and the agent learns each table’s shape and valid field values at runtime through the discovery tools (`get_table_schema`, `get_field_choices`). Customer-specific structure is supplied as configuration, not code: - Projections set the default field list returned per table (a ServiceNow incident carries about 100 fields, so projections keep responses small) and gate which tables the generic `query_records` tool may read. - Hierarchies declare dependent-field chains (for example, service area, then category, then subcategory). The agent reads the chain order with `describe_field_hierarchies` and resolves each field’s valid values at runtime with `get_field_choices`. No field values are stored in the configuration. It is not a general SQL interface, and it does not delete records: writes are limited to the typed incident tools plus generic record reads. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A ServiceNow instance and its URL (for example, `[https://acme.service-now.com](https://acme.service-now.com)`). - A ServiceNow service account, or an OAuth client, with the roles your workflows need (typically `itil` for incidents, `knowledge` for the knowledge base, and read access to `sys_dictionary` and `sys_choice` for schema discovery). - For User-OAuth mode: an [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). ## [](#choose-an-authentication-method)Choose an authentication method The MCP authenticates to ServiceNow as a single principal; it does not impersonate end-users. When creating an incident, the agent passes the requesting user’s `sys_id` in `caller_id`. Pick one of three methods: - Basic authentication (simplest): A dedicated ServiceNow user (for example, `svc_redpanda`) and its password. The username is plaintext; the password lives in the Agentic Data Plane secret store. - OAuth client credentials (service account): An OAuth API client registered under **System OAuth > Application Registry**. ServiceNow’s token endpoint is `[https://.service-now.com/oauth_token.do](https://\.service-now.com/oauth_token.do)`. - User OAuth: Per-user delegation through an OAuth Provider. ServiceNow’s recommended integration model is the service account, so prefer Basic authentication or OAuth client credentials unless you specifically need per-user attribution. ## [](#configure)Configure Create a new ServiceNow MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `ServiceNow` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the ServiceNow configuration form: | Field | Notes | | --- | --- | | instance_url | Base URL of your ServiceNow instance (for example, https://acme.service-now.com). Must be an https:// URL. | | auth | basic_auth, oauth, or user_oauth. | | basic_auth (Basic-auth mode) | username (ServiceNow service-account username, for example svc_redpanda) and password_secret_ref (secret-store reference, UPPER_SNAKE_CASE). | | oauth (OAuth mode) | client_id, client_secret_ref (secret-store reference), and token_url (https://.service-now.com/oauth_token.do). | | user_oauth (User-OAuth mode) | provider_name (the OAuth Provider you configured) and the minimum required scopes. | | projections | Per-table default field lists and the read allow-list for query_records. | | hierarchies | Dependent-field chains the agent resolves at runtime. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI For a managed server, set the auth method inside the `--managed-config` JSON. The `auth` field is required. #### Basic authentication ```bash rpk ai mcp-server create --name acme-servicenow --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.servicenow.v1.ServiceNowMCPConfig", "instance_url": "https://acme.service-now.com", "basic_auth": { "username": "svc_redpanda", "password_secret_ref": "SERVICENOW_PASSWORD" }, "projections": [ { "table": "incident", "default_fields": ["number", "short_description", "state", "priority", "assignment_group", "caller_id"], "queryable": true }, { "table": "change_request", "queryable": true } ], "hierarchies": [ { "table": "incident", "name": "service", "fields": ["u_service_area", "u_service_category", "u_service_subcategory"] } ] }' ``` #### OAuth client credentials ```bash rpk ai mcp-server create --name acme-servicenow-oauth --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.servicenow.v1.ServiceNowMCPConfig", "instance_url": "https://acme.service-now.com", "oauth": { "client_id": "", "client_secret_ref": "SERVICENOW_CLIENT_SECRET", "token_url": "https://acme.service-now.com/oauth_token.do" } }' ``` #### User OAuth ```bash rpk ai mcp-server create --name acme-servicenow-user --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.servicenow.v1.ServiceNowMCPConfig", "instance_url": "https://acme.service-now.com", "user_oauth": { "provider_name": "servicenow-prod" } }' ``` Replace `` with the client ID from your ServiceNow OAuth application registry, and `servicenow-prod` with the name of the OAuth Provider you configured. ## [](#tools)Tools The ServiceNow MCP exposes tools across incidents, knowledge, schema discovery, lookups, and generic reads: | Tool | Description | | --- | --- | | search_incidents | Search incidents with a ServiceNow encoded query. | | get_incident | Fetch one incident by sys_id. | | create_incident | Create an incident (typed fields plus an additional_fields map for custom fields). | | update_incident | Update state, assignment, notes, urgency, or impact, or resolve or close an incident. | | add_work_note | Append a work note (internal) or comment (customer-visible) to an incident. | | search_knowledge | Full-text search over published knowledge articles. | | get_article | Fetch one knowledge article by sys_id or number. | | get_table_schema | Field metadata (name, type, mandatory, reference) for a table, from sys_dictionary. | | get_field_choices | Valid choice values for a field from sys_choice, optionally scoped to a parent value. | | describe_field_hierarchies | Declared dependent-field chains for a table, from the server configuration. | | lookup_user | Find users by name, email, or username; returns their sys_id. | | lookup_group | Find assignment groups by name; returns their sys_id. | | search_cmdb_ci | Search configuration items (cmdb_ci) with a ServiceNow encoded query; the sys_class_name field distinguishes CI classes such as servers, computers, applications, and services. | | query_records | Read any allow-listed table with an encoded query. | ### [](#example-search-open-incidents)Example: Search open incidents ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-servicenow \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "search_incidents", "arguments": { "query": "active=true^priority=1" } } }' ``` Replace `` with your cluster ID and `$TOKEN` with a gateway access token. ### [](#example-create-an-incident)Example: Create an incident The agent typically calls `lookup_user` and `get_field_choices` first to resolve `caller_id` and a valid `category` value. ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-servicenow \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "create_incident", "arguments": { "short_description": "VPN drops every few minutes in Dresden", "caller_id": "a1b2c3d4e5f6...", "urgency": 2, "impact": 2, "category": "network" } } }' ``` ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | 401 Unauthorized | Confirm the service-account credentials. For Basic authentication, check the username and SERVICENOW_PASSWORD; for OAuth, check the client ID, SERVICENOW_CLIENT_SECRET, and token URL. | | 403 Forbidden | The service account lacks the role for the operation (for example, itil for incidents or read access to sys_dictionary for get_table_schema). Grant the role in ServiceNow. | | query_records rejects a table | The table is not listed in projections with queryable set to true. Add a projection entry for it. | | OAuthConnectionRequired (User-OAuth mode) | First call from a user with no stored token. The user completes the ServiceNow OAuth consent flow, the token lands in the vault, and later calls reuse it. See User-delegated OAuth. | ## [](#limitations)Limitations This MCP server does not cover: - Record deletion: Writes are limited to the typed incident tools; the MCP never deletes records. - Arbitrary SQL: Reads go through the REST Table API and encoded queries, not a SQL interface. Use `query_records` against allow-listed tables instead. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 32: SharePoint Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/sharepoint.md --- # SharePoint Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: SharePoint Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/sharepoint page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/sharepoint.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/sharepoint.adoc description: Let agents read SharePoint sites, document libraries, lists, and files, upload files, and search content through the Microsoft Graph API, using each end-user's own Microsoft identity through user-delegated OAuth. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Register a Microsoft Entra app and an OAuth provider for the SharePoint managed MCP server learning-objective-2: Create the SharePoint managed MCP server with the correct Microsoft Graph scopes learning-objective-3: Authorize a user and run the SharePoint tools from the Inspector or an agent page-git-created-date: "2026-06-03" page-git-modified-date: "2026-08-06" --- The **SharePoint** managed MCP server lets agents read SharePoint sites, document libraries, lists, and files, upload files, and search content through the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/api/resources/sharepoint). Each agent caller authenticates against Microsoft with their own identity through user-delegated OAuth, so every call stays bounded by the signed-in user’s own SharePoint permissions: the server never holds workspace-wide access. After reading this page, you will be able to: - Register a Microsoft Entra app and an OAuth provider for the SharePoint managed MCP server - Create the SharePoint managed MCP server with the correct Microsoft Graph scopes - Authorize a user and run the SharePoint tools from the Inspector or an agent > 📝 **NOTE** > > The SharePoint managed MCP server is in preview. If you don’t see SharePoint in the picker, contact Redpanda support. ## [](#what-this-mcp-server-does)What this MCP server does The SharePoint managed type wraps the Microsoft Graph API (`[https://graph.microsoft.com/v1.0](https://graph.microsoft.com/v1.0)`) and authenticates per user through the gateway’s [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault). Per-user OAuth is the only authentication mode this type supports. It exposes the following tools: | Tool | What it does | | --- | --- | | list_sites | Search for SharePoint sites. The search term passes straight to Graph’s /sites?search= endpoint. An empty search returns no results, so pass * to list every site or a name fragment to narrow the search. | | get_site | Fetch a single site by composite ID (contoso.sharepoint.com,,) or hostname path (contoso.sharepoint.com:/sites/team). | | list_drive_items | Browse a site’s default document library. Lists the library root by default; pass folder_id to descend into a folder. | | get_file_content | Read a file from a site’s document library as UTF-8 text, with its filename, MIME type, and size. | | upload_file | Upload a text file to a site’s document library, into the root or a folder. Requires the Sites.ReadWrite.All scope. | | search_content | Search across all SharePoint sites, files, and list items the user can see. The query supports Keyword Query Language (KQL). | | list_lists | List the SharePoint lists on a site. Hidden lists are excluded. | | list_list_items | List the items in a SharePoint list, with each item’s column values returned as JSON in fields_json. | The file tools operate on the site’s own default document library (`/sites//drive/…​`), never on a user’s personal OneDrive. File content travels as UTF-8 text in both directions: downloads through `get_file_content` are capped at 10 MiB, and uploads through `upload_file` are capped at 4 MiB (the Microsoft Graph simple-upload limit). Binary files aren’t supported; open the `web_url` returned by `list_drive_items` to access them directly. The `list_sites`, `list_drive_items`, and `list_list_items` tools return up to 100 items by default and accept a `page_size` of up to 999. The `search_content` tool returns up to 25 hits by default and accepts a `page_size` of up to 500. The `list_lists` tool takes no `page_size`. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - Access to a Microsoft SharePoint tenant. - Permission to register an app in the [Microsoft Entra admin center](https://entra.microsoft.com) (or an Entra admin who can register one for you). - A way to write the OAuth client secret into the Redpanda Agentic Data Plane secret store. - For most tenants, a tenant admin who can grant admin consent for the Microsoft Graph scopes. See [Grant admin consent](#grant-admin-consent). - Familiarity with [user-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#register-the-microsoft-entra-app)Register the Microsoft Entra app In the Microsoft Entra admin center, go to **App registrations > New registration** and create an app: 1. Give the app a descriptive name, for example, `Redpanda AI Gateway - SharePoint MCP`. 2. Set **Supported account types** to single tenant. 3. Set the **Redirect URI** to platform **Web** with the gateway’s OAuth callback for your cluster: ```no-highlight https://aigw..clusters.rdpa.co/oauth/v1/callback ``` Replace `` with your cluster identifier. The callback path is `/oauth/v1/callback`. 4. After registering, note the **Application (client) ID** and **Directory (tenant) ID** from the app’s Overview page. You need both when you configure the [OAuth provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider). > 💡 **TIP** > > Single-tenant apps must use the tenant-specific Microsoft OAuth endpoints (`[https://login.microsoftonline.com//oauth2/v2.0/…​](https://login.microsoftonline.com/\/oauth2/v2.0/…​);`), not the `/common/` endpoints, which can return `AADSTS` errors for single-tenant apps. Replace `` with the Directory (tenant) ID from the app’s Overview page. ## [](#add-microsoft-graph-permissions)Add Microsoft Graph permissions In the Microsoft Entra admin center, open the app’s **API permissions**, select **Add a permission > Microsoft Graph > Delegated permissions**, then add: | Scope | Why | | --- | --- | | Sites.ReadWrite.All | Read and write everything the server touches: sites, document libraries, files, lists, and upload_file. For read-only deployments, use Sites.Read.All instead, which drops upload_file. | | offline_access | Issues refresh tokens so connections stay valid over time. | The `openid`, `profile`, `email`, and `User.Read` scopes come from the OAuth provider’s standard scopes. > ❗ **IMPORTANT** > > Use the **Microsoft Graph** permissions, not the legacy **Office 365 SharePoint Online** API in the picker. The server calls Microsoft Graph only. > > Use the `Sites.` **scopes, not `Files.`**. The server’s file and list endpoints all live under `/sites//…​`, and its site lookups and content search are also authorized by the `Sites.*` scopes. The `Files.ReadWrite.All` scope additionally grants access to users' personal OneDrive, which the server never uses. ## [](#create-a-client-secret)Create a client secret In the Microsoft Entra admin center, open the app’s **Certificates & secrets** and select **New client secret**. Give it a description, choose an expiry, and select **Add**. Copy the secret value immediately; Microsoft never shows it again. Store the secret in the Agentic Data Plane secret store under an `UPPER_SNAKE_CASE` key, for example, `SHAREPOINT_CLIENT_SECRET`. The OAuth provider references the secret by name, so the plaintext never enters the MCP server configuration. ## [](#grant-admin-consent)Grant admin consent The SharePoint Graph scopes are high-privilege delegated scopes, so most tenants require a tenant admin to consent before any user can connect. A Global Administrator, Privileged Role Administrator, or Cloud Application Administrator opens the app, selects **API permissions**, then selects **Grant admin consent for ``**. Admin consent does not escalate access: delegated scopes always stay bounded by the signed-in user’s own SharePoint permissions. Admin consent approves the app to request the scopes so that individual users aren’t each prompted. Without it, a non-admin user sees the **Need admin approval** message on the consent screen. ## [](#configure-the-oauth-provider)Configure the OAuth provider Register an OAuth provider in Agentic Data Plane that points at Microsoft’s tenant-specific endpoints and references the client secret. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). The **Microsoft / Azure AD** preset in the provider catalog pre-fills the endpoints, standard scopes, PKCE, and token auth method; for a single-tenant app, replace its `/common/` endpoints with the tenant-specific ones. Use these values: - **Authorization endpoint**: `[https://login.microsoftonline.com//oauth2/v2.0/authorize](https://login.microsoftonline.com/\/oauth2/v2.0/authorize)` - **Token endpoint**: `[https://login.microsoftonline.com//oauth2/v2.0/token](https://login.microsoftonline.com/\/oauth2/v2.0/token)` - **Client ID**: The Application (client) ID from the Entra app. - **Client secret reference**: The secret-store key, for example, `SHAREPOINT_CLIENT_SECRET`. - **Token auth method**: Client secret in the POST body, with PKCE enabled (matches the Microsoft preset). - **Scopes**: `openid`, `email`, `profile`, `User.Read`, `Sites.ReadWrite.All`, `offline_access`. The provider page also shows the **Authorization Callback URL** to register as the Entra app’s redirect URI. ## [](#create-the-server)Create the server Create a new SharePoint MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create server**. 2. Pick **SharePoint** from the marketplace picker. 3. Fill in the identity fields (`name`, `description`). 4. Under **Auth**, select **User OAuth** (the only option for this type), then set **Provider Name** to the SharePoint OAuth provider you configured. 5. Under **Required Scopes**, add `Sites.ReadWrite.All` (or `Sites.Read.All` for a read-only server). 6. Click **Submit**. The required scopes are enforced against each user’s connection. A connection with insufficient scopes returns `scope_upgrade_required`, and the user re-consents with the higher scope. ### [](#configure-from-the-cli)Configure from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) to create the server with a managed config. Set the OAuth provider inside the managed config; the top-level `--user-oauth-provider` and `--user-oauth-scopes` flags apply to self-managed servers only. ```bash rpk ai mcp-server create --name sharepoint \ --description "SharePoint MCP over Microsoft Graph with per-user OAuth" \ --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.sharepoint.v1.SharePointMCPConfig", "userOauth": { "providerName": "sharepoint", "requiredScopes": ["Sites.ReadWrite.All"] } }' ``` ## [](#authorize-a-user-and-test)Authorize a user and test The SharePoint server uses per-user OAuth, so each user connects their Microsoft identity before tools work for them: 1. Open the server in Agentic Data Plane. If you haven’t connected to the SharePoint OAuth provider yet, a connection banner with a **Connect** button appears above the server’s tabs. Click **Connect**, sign in to Microsoft, and approve the consent screen. The token lands in the vault keyed to your identity. If you call a tool before connecting, the call returns an `OAuthConnectionRequired` error carrying an `authorize_url` that points at the gateway’s `/oauth/v1/authorize` endpoint. Opening that URL in a browser starts the same Microsoft consent flow. 2. Open the server’s **Inspector** tab and run `list_sites` with the argument `{"search": "*"}`. It returns the sites you can see. See [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/). > 💡 **TIP** > > The `list_sites` tool passes the search term straight to Graph’s `/sites?search=` endpoint, which returns nothing for an empty search. Pass `*` to list every site, or a name fragment to narrow the search. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | OAuthConnectionRequired | First call from a user with no stored token. The user completes Microsoft’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. See User-delegated OAuth. | | Need admin approval on the Microsoft consent screen | A tenant admin hasn’t granted admin consent for the Graph scopes. See Grant admin consent. | | scope_upgrade_required | The user’s connection has fewer scopes than the server requires, either because the server’s required scopes grew or because the user originally consented with fewer. The user re-consents with the higher scope. | | AADSTS errors during authorization | A single-tenant app is using the /common/ OAuth endpoints. Switch the OAuth provider to the tenant-specific endpoints (https://login.microsoftonline.com//oauth2/v2.0/…​;). | | SharePoint API error (status 403, code: accessDenied) | The signed-in user lacks permission on that site, file, or list. Graph errors surface only the HTTP status and Graph error code; the upstream error message is withheld. | | list_sites returns no sites | An empty search returns nothing. Pass * to list every site, or a name fragment. | | upload_file fails | The upload_file tool requires the Sites.ReadWrite.All scope; a read-only server configured with Sites.Read.All cannot upload. Content is also capped at 4 MiB and must be UTF-8 text. | | get_file_content reports the file exceeds the byte cap | Downloads are capped at 10 MiB. Open the file’s web_url directly instead. | | Token expired and no refresh token available | The stored token expired and couldn’t be refreshed. Confirm offline_access is in the OAuth provider’s scopes, then have the user reconnect through the same consent flow. | ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test MCP tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 33: Slack Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/slack.md --- # Slack Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Slack Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/slack page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/slack.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/slack.adoc description: Let agents read threads, post messages, and add reactions in your Slack workspace using a shared bot token or each end-user's own Slack identity through user-delegated OAuth. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Slack managed MCP server with user-delegated OAuth learning-objective-2: Walk through the consent flow and verify the connection on the My Connections page learning-objective-3: Send a test message through the Inspector page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Slack** managed MCP server is the canonical user-delegated OAuth example for Redpanda Agentic Data Plane. Each agent caller authenticates against Slack with their own credentials, and Redpanda injects their token at call time: so messages posted by the agent appear as the user, not as a shared bot. After reading this page, you will be able to: - Configure the Slack managed MCP server with user-delegated OAuth - Walk through the consent flow and verify the connection on the My Connections page - Send a test message through the Inspector ## [](#what-this-mcp-server-does)What this MCP server does The Slack managed type exposes the following tools: - `read_thread`: Read all messages in a thread, given a channel ID and thread timestamp. - `post`: Send a message to a channel, optionally as a threaded reply, using plain text or Block Kit blocks. - `react`: Add or remove an emoji reaction on a message. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Slack workspace where you can install or authorize an OAuth app. - A Slack OAuth app registered (your own or a Redpanda-published reference app). - An [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Agentic Data Plane on the **Outbound providers** tab of **Integrations setup**, pointing at Slack’s authorize/token URLs and carrying the OAuth app’s client credentials. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). - Familiarity with [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#configure)Configure Create a new Slack MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick **Slack** from the marketplace picker. 3. Fill in the identity fields (`name`, `description`). 4. In the Slack configuration form: - `Auth`: Choose a `Bot token` (a shared `xoxb-` token from the secret store) or `User-delegated OAuth` (per-user identity). - For `User-delegated OAuth`: Pick the Slack OAuth Provider you configured and set `Required scopes` for the tools you use (see [Authentication scopes](#authentication-scopes)). - `Post options` (optional): Toggle markdown formatting, link unfurling, media unfurling, and automatic link names. - `Reaction options` (optional): Maximum concurrent reaction calls. Defaults to 64. 5. Click **Create**. There’s no workspace field: the bot token or the user’s OAuth grant determines the workspace. ## [](#test-the-consent-flow)Test the consent flow After creating the server, run a tool that requires Slack authentication to verify the consent flow end-to-end: 1. Open the **Inspector** tab. 2. Run a tool that requires the user’s identity, for example `post`. 3. The first call returns `OAuthConnectionRequired` with a Slack `authorize_url`. The Inspector surfaces it as a consent prompt. 4. Click **Authorize**. You’re redirected to Slack; pick the workspace and approve the requested scopes. 5. Slack redirects back to Agentic Data Plane. Your connection now appears under **My Connections**. 6. Re-run the original tool call. The message posts to Slack as your user. ## [](#authentication-scopes)Authentication scopes When you use user-delegated OAuth, the tools need Slack scopes that match what they do: - `post` needs `chat:write`. To post in channels the user isn’t a member of, add `chat:write.public`. - `read_thread` needs a history scope for the channel type, for example `channels:history` (public) or `groups:history` (private). - `react` needs `reactions:write`. Other gotchas: - `chat:write` is a _user_ scope; `chat:write.public` is separate. - Tokens are workspace-scoped: the same user authorizing twice across two workspaces produces two separate connections. ## [](#use-with-agents)Use with agents After the server is created and at least one user has consented, you can point an agent at the `Server URL` on the server’s **Connection** tab. Each user calling the agent will trigger their own consent flow if they haven’t connected yet. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | OAuthConnectionRequired even after consent | The token might be revoked or expired with no refresh. Check My Connections; disconnect and reconnect if needed. | | scope_upgrade_required | The server’s required scopes were widened after the user consented. The user re-consents to grant the new scope. | | "channel not found" on a channel the user can see in Slack | Reading a private channel’s thread needs the groups:history scope. Add it to the server’s required scopes and have users re-consent. | | invalid_auth from Slack | The OAuth Provider’s client credentials are wrong, or the OAuth app has been suspended in Slack. Check the provider config. | ## [](#limitations)Limitations This page does not cover: - **Configuring the Slack OAuth app**: Slack-side configuration (creating the app, picking redirect URIs, choosing scopes) happens in api.slack.com, not in Agentic Data Plane. - **Listing channels and looking up users**: Not exposed as tools. The server reads threads, posts messages, and adds reactions. --- # Page 34: SQL Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/sql.md --- # SQL Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: SQL Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/sql page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/sql.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/sql.adoc description: Query and write to SQL databases (PostgreSQL, MySQL, ClickHouse, SQL Server, and SQLite) through a managed MCP server hosted by Redpanda. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the SQL managed MCP server for your database learning-objective-2: Run a canonical SELECT query through the Inspector learning-objective-3: Pick the right driver and connection-string pattern for your database page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-12" --- The **SQL** managed MCP server gives agents read and write access to a SQL database through MCP. Redpanda runs the server in-process; you provide a connection string. Guardrails can restrict it to read-only. After reading this page, you will be able to: - Configure the SQL managed MCP server for your database - Run a canonical SELECT query through the Inspector - Pick the right driver and connection-string pattern for your database ## [](#what-this-mcp-server-does)What this MCP server does Redpanda’s SQL managed type proxies SQL statements to one of these database engines: - PostgreSQL (`postgres` or `pgx`) - MySQL / MariaDB - ClickHouse - Microsoft SQL Server (`mssql`) - SQLite (`sqlite3`) It exposes the following tools: - `query`: Run a read-only `SELECT` and return rows. Accepts positional parameters and an optional max-rows cap. - `execute`: Run a write statement (`INSERT`, `UPDATE`, `DELETE`, or DDL) and return the affected row count. Guardrails on the server config can restrict it: set `readonly` to disable `execute`, cap returned rows, restrict allowed schemas, block statement patterns, and set a per-query timeout. ## [](#prerequisites)Prerequisites - A SQL database reachable from the Agentic Data Plane. - A connection string (DSN) for the database, including the credentials. The DSN is stored as a sensitive value. ## [](#configure)Configure 1. Open **MCP Servers > Create Server**. 2. Pick **SQL** from the marketplace picker. 3. Fill in the identity fields (`name`, `description`). 4. In the SQL configuration form, provide: - `Driver`: One of `postgres`, `pgx`, `mysql`, `clickhouse`, `mssql`, or `sqlite3`. - `DSN`: The driver-specific connection string, including credentials. Stored as a sensitive value. - `Connection pool` (optional): Maximum open and idle connections, and connection lifetimes. - `Guardrails` (optional): `readonly` (disables `execute`), default and maximum row caps, a per-query timeout, allowed schemas, and blocked statement patterns. - `Row Format` (optional): How the `query` tool encodes result rows. Leave it unset (or select Positional) for the default, or select Object to key each row by column name. See [Choose a row format](#row-format). 5. Click **Create**. ## [](#test)Test After create, exercise the server through the Inspector tab. See [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/). A canonical first call: 1. Open the **Inspector** tab. 2. In **Tools**, select the `query` tool. 3. Run `SELECT 1` (or your driver’s equivalent). Confirm a single-row response. 4. To list tables, run `query` against the catalog, for example `SELECT table_name FROM information_schema.tables` on PostgreSQL or MySQL. ## [](#row-format)Choose a row format The `query` tool can encode result rows in two ways, controlled by the server’s `Row Format` setting: - Positional (the default): Each row is an array of values addressed by column position. This is the denser encoding. - Object (`ROW_FORMAT_OBJECT`): Each row is a JSON object keyed by column name. Choose the Object format when you want an [data policy](https://docs.redpanda.com/agentic-data-plane/connect/data-policies/) to mask, redact, or filter individual columns. A data policy selects fields by name, so with named objects it can target a single named column, for example a `salary` column, and leave the rest readable. With positional rows a policy can act only on the whole row, because there are no column names to select. The two encodings are mutually exclusive: the server returns one or the other, never both. ## [](#authentication)Authentication The SQL managed type has no MCP-level authentication modes. Credentials are carried inside the `DSN` (connection string), which is stored as a sensitive value. Supply them the way your driver expects, for example `postgres://user:password@host:5432/dbname`. User-delegated OAuth and service-account OAuth are not supported for SQL: there’s no per-user identity model that maps to a database connection. ## [](#use-with-agents)Use with agents After the SQL server is created, point an agent at the `Server URL` on the server’s **Connection** tab. The agent sees the SQL tools alongside any other MCP servers it has access to. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | connection refused from create | Database isn’t reachable from Agentic Data Plane. Confirm host, port, and any egress / firewall rules. | | authentication failed for user | Password reference points at the wrong secret, or the secret content is wrong. Re-create the secret with UPPER_SNAKE_CASE and verify it. | | Query timeout | Long-running queries exceed the configured timeout. Either tighten the query or raise the timeout on the server config. | | Tools list returns empty | The driver couldn’t introspect the schema. Verify the user has read access to information_schema (Postgres/MySQL) or the equivalent. | ## [](#limitations)Limitations - **Write access**: The `execute` tool runs writes and DDL unless `readonly` is set in the guardrails. Scope the database user’s permissions accordingly. - **Per-user database identities**: See [Authentication](#authentication). --- # Page 35: Workday Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/workday.md --- # Workday Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Workday Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/workday page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/workday.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/workday.adoc description: Drive Workday Human Resources business processes from an LLM agent. The Workday managed MCP wraps Workday's Human_Resources SOAP web services and authenticates with a service-account refresh-token grant. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Workday managed MCP server with an Integration System User (ISU) refresh token learning-objective-2: Choose the right WSDL version and tenant settings learning-objective-3: Run a Change_Personal_Information business process from the Inspector or an agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Workday** managed MCP server lets agents drive Workday Human Resources business processes (multi-step, approval-driven workflows like onboarding, hiring, and personal-info changes) through Workday’s `Human_Resources` SOAP API. After reading this page, you will be able to: - Configure the Workday managed MCP server with an Integration System User (ISU) refresh token - Choose the right WSDL version and tenant settings - Run a Change\_Personal\_Information business process from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Workday is a SaaS HR and payroll platform. Customer mutations land through **business processes**: multi-step, approval-driven workflows. Workday’s REST API covers a partial read-side surface, but the business processes themselves live behind the SOAP `Human_Resources` WSDL. This MCP wraps the SOAP surface so an LLM can drive a business process the same way it would call any other tool. It is **not** a generic Workday browser. There is no SQL/RaaS access, no report execution, and no general "search the tenant" tool. Each MCP tool maps 1:1 to one business process. The current build exposes a single tool, `change_personal_information`, with more business processes landing as customers ask for them. ## [](#authentication-model)Authentication model Workday’s `Human_Resources` SOAP API authenticates with the OAuth 2.0 **refresh-token grant** plus HTTP Basic on the token endpoint. Unlike most managed MCPs, this is a vendor-specific auth shape that doesn’t fit the shared `static_key`, `service_account_oauth`, or `user_delegated_oauth` modes; Workday uses an `oauth_refresh_token` variant. The MCP exchanges the refresh token (in the request body) plus `username:password` (HTTP Basic) for a short-lived access token at `https:///ccx/oauth2//token`, then sends `Authorization: Bearer ` on every SOAP call. Authentication is one ISU per MCP instance, not per end-user. Customers that need per-user-delegated access mount multiple MCP instances (one per ISU/scope), not multiple users behind one MCP. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Workday tenant where you can create an Integration System User and register an API client - Admin access to **Workday > Create Integration System User** and **Workday > Register API Client for Integrations** - Two Redpanda Agentic Data Plane secret-store entries: - `WORKDAY_PASSWORD`: The ISU password. - `WORKDAY_REFRESH_TOKEN`: The non-expiring refresh token. ## [](#get-workday-credentials)Get Workday credentials Set up authentication on the Workday side before configuring the MCP: 1. **Create an Integration System User (ISU)** under **Workday > Create Integration System User**. Note the username; it usually ends up as `@`. 2. **Register an API Client for Integrations** under **Workday > Register API Client for Integrations**: - **Grant types**: Include both `Refresh Token` (required) and `Authorization Code`. Workday’s UX requires both to be checked even when only the refresh-token grant is used at runtime. - **Non-Expiring Refresh Tokens**: Tick this option. Required for static-credential MCP usage; if Workday rotates the refresh token on every exchange, the cached value goes stale and authentication breaks. - **Scope**: Include `Human Resources` (and any other functional areas your business processes touch). 3. **Issue a refresh token to the ISU** by completing the one-time authorization-code flow Workday walks you through, or by using **View API Clients > Manage Refresh Tokens for Integrations** to mint one directly. 4. Save four values: the `tenant`, the `host` (the Workday data-center hostname, for example `wd2-impl-services1.workday.com`), the ISU `username`, and the ISU `password`. Save the `refresh_token` separately. ## [](#configure)Configure Create a new Workday MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick **Workday** from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Workday configuration form: | Field | Notes | | --- | --- | | Tenant | Your Workday tenant identifier, for example acme. | | Host | The Workday data-center hostname, for example wd2-impl-services1.workday.com. The MCP exchanges credentials at https:///ccx/oauth2//token. | | WSDL version | Optional; defaults to v46.0. Older tenants on v44.x or v45.x must set this explicitly to match the WSDL surface their tenant has enabled. | | Username | The ISU username (typically @). | | Password ref | Secret-store reference for the ISU password (UPPER_SNAKE_CASE). Example: WORKDAY_PASSWORD. | | Refresh token ref | Secret-store reference for the non-expiring refresh token (UPPER_SNAKE_CASE). Example: WORKDAY_REFRESH_TOKEN. | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI ```bash rpk ai mcp-server create --name workday-hr --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.workday.v1.WorkdayMCPConfig", "tenant": "acme", "host": "wd2-impl-services1.workday.com", "wsdl_version": "v46.0", "oauth_refresh_token": { "username": "isu_user@acme", "password_secret_ref": "${secrets.WORKDAY_PASSWORD}", "refresh_token_secret_ref": "${secrets.WORKDAY_REFRESH_TOKEN}" } }' ``` ## [](#tools)Tools The Workday MCP exposes the following tools: | Tool | Description | | --- | --- | | change_personal_information | Kicks off the Change_Personal_Information business process for a worker. All fields except worker_id are optional. Only fields you set are sent to Workday, leaving the rest of the worker’s personal data unchanged. | ### [](#example-change-a-workers-date-of-birth-and-marital-status)Example: Change a worker’s date of birth and marital status ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/workday-hr \ -H 'Content-Type: application/json' -d '{ "jsonrpc":"2.0","method":"tools/call","id":1, "params":{ "name":"change_personal_information", "arguments":{ "worker_id":"E1001", "worker_id_type":"Employee_ID", "effective_date":{"year":2026,"month":5,"day":1}, "date_of_birth":{"year":1990,"month":5,"day":20}, "marital_status":"Married" } } }' ``` Dates use the `google.type.Date` shape (`{year, month, day}`); a missing field, or one with `year: 0`, is treated as "unset" and Workday applies its own default (today, for `effective_date`). A successful response surfaces the Workday Event WID and confirms the worker WID: ```json { "result": { "content": [{ "type": "text", "text": "{\"event_wid\":\"ev-wid-001\",\"worker_wid\":\"worker-wid-002\",\"version\":\"v46.0\"}" }] } } ``` If Workday returns a SOAP Fault (validation error, missing permissions, worker not found), the MCP surfaces the `faultstring` as a structured tool error so the LLM can decide whether to retry or ask the user. ## [](#tenant-specific-values)Tenant-specific values `gender`, `marital_status`, and `citizenship_status_ids` accept Workday IDs from the **customer’s** tenant configuration. Common defaults like `Single` / `Married` and ISO country codes work in most tenants, but check Workday’s "Maintain Marital Status" and "Maintain Citizenship Status" reports if a value is rejected. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | 401 Unauthorized on token exchange | ISU credentials wrong, or the refresh token has been rotated. Confirm WORKDAY_PASSWORD and WORKDAY_REFRESH_TOKEN in the secret store are correct, and re-mint the refresh token in View API Clients > Manage Refresh Tokens for Integrations if needed. | | invalid_grant on every refresh | Non-Expiring Refresh Tokens was not checked when you registered the API client. Edit the client, tick the option, and re-mint the refresh token. | | SOAP fault: Invalid_Field_Value | A tenant-specific field ID (marital status, citizenship status, ethnicity) doesn’t match what your tenant accepts. Check the corresponding "Maintain …​" report in Workday for the exact IDs. | | SOAP fault: Insufficient_Permissions | The ISU lacks rights for the business process you’re invoking. Grant the relevant security domain on the ISU’s security group. | | SOAP fault: Worker_Not_Found | The worker_id plus worker_id_type doesn’t resolve. Verify the type (Employee_ID, Workday_ID, Contingent_Worker_ID) and the value. | ## [](#limitations)Limitations This page does not cover: - **Per-user-delegated access**: Workday authentication is one shared ISU per MCP. For per-user identities, mount multiple MCP instances (one per ISU/scope). - **Custom report execution**: This MCP wraps SOAP business processes, not reports. Use Workday RaaS or the report API for custom reports. - **Read-side data exploration**: There is no general _search Workday_ tool. Add specific business-process tools as needed. ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Test a server’s tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 36: Zendesk Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/managed/zendesk.md --- # Zendesk Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Zendesk Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed/zendesk page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: managed/zendesk.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/managed/zendesk.adoc description: Search and manage Zendesk Support tickets, users, and Help Center articles. Service-account API token or per-user OAuth, plus token-efficient response shaping for LLM agents. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Configure the Zendesk managed MCP server in API-token or User-OAuth mode learning-objective-2: Pick the right scopes and Zendesk role for your workflows learning-objective-3: Search, read, create, and update tickets from the Inspector or an agent page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- The **Zendesk** managed MCP server lets agents search, read, create, and update tickets in your Zendesk Support instance, look up users and organizations, and search, read, create, and update Help Center articles. After reading this page, you will be able to: - Configure the Zendesk managed MCP server in API-token or User-OAuth mode - Pick the right scopes and Zendesk role for your workflows - Search, read, create, and update tickets from the Inspector or an agent ## [](#what-this-mcp-server-does)What this MCP server does Wraps the Zendesk REST API. The following authentication modes are supported: - **API token (Basic authentication)**: A long-lived agent token paired with the agent’s email. Best for service-account-style use. - **User OAuth**: Per-user Zendesk OAuth tokens resolved from the gateway’s [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault). Best when you want each agent action attributed to the calling end-user. Responses are curated for token efficiency: HATEOAS URLs, transport metadata, and rarely-used fields are dropped before reaching the LLM. Related users, groups, and organizations are resolved into nested ref objects through Zendesk side-loading (single round trip), and Help Center article HTML is converted to GitHub-flavored markdown. Typical responses are 3–7× smaller than raw Zendesk JSON. It is **not** intended for Zendesk admin operations (managing macros, triggers, ticket forms, custom fields, schedules, or SLAs); use the Zendesk Admin Center or a Terraform provider for those. ## [](#prerequisites)Prerequisites Before you create the server, make sure you have: - A Zendesk Support instance. - For **API token** mode: ability to create an API token under **Apps and integrations > APIs > Zendesk API**. - For **User OAuth** mode: a Zendesk OAuth client and an [OAuth Provider](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#oauth-provider) configured in Redpanda Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). ## [](#get-zendesk-credentials)Get Zendesk credentials ### [](#option-1-api-token-recommended-for-service-accounts)Option 1: API token (recommended for service accounts) 1. In the Zendesk Admin Center, go to **Apps and integrations > APIs > Zendesk API**. 2. On the **Settings** tab, enable **Token access**. 3. Click **Add API token**, give it a descriptive label (for example, `redpanda-ai-gateway`), and copy the token value. It is shown only once. 4. Note the **agent email** the token will act as (the email of the user who created the token). The HTTP Basic authentication string the MCP builds is `base64(/token:)`. The `/token` literal is Zendesk’s API-token quirk. 5. Store the token in the Agentic Data Plane secret store under a name like `ZENDESK_API_TOKEN`. **Required role**: Agents and Admins can use the API. Most ticket operations work for the Agent role; reading users with `search_users` requires Light Agent or higher; Help Center search works for any authenticated user. ### [](#option-2-user-oauth)Option 2: User OAuth For per-user authentication, register an OAuth client on Zendesk and a matching OAuth Provider in Agentic Data Plane: 1. Configure a Zendesk OAuth client under **Apps and integrations > APIs > OAuth Clients** (Confidential client, Authorization Code grant). 2. Register a matching OAuth Provider in Agentic Data Plane. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). Use Zendesk’s authorize and token endpoints. 3. Each end-user authenticates once through the OAuth flow; tokens are stored in the gateway’s token vault. **Required scopes**: `read tickets:write hc:read hc:write` covers all tools. Drop `tickets:write` (ticket writes) and `hc:write` (article create/update) if the MCP only needs to read. ## [](#configure)Configure Create a new Zendesk MCP server in Agentic Data Plane: 1. Open **MCP Servers > Create Server**. 2. Pick `Zendesk` from the marketplace picker. 3. Fill in identity fields (`name`, `description`). 4. In the Zendesk configuration form: | Field | Notes | | --- | --- | | subdomain | Your Zendesk subdomain (the part before .zendesk.com). For acme.zendesk.com, set this to acme. | | auth | basic_auth for API-token mode, or user_oauth for per-user mode. | | basic_auth (API-token mode) | username (agent email used with the API token, for example agent@acme.com) and password_secret_ref (secret-store reference for the API token, UPPER_SNAKE_CASE). | | user_oauth (User-OAuth mode) | provider_name (the Zendesk OAuth Provider you configured) and the minimum required scopes (read, tickets:write, hc:read, hc:write covers all tools). | 5. Click **Create**. ### [](#configure-from-the-cli)Configure from the CLI For a managed server, set the auth method inside the `--managed-config` JSON. The `auth` field is required. #### API-token mode ```bash rpk ai mcp-server create --name acme-zendesk --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.zendesk.v1.ZendeskMCPConfig", "subdomain": "acme", "basic_auth": { "username": "agent@acme.com", "password_secret_ref": "ZENDESK_API_TOKEN" } }' ``` #### User-OAuth mode ```bash rpk ai mcp-server create --name acme-zendesk-oauth --managed-config '{ "@type": "type.googleapis.com/redpanda.mcps.zendesk.v1.ZendeskMCPConfig", "subdomain": "acme", "user_oauth": { "provider_name": "zendesk-prod", "required_scopes": ["read", "tickets:write", "hc:read"] } }' ``` ## [](#tools)Tools The Zendesk MCP exposes tools across tickets, users, organizations, and Help Center articles. Article writes (`create_article`, `update_article`) require the `hc:write` scope (User-OAuth mode) or a Help Center manager role (API-token mode): | Tool | Description | | --- | --- | | search_tickets | Search tickets with Zendesk’s search syntax (status:open priority:high tags:bug). The handler enforces a type:ticket qualifier and post-filters results, so non-ticket records can never leak through. | | list_tickets | Filter tickets by discrete fields (status, assignee_id, group_id). Faster than search_tickets and not subject to the search-API quota. Prefer this for "show me all open tickets assigned to X" workflows. | | get_ticket | Fetch a single ticket by ID with side-loaded requester, submitter, assignee, group, and organization. Set include_comments=true to embed up to 500 comments inline in the same response. | | list_ticket_comments | List the comment thread on a ticket with explicit pagination. Use this when the thread exceeds 500 comments; otherwise prefer get_ticket with include_comments=true. | | create_ticket | Open a new ticket (subject, description, optional priority/type/assignee/group/tags). Subject ≤150 chars, description ≤65536 chars, tags ≤50 chars each. | | update_ticket | Modify a ticket: status, priority, type, assignee, group, tags. Optionally append a public or internal comment in the same call. Distinct tags (replace) / add_tags / remove_tags fields. | | search_users | Find a Zendesk user by name, email, or other user-search fields. Returns full User objects. | | get_user | Fetch a single user by ID. Used to drill into a UserRef from a side-load when the agent needs the full User shape. | | list_organizations | List organizations in the Zendesk account. | | get_organization | Fetch a single organization by ID. Drills into an OrganizationRef from a side-load. | | search_articles | Search Help Center articles. Body is converted from HTML to GitHub-flavored markdown (tables included). | | get_article | Fetch a single Help Center article by ID. | | create_article | Create a new Help Center article from markdown (converted to HTML on upload), with an optional list of labels. The article is always created as a draft, so a human must publish it in Zendesk. | | update_article | Update an existing Help Center article. Only the fields you send change; omitted fields are left as-is. Markdown is converted to HTML on upload, and you can toggle draft status or clear all labels by sending an empty label list. | ### [](#example-triage-open-tickets)Example: Triage open tickets ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-zendesk \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "search_tickets", "arguments": { "query": "status:open priority:urgent", "max_results": 10 } } }' ``` ### [](#example-solve-a-ticket-with-a-closing-comment)Example: Solve a ticket with a closing comment ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-zendesk \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "update_ticket", "arguments": { "ticket_id": 12345, "status": "solved", "add_tags": ["resolved-by-agent"], "comment": { "body": "Resetting your password should fix this. Reopen if it persists.", "public": true } } } }' ``` ### [](#example-read-a-ticket-with-its-full-comment-thread)Example: Read a ticket with its full comment thread For "summarize this ticket" flows, inline the comments: ```bash curl -s https://aigw..clusters.rdpa.co/mcp/v1/acme-zendesk \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": { "name": "get_ticket", "arguments": { "ticket_id": 12345, "include_comments": true } } }' ``` The handler follows Zendesk’s `next_page` URLs only when they point at the same host as the configured subdomain, so pagination cannot be hijacked by a malicious upstream response. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | 401 Unauthorized (API-token mode) | Confirm ZENDESK_API_TOKEN content matches the value Zendesk showed at token creation, and the Username field is the agent email of the user who created the token. | | 403 Forbidden on search_users | The agent role on Zendesk’s side is below Light Agent. Upgrade the role or use API-token mode with a Light Agent or Admin email. | | OAuthConnectionRequired (User-OAuth mode) | First call from a user with no stored token. The user completes Zendesk’s OAuth consent flow, the token lands in the vault, and subsequent calls reuse it. See User-delegated OAuth. | | scope_upgrade_required (User-OAuth mode) | Server’s required_scopes was extended after users had already consented. Users re-consent with the higher scope. | | Search returns non-ticket records | Cannot happen: the handler enforces a type:ticket qualifier and post-filters results. If you see something unexpected, file an issue. | ## [](#limitations)Limitations This page does not cover: - **Zendesk admin operations**: Managing macros, triggers, ticket forms, custom fields, schedules, or SLAs. Use the Zendesk Admin Center or a Terraform provider. - **Voice / chat / Talk**: This MCP wraps Support tickets and Help Center; voice and chat are separate Zendesk products with their own APIs. ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) --- # Page 37: How MCP Servers Work **URL**: https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview.md --- # How MCP Servers Work > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: How MCP Servers Work latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: mcp-overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: mcp-overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/mcp-overview.adoc description: Connect AI agents to your databases, APIs, and business systems through managed MCP servers Redpanda hosts for you, or self-managed servers you register with Agentic Data Plane. page-topic-type: overview personas: agent_builder, platform_engineer learning-objective-1: Explain how AI agents connect to tools through MCP servers in Agentic Data Plane learning-objective-2: Distinguish managed from self-managed MCP server backends and choose the right type for your use case learning-objective-3: Identify key MCP capabilities including tool discovery, authentication, and code mode page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- The Model Context Protocol (MCP) is how AI agents talk to tools. In the Agentic Data Plane, you create **MCP Servers** that agents can connect to. Each MCP server exposes a set of [tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool), [resources](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#resource), and [prompts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#prompt) that an agent can discover at runtime through `tools/list` and invoke through structured JSON-RPC calls. Redpanda handles upstream credentials, authentication flows, observability, and aggregation, so you can focus on what your agents do, not on the plumbing. Current MCP versions keep discovery and calls interoperable while allowing clients and servers to negotiate newer capabilities, such as long-running work and richer user experiences. The Inspector shows the negotiated protocol version and only surfaces capabilities that the server advertises. After reading this page, you will be able to: - Explain how AI agents connect to tools through MCP servers in Agentic Data Plane - Distinguish managed from self-managed MCP server backends and choose the right type for your use case - Identify key MCP capabilities including tool discovery, authentication, and code mode ![MCP Gateway connection flow. An Agentic Data Plane agent or external MCP client authenticates to MCP Gateway, discovers and calls tools, and receives results. MCP Gateway applies identity, access, authentication, policies, and observability before routing to either a Redpanda-managed MCP implementation or a self-managed MCP endpoint, which then reaches the upstream business system.](https://docs.redpanda.com/agentic-data-plane/connect/_images/mcp-gateway-connection-flow.svg) Figure 1. MCP Gateway gives managed and self-managed servers one governed tool path Redpanda offers two kinds of MCP server backends: | Backend | When to use | | --- | --- | | Managed | Redpanda hosts the server in-process. You pick a type from the marketplace (SQL, Kafka, Slack, Jira, OpenAPI, and many more) and configure it with your credentials. No infrastructure to run. | | Self-managed | You already run an MCP server somewhere and want Redpanda to proxy it for authentication, observability, aggregation, and agent access. Redpanda fronts your server with a managed URL and resolves authentication at the gateway. | The default catalog covers databases, cloud and productivity services, streaming systems, and communication tools. See [Managed catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) for the full list. ## [](#when-to-use-each)When to use each | Concern | Managed | Self-managed | | --- | --- | --- | | Infrastructure ownership | Redpanda hosts and operates | You host and operate | | Upstream credentials | Redpanda secret store; per-type schema | Redpanda secret store or token vault; you control the upstream authentication | | Custom tool logic | Limited to what the type exposes | Anything the MCP protocol supports | | Startup latency | In-process; effectively zero | Network round-trip to your endpoint | | Upgrade cadence | Redpanda manages | You manage | ## [](#whats-in-redpanda-agentic-data-plane)What’s in Redpanda Agentic Data Plane Agentic Data Plane has these top-level areas: - **LLM Providers**: OpenAI, Anthropic, Bedrock, Google AI, OpenAI-compatible endpoints. See [Configure your LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). - **MCP Servers**: You are here. Both managed and self-managed servers live in this list. - **My Connections**: Each user’s own per-user OAuth connections for user-delegated MCP servers. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). - **Integrations setup**: The admin home for OAuth plumbing. Its **Outbound providers** tab holds reusable upstream OAuth provider definitions used by user-delegated MCP authentication (see [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/)); its **Inbound clients** tab holds external tools registered to request access tokens from the gateway (see [Connect Remote MCP Clients](https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients/)). ## [](#key-capabilities)Key capabilities - **Tool discovery**: After you create or register a server, Redpanda performs a live `tools/list` against it and populates the server’s detail page so you can see which tools are visible to agents. - **Service-account and user-delegated authentication**: Pick a single shared upstream identity for all callers, or have each end-user authenticate against the upstream system with their own credentials. - **Code mode**: For a server with a large tool catalog, turn on code mode to expose just two tools, `search` and `execute`, so an agent can find tools by regex and run them through a JavaScript sandbox instead of loading every tool definition into its context. See [Code Mode](https://docs.redpanda.com/agentic-data-plane/gateway/code-mode/). - **Inspector**: Test each tool, resource, and prompt directly from Agentic Data Plane before pointing an agent at the server. See [Test a server’s tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/). ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Register a Self-Managed MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/) - [Plug in an App, Database, or Tool](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) --- # Page 38: Connect Tools with MCP Servers **URL**: https://docs.redpanda.com/agentic-data-plane/connect/mcp-servers.md --- # Connect Tools with MCP Servers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect Tools with MCP Servers latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: mcp-servers page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: mcp-servers.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/mcp-servers.adoc description: Learn how MCP servers expose tools and data to agents, then plug in a managed type or build, register, and test your own. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-06-18" --- MCP servers expose tools and data to your agents. Plug in a managed type from the catalog, or build, register, and test a server of your own. - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) Connect AI agents to your databases, APIs, and business systems through managed MCP servers Redpanda hosts for you, or self-managed servers you register with Agentic Data Plane. - [Plug in an App, Database, or Tool](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) Reference of every managed MCP server type Redpanda hosts in-process, grouped by category, with display name, description, and a link to a setup guide where one exists. - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) Create an MCP server in the Agentic Data Plane by picking a managed type from the marketplace or registering a self-managed server you host yourself, then configure authentication, code mode, and access. - [Register a Self-Managed MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/register-remote/) Front your own MCP server with a managed Redpanda URL. Pick a transport, configure authentication, and let agents discover the server's tools through Redpanda. - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) Use the Inspector tab to call tools, inspect resources and prompts, and verify your MCP server works without standing up an agent. - [Shape Tool Data with Data Policies](https://docs.redpanda.com/agentic-data-plane/connect/data-policies/) Add data policies to an MCP server to mask, drop, or filter the data its tools send and return, restrict argument values, and preview the effect before saving. - [Code Mode](https://docs.redpanda.com/agentic-data-plane/gateway/code-mode/) Turn on code mode for an MCP server to replace a large tool catalog with two tools, search and execute, and cut the token cost of tool-heavy servers. --- # Page 39: Configure an OAuth Provider **URL**: https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers.md --- # Configure an OAuth Provider > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Configure an OAuth Provider latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: oauth-providers page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: oauth-providers.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/oauth-providers.adoc description: Register an OAuth provider in Agentic Data Plane so MCP servers can authenticate users (or service accounts) against an upstream system like Slack, Jira, GitHub, or Salesforce. page-topic-type: how-to personas: platform_engineer, security_compliance_lead learning-objective-1: Register an OAuth provider for an upstream system you want MCP servers to authenticate against learning-objective-2: Grant the right permissions so principals can attach the provider to MCP servers learning-objective-3: Edit, rotate credentials on, or delete an OAuth provider page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-04" --- Register an OAuth provider so AI Gateway can authenticate against an upstream system on behalf of users or service accounts. Any MCP server for that upstream attaches to the provider instead of managing its own credentials. After completing this guide, you will be able to: - Register an OAuth provider for an upstream system you want MCP servers to authenticate against - Grant the right permissions so principals can attach the provider to MCP servers - Edit, rotate credentials on, or delete an OAuth provider > ❗ **IMPORTANT** > > OAuth providers and OAuth clients govern **opposite directions** of authentication and are separate resources. > > - An **OAuth provider** (this page) governs **outbound** authentication: AI Gateway authenticating to an upstream system (GitHub, Slack, Salesforce, and so on) on a user’s behalf so MCP servers can call that upstream. > > - An **OAuth client** governs **inbound** authentication: an external app (Claude Desktop, ChatGPT, Copilot Studio) authenticating to AI Gateway so the app’s users can invoke MCP tools. > > > Both are managed under the **Integrations setup** entry in the sidebar: OAuth providers on the **Outbound providers** tab, OAuth clients on the **Inbound clients** tab. They remain separate resources with separate API definitions, permissions, and lifecycles. To register or manage an OAuth client (including revoking its refresh tokens to force a re-sign-in), see [Connect remote MCP clients to AI Gateway](https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients/). ## [](#prerequisites)Prerequisites Before you register the provider, make sure you have: - An OAuth 2.0 application registered with the upstream provider, with the gateway’s redirect URI configured. The redirect URI is the AI Gateway’s OAuth callback (typically `https://aigw..clusters.rdpa.co/oauth/v1/callback`). - The OAuth app’s **client ID** and **client secret**. - A secret already created in the Redpanda Agentic Data Plane secret store for the client secret. Secret references must be `UPPER_SNAKE_CASE`, for example `SLACK_CLIENT_SECRET`. - The list of **scopes** the upstream API needs. Include every scope any MCP server attached to this provider may need; users re-consent when scopes are added later. ## [](#required-permissions)Required permissions OAuth providers are governed by their own permission set. Among the built-in roles only Admin holds these permissions, so anyone else acts on OAuth providers through an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). A policy names the matching `OAuthProvider` actions: `get`, `list`, `create`, `update`, `delete`, and `attach`. The `attach` action is worth separating from the rest. A policy can grant someone the ability to bind an existing provider to an MCP server without also letting them create or edit providers. | Permission | Allows | | --- | --- | | dataplane_aigateway_oauthprovider_create | Create new OAuth providers. | | dataplane_aigateway_oauthprovider_get | Read existing OAuth providers. | | dataplane_aigateway_oauthprovider_update | Edit an existing OAuth provider’s endpoints, scopes, or credentials. | | dataplane_aigateway_oauthprovider_delete | Delete an OAuth provider. | | dataplane_aigateway_oauthprovider_attach | Required to attach this provider to an MCP server. Enforced as a sub-resource check in CreateMCPServer and UpdateMCPServer whenever authConfig.userOauth.provider_name is set or swapped. Without this permission, a principal with mcpserver_update could otherwise bind any provider’s token vault to an MCP they control and indirectly consume its tokens. | > 📝 **NOTE** > > The `_attach` permission is independent from `_get`, `_create`, `_update`, and `_delete`. Grant it only to the principals that need to bind a given provider’s token vault to an MCP server, and scope the policy to that provider with `resource == OAuthProvider::""`. ## [](#browse-oauth-providers)Browse OAuth providers Open **Integrations setup** in the sidebar and select the **Outbound providers** tab. The tab lists every provider registered in your organization. (The older `/oauth-providers` path still works but redirects here.) The list shows the following columns: | Column | What it shows | | --- | --- | | Name | The provider’s machine identifier (used in MCP server configuration to attach this provider). | | Grant types | A badge per grant type. Typically Browser consent for user-delegated OAuth. | | Status | Enabled or Disabled. | | Scopes | A chip list of the supported scopes, for example read:user, repo, read:org for a GitHub provider. | A **Filter** button narrows the list. The **Create provider** button opens the create form. ## [](#register-an-oauth-provider-in-the-ui)Register an OAuth provider in the UI Walk through the create form to register the upstream: 1. Open **Integrations setup** in the sidebar and select the **Outbound providers** tab. 2. Click **Create provider**. 3. Pick how to configure the provider: - Choose a preset from the catalog to pre-fill the standard authorization and token endpoints and recommended scopes. - Choose **Custom Provider** to enter the endpoints yourself. - Choose **Discover from MCP server URL** to point the gateway at a remote MCP server and let it detect the OAuth configuration for you. See [Discover a provider from an MCP server URL](#discover-a-provider-from-an-mcp-server-url). The catalog groups presets into categories you can filter with the category chips: **Identity & SSO**, **Code & Dev**, **Productivity**, **Files & Storage**, **Communication**, **CRM & Support**, **Data Platform**, **Observability**, **Infrastructure**, and **HR & Legal**. It covers common providers across these categories, such as GitHub, Google, Okta, Microsoft, and Slack, with more added over time. For any upstream without a preset, use **Custom Provider** and enter its endpoints manually. 4. Fill in the identity fields: | Field | Required | Notes | | --- | --- | --- | | Name | Yes | Lowercase letters, numbers, and hyphens only. Used to reference the provider in MCP server configuration. Immutable after create. | | Display name | Yes | Human-readable label shown in the UI. | | Authorization endpoint | Yes | The upstream’s OAuth authorize URL, for example https://slack.com/oauth/v2/authorize. | | Token endpoint | Yes | The upstream’s OAuth token URL, for example https://slack.com/api/oauth.v2.access. | | Revocation endpoint | No | RFC 7009 token-revocation URL. When set, the gateway calls it on disconnect (best-effort). Not all providers support this. | 5. Pick a `Grant type`: - **Browser Consent**: The user approves access in their browser (OAuth 2.0 Authorization Code flow). The default for user-delegated OAuth. - **Token Exchange**: RFC 8693 server-to-server token exchange. Shown in the form but not yet functional (future release); the gateway exchanges the user’s identity-provider JWT for a provider-scoped token without browser interaction. 6. Pick a `Token-endpoint authentication method`: - **HTTP Basic**: `client_id:client_secret` sent as the Basic authentication header. Most common. - **POST body**: Credentials sent as form fields in the token-request body. - **None**: For public clients that rely on PKCE only. Pick this when the upstream OAuth app is registered as a public client and AI Gateway authenticates by proving possession of a PKCE code verifier rather than a stored client secret. Leave the client-secret reference unset. 7. Provide the `Client ID` and a `Client Secret Ref` (a secret-store key in `UPPER_SNAKE_CASE`, for example `SLACK_CLIENT_SECRET`). Use the **Existing** tab to pick a secret already in the store, or **New** to create one inline. Leave the reference empty only for public clients that authenticate with PKCE. 8. Define the `Scopes`. Include every scope any MCP server may need. 9. Optionally turn on `Require PKCE` to add PKCE (RFC 7636) to the consent flow. Leave `Enabled` on so the provider can start consent flows. 10. Click **Create Provider**. The provider appears in the **Outbound providers** list. ## [](#discover-a-provider-from-an-mcp-server-url)Discover a provider from an MCP server URL When you connect to a remote MCP server that authenticates users over OAuth, you can let AI Gateway detect the server’s OAuth configuration instead of entering endpoints and credentials by hand. If the remote server supports client metadata documents, AI Gateway identifies itself with an HTTPS URL and does not create a registration. If the server instead supports dynamic client registration, the gateway registers itself automatically. In either case, you have no endpoints or client credentials to paste. > 📝 **NOTE** > > This is the _outbound_ direction: AI Gateway identifies itself to a remote MCP server’s OAuth server. It is distinct from the inbound direction, where external MCP clients identify themselves to AI Gateway. To let clients such as Claude Code or ChatGPT connect to the gateway, see [Connect remote MCP clients to AI Gateway](https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients/). To discover a provider: 1. Open **Integrations setup** in the sidebar and select the **Outbound providers** tab. 2. Click **Create provider**, then select the **Discover from MCP server URL** card. 3. Enter the remote MCP server’s URL and start the discovery probe. 4. Review the result and finish the setup. The gateway takes one of these paths, depending on what the probe finds: | Result | What happens | | --- | --- | | Server supports client metadata documents | The gateway identifies itself with the client metadata document it publishes and pre-fills the provider. No client registration or shared client secret is required. Confirm the details to create the provider. | | Server supports dynamic client registration | The gateway registers itself as an OAuth client with the remote server’s authorization server and pre-fills the provider. Confirm the details to create the provider. | | A matching provider already exists | The gateway points you at the provider already registered for that server so you can attach it instead of creating a duplicate. | | Server has no OAuth, or OAuth without client metadata documents or dynamic client registration | The gateway falls back to the manual create form, pre-filling whatever it could detect. Complete the remaining fields as described in Register an OAuth provider in the UI. | A provider created this way is marked with a **Discovered** badge in the list and on its detail page, which distinguishes it from a provider you configured manually. Its detail page also shows a **Registration** card. See [View provider details](#view-provider-details). You can reach the same automatic setup from the other direction. When you create a self-managed MCP server that authenticates users over OAuth, Redpanda checks the server URL and sets up the OAuth client as part of creating the server, with no provider registered first. The provider it creates appears here with the same **Discovered** badge. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#register-from-the-cli)Register from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) to script provider registration: ```bash rpk ai oauth-provider create \ --name ramp \ --display-name "Ramp" \ --auth-endpoint "https://app.ramp.com/v1/authorize" \ --token-endpoint "https://api.ramp.com/developer/v1/token" \ --client-id "$RAMP_CLIENT_ID" \ --client-secret-ref RAMP_CLIENT_SECRET \ --scopes "transactions:read,cards:read,users:read" ``` | Flag | Notes | | --- | --- | | --name | Resource name. Lowercase letters, numbers, hyphens. Immutable. Required. | | --display-name | Human-readable display name shown in the UI. Required. | | --auth-endpoint | OAuth authorization endpoint URL. Required. | | --token-endpoint | OAuth token endpoint URL. Required. | | --client-id | Client ID from the upstream OAuth app. Required. | | --client-secret-ref | Secret-store reference (UPPER_SNAKE_CASE). | | --scopes | Comma-separated scope list. | | --grant-types | Grant types: browser-consent (default), token-exchange. Comma-separated. | | --token-auth-method | Token-endpoint authentication method: client-secret-basic (default), client-secret-post, none. | | --pkce | Require PKCE for authorization code grants. | | --revocation-endpoint | OAuth token revocation endpoint URL. | | --enabled | Whether the provider is enabled (default true). | ## [](#list-and-filter-providers-with-the-api)List and filter providers with the API To enumerate OAuth providers programmatically, call `OAuthProviderService.ListOAuthProviders`. Beyond the `name_contains` substring match, the request accepts an AIP-160 `filter` expression and an AIP-132 `order_by`, so you can scope and sort the list on the server (for example, to fetch only enabled providers, ordered by display name) instead of filtering the loaded page. The list endpoint uses the same OIDC client-credentials authentication as the rest of AI Gateway. Mint a service-account access token using the flow in [Authenticate with OIDC client credentials](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/#authenticate-with-oidc-client-credentials), then pass it as a bearer token. The service account needs `dataplane_aigateway_oauthprovider_list`. The request’s `filter` object carries two independent fields: | Field | Meaning | | --- | --- | | name_contains | Case-insensitive substring match on the provider name. | | filter | AIP-160 expression (maximum 1024 characters) that the server ANDs with name_contains and the tenant scope. Filterable columns: name, display_name, enabled, grant_types, created_at, and updated_at. Use it for multi-value OR semantics, for example enabled = true or grant_types = "OAUTH_GRANT_TYPE_BROWSER_CONSENT". The grant_types column accepts the OAuthGrantType enum values OAUTH_GRANT_TYPE_BROWSER_CONSENT and OAUTH_GRANT_TYPE_TOKEN_EXCHANGE; enabled accepts true or false. | Set `order_by` (maximum 256 characters) to a comma-separated list of fields, each optionally suffixed with \` desc\` (ascending by default), for example `display_name` or `created_at desc`. Orderable fields are the keyset columns `name`, `display_name`, and `created_at`. An unsupported field returns `InvalidArgument`. Leave `order_by` empty to use the server default (`created_at desc`). The order is encoded into the page token, so keep it stable across a paged walk. ```bash ACCESS_TOKEN="" # from the client_credentials flow DATAPLANE_BASE="https://aigw..clusters.rdpa.co" curl -s --request POST \ --url "${DATAPLANE_BASE}/redpanda.api.adp.v1alpha1.OAuthProviderService/ListOAuthProviders" \ --header "Authorization: Bearer ${ACCESS_TOKEN}" \ --header 'Content-Type: application/json' \ --data '{ "filter": { "filter": "enabled = true" }, "order_by": "display_name", "page_size": 50 }' | jq ``` The response carries the matching `oauth_providers`, a `next_page_token` for the next page, an estimated `total_size` (AIP-158) of all providers matching the filter (ignoring pagination), and the gateway-wide `callback_url` (the OAuth redirect URI to register with your identity provider). ## [](#attach-to-an-mcp-server)Attach to an MCP server To attach an OAuth provider to an MCP server, the principal needs `dataplane_aigateway_oauthprovider_attach` on the named provider plus the usual `mcpserver_create` / `mcpserver_update` permission. See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) for the full attach flow and [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) for the consent flow that runs on first call. ## [](#view-provider-details)View provider details Click a provider in the list to open its detail page. The header shows the display name, status badge, and resource name, with **Connect**, **Edit**, and **Delete** actions. A provider that was set up through discovery also shows a **Discovered** badge. The page groups the configuration into cards: - **Authorization Callback URL**: The gateway callback to register on the upstream OAuth app. - **OAuth Endpoints**: The authorization, token/refresh, and revocation endpoints. - **Client Configuration**: Client ID, Client Secret Ref, Auth Method, and whether PKCE is required. - **Registration** (discovered providers only): The dynamic-registration state. It shows the registration endpoint, the client-configuration URI (the RFC 7592 endpoint used to verify, refresh, or delete the registered client), and when the registration was last verified. The registration access token is a managed credential kept in the secret store, and its value is never shown. - **Scopes & Grant Types**: The default scopes and grant types. - **Metadata**: When the provider was created and last updated. ## [](#edit-and-rotate-credentials)Edit and rotate credentials You can change the provider’s configuration or rotate its client secret without re-creating the resource: - **Edit**: Open the provider’s detail page and click **Edit**. Endpoints, scopes, display name, and the client-secret reference can change. The `Name` is immutable. - **Rotate credentials**: Update the secret content in the secret store under the same name (for example, `SLACK_CLIENT_SECRET`). The provider’s reference is unchanged. Existing tokens in the vault stay valid; the new client secret is used the next time AI Gateway exchanges credentials. - **Disable temporarily**: Deleting the provider invalidates every MCP server’s connections that reference it, so disable the dependent MCP servers first if you want to pause traffic without losing user consent. ## [](#delete-a-provider)Delete a provider Deleting an OAuth provider: - Removes the provider record. - Causes every MCP server that referenced it to fail authentication on the next call (the `provider_name` reference no longer resolves). - Leaves user-stored tokens in the vault until garbage-collected, but they’re unusable without the provider definition. Plan the deletion: disable or reconfigure dependent MCP servers first, communicate the cutover to users so they can re-consent against a replacement provider, then delete. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | PermissionDenied when creating an MCP server with this provider attached | Nothing grants the principal dataplane_aigateway_oauthprovider_attach on this provider. Ask an admin for a policy granting Action::"OAuthProvider.attach", scoped to this provider. | | Consent flow fails with redirect_uri_mismatch | The OAuth app’s registered redirect URI doesn’t match the gateway’s callback. Update the upstream OAuth app to include https://aigw..clusters.rdpa.co/oauth/v1/callback. | | invalid_client during token exchange | Client ID or client secret is wrong, or the Token-endpoint authentication method doesn’t match what the upstream expects. Check the upstream OAuth app’s settings. | | invalid_scope during consent | A scope in Supported scopes isn’t valid for the upstream. Check the upstream’s scope reference and remove or rename the offending scope. | ## [](#next-steps)Next steps - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Plug in an App, Database, or Tool](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) --- # Page 40: Integrate with Redpanda Pipelines **URL**: https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns.md --- # Integrate with Redpanda Pipelines > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Integrate with Redpanda Pipelines latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: pipeline-integration-patterns page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: pipeline-integration-patterns.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/pipeline-integration-patterns.adoc description: Build Redpanda Connect pipelines that invoke agents for event-driven processing and streaming enrichment. page-topic-type: best-practices personas: agent_builder, platform_engineer learning-objective-1: Identify when pipelines should call agents for stream processing learning-objective-2: Design event-driven agent invocation using the a2a_message processor learning-objective-3: Implement streaming enrichment with AI-generated fields page-git-created-date: "2026-05-28" page-git-modified-date: "2026-06-18" --- Build Redpanda Connect pipelines that invoke agents for automated, event-driven processing. Pipelines use the `a2a_message` processor to call agents for each event in a stream when you need AI reasoning, classification, or enrichment at scale. After reading this page, you will be able to: - Identify when pipelines should call agents for stream processing - Design event-driven agent invocation using the `a2a_message` processor - Implement streaming enrichment with AI-generated fields This page focuses on pipelines calling agents (pipeline-initiated integration). For agents invoking MCP tools, see [Agent needs capabilities](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/#agent-needs-capabilities). For external applications calling agents, see [External system calls agent](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/#external-system-calls-agent). ## [](#how-pipelines-invoke-agents)How pipelines invoke agents Pipelines use the [`a2a_message`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/a2a_message/) processor to invoke agents for each event in a stream. The processor uses the [A2A protocol](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/) to discover and communicate with agents. When the `a2a_message` processor receives an event, it sends the event data to the specified agent along with any prompt you provide. The agent processes the event using its reasoning capabilities and returns a response. The processor then adds the agent’s response to the event for further processing or output. The pipeline determines when to invoke agents based on events, not agent reasoning. ## [](#when-to-use-this-pattern)When to use this pattern Use the `a2a_message` processor when pipelines need AI reasoning for every event in a stream. The `a2a_message` processor is appropriate when: - **Every event needs AI analysis:** Each message requires reasoning, classification, or decision-making. - **You need streaming enrichment:** Add AI-generated fields to events at scale. - **Processing is fully automated:** No human in the loop, event-driven workflows. - **Batch latency is acceptable:** Agent reasoning time is tolerable for your use case. - **You’re handling high-volume streams:** Processing thousands or millions of events. ## [](#use-cases)Use cases Use the `a2a_message` processor in pipelines for these common patterns. ### [](#event-driven-agent-invocation)Event-driven agent invocation Use this pattern for real-time fraud detection on every transaction. Invoke agents automatically for each event: ```yaml # Event-driven agent invocation pipeline # Invokes an agent for each event in a stream input: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topics: [transactions] consumer_group: fraud-detector tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" pipeline: processors: - a2a_message: agent_card_url: "${AGENT_CARD_URL}" prompt: "Analyze this transaction: ${!content()}" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: fraud-alerts tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" ``` Replace `AGENT_CARD_URL` with your actual agent card URL. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). ### [](#streaming-data-enrichment)Streaming data enrichment Use this pattern to add sentiment scores to every customer review in real time. Add AI-generated metadata to events: ```yaml processors: - branch: request_map: 'root = this.text' processors: - a2a_message: agent_card_url: "${AGENT_CARD_URL}" result_map: 'root.sentiment = content()' ``` Replace `AGENT_CARD_URL` with your actual agent card URL. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). ### [](#asynchronous-workflows)Asynchronous workflows Use this pattern for nightly batch summarization of reports where latency is acceptable. Process events in the background: ```yaml input: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topics: [daily-reports] consumer_group: report-analyzer tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" pipeline: processors: - a2a_message: agent_card_url: "${AGENT_CARD_URL}" prompt: "Summarize this report: ${!content()}" ``` Replace `AGENT_CARD_URL` with your actual agent card URL. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). ### [](#multi-agent-pipeline-orchestration)Multi-agent pipeline orchestration Use this pattern to translate feedback, analyze sentiment, and route it to the appropriate team. Chain multiple agents in sequence: ```yaml processors: - a2a_message: agent_card_url: "${TRANSLATOR_AGENT_URL}" - a2a_message: agent_card_url: "${SENTIMENT_AGENT_URL}" - a2a_message: agent_card_url: "${ROUTER_AGENT_URL}" ``` Replace the agent URL variables with your actual agent card URLs. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). ### [](#agent-as-transformation-node)Agent as transformation node Use this pattern to convert natural language queries to SQL for downstream processing. Use agent reasoning for complex transformations: ```yaml processors: - a2a_message: agent_card_url: "${AGENT_CARD_URL}" prompt: "Convert to SQL: ${!this.natural_language_query}" ``` Replace `AGENT_CARD_URL` with your actual agent card URL. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). ## [](#when-not-to-use-this-pattern)When not to use this pattern Do not use the `a2a_message` processor when: - Users need to interact with agents interactively. - The transformation is simple and does not require AI reasoning. - Agents need to dynamically decide what data to fetch based on context. For a detailed comparison between pipeline-initiated and agent-initiated integration patterns, see [Pattern comparison](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/#pattern-comparison). ## [](#example-real-time-fraud-detection)Example: Real-time fraud detection This example shows a complete pipeline that analyzes every transaction with an agent. ### [](#pipeline-configuration)Pipeline configuration ```yaml # Fraud detection pipeline with score-based routing # Analyzes every transaction and routes to different topics based on fraud score input: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topics: [transactions] consumer_group: fraud-detector tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" pipeline: processors: - branch: request_map: | root.transaction_id = this.id root.amount = this.amount root.merchant = this.merchant root.user_id = this.user_id processors: - a2a_message: agent_card_url: "${AGENT_CARD_URL}" prompt: | Analyze this transaction for fraud: Amount: ${! json("amount") } Merchant: ${! json("merchant") } User: ${! json("user_id") } Return JSON: { "fraud_score": 0-100, "reason": "explanation", "recommend_block": true/false } result_map: | root = this root.fraud_analysis = content().parse_json().catch({}) - mapping: | root = this meta fraud_score = this.fraud_analysis.fraud_score output: switch: cases: - check: 'meta("fraud_score") >= 80' output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: fraud-alerts-high tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" - check: 'meta("fraud_score") >= 50' output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: fraud-alerts-medium tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" - output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: transactions-cleared tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "${REDPANDA_USERNAME}" password: "${REDPANDA_PASSWORD}" ``` Replace `AGENT_CARD_URL` with your agent card URL. See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location). This pipeline: - Consumes every transaction from the `transactions` topic. - Sends each transaction to the fraud detection agent using `a2a_message`. - Routes transactions to different topics based on fraud score. - Runs continuously, analyzing every transaction in real time. ## [](#next-steps)Next steps - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - [Choose an Integration Pattern](https://docs.redpanda.com/agentic-data-plane/connect/integration-overview/) - [Connect Agents with A2A](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/) --- # Page 41: Register a Self-Managed MCP Server **URL**: https://docs.redpanda.com/agentic-data-plane/connect/register-remote.md --- # Register a Self-Managed MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Register a Self-Managed MCP Server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: register-remote page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: register-remote.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/register-remote.adoc description: Front your own MCP server with a managed Redpanda URL. Pick a transport, configure authentication, and let agents discover the server's tools through Redpanda. page-topic-type: how-to personas: platform_engineer, agent_builder learning-objective-1: Register a self-managed MCP server in Agentic Data Plane learning-objective-2: Pick the right transport (SSE vs. Streamable HTTP) and authentication mode learning-objective-3: Confirm tool discovery completed and the server is reachable through its proxy URL page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- Register your existing MCP server with Redpanda to add authentication, observability, and agent aggregation without changing your server’s code. This guide covers the self-managed path from [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) in depth. Choose this when you already run a server and want Redpanda to proxy it. After completing this guide, you will be able to: - Register a self-managed MCP server in Agentic Data Plane - Pick the right transport (SSE vs. Streamable HTTP) and authentication mode - Confirm tool discovery completed and the server is reachable through its proxy URL ## [](#when-to-use-this)When to use this Choose self-managed registration when: - The MCP server is already deployed in your environment and you don’t want to migrate it to a managed type. - You need custom tool logic that no managed type provides. - You want a unified MCP URL (and Inspector / observability / agent access) across servers without standing up your own gateway. If you don’t already run a server, prefer a managed type. See [Managed catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) for the catalog. ## [](#prerequisites)Prerequisites - An MCP server reachable from the Agentic Data Plane. - The endpoint URL. `http://` is allowed for everything except user-delegated OAuth, which requires `https://` (proto rule `remote_mcp.user_oauth_requires_https`). - Knowledge of which transport the server speaks (SSE or Streamable HTTP). If you don’t know, see [Transport choice](#test-transport). - If using static-key or service-account-OAuth: secrets pre-created in the Redpanda Agentic Data Plane secret store, `UPPER_SNAKE_CASE` (proto regex `^[A-Z][A-Z0-9_]*$`). - If using user-delegated OAuth with automatic setup: nothing, when the server supports dynamic client registration. If it doesn’t, register an OAuth app with the upstream vendor and have its client ID ready, plus a client-secret reference in the secret store for a confidential app. - If attaching an OAuth provider your organization already manages instead: the provider registered first. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#create-the-server)Create the server 1. Open **MCP Servers** in the sidebar and click **Create Server**. 2. In the marketplace picker, choose **Remote (Proxied)**. 3. Fill in the identity fields (`name`, `description`, `enabled`); same constraints as in [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/). 4. Provide the `URL` and `Transport`. 5. Configure authentication (see [Authentication](#authentication)). 6. Click **Create**. ## [](#test-transport)Transport choice Two transports are available on the proto enum `MCPTransport`: - **SSE**: Server-sent events. The traditional MCP transport. - **Streamable HTTP**: Newer bidirectional protocol. Pick whichever your server actually speaks. To probe a server quickly: ```bash # SSE handshake: expect a stream of `event:` lines on success. curl -N -H "Accept: text/event-stream" https://your-server.example.com/mcp/sse # Streamable HTTP: expect a JSON-RPC response on success. curl -X POST -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ https://your-server.example.com/mcp ``` ## [](#authentication)Authentication The authentication modes from [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/#configure-authentication) all apply. Several patterns are particularly common for self-managed servers: | Pattern | Use when | | --- | --- | | Static key with a custom header | First-party servers that authenticate with a non-Authorization header. Provide a header_name (for example, X-Api-Key) and a key_ref pointing at a UPPER_SNAKE_CASE secret. | | Service-account OAuth | The upstream system supports OAuth client credentials and you want one shared identity for all callers. Provide client_id, client_secret_ref, token_url, and any required scopes. | | Token passthrough | The upstream server already validates client tokens; Redpanda just forwards the caller’s Authorization header. | For user-delegated OAuth, the URL must be `https://`. Redpanda can set up OAuth automatically for the server, or you can attach an OAuth provider you already registered. See [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/). ## [](#tool-discovery)Tool discovery After create, Redpanda runs a live `tools/list` (the `ListMCPServerTools` RPC) against the server. The result is cached on the `MCPServer.tools` output-only field and shown on the detail page’s **Overview** tab. The **Inspector** tab (see [Test a server’s tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/)) exercises individual [tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool). If the tools list is empty or stale, open the **Inspector** tab and click **Refresh** to re-run discovery against the server. ## [](#connection-errors-and-retry)Connection errors and retry | Error | What it means | | --- | --- | | OAuthConnectionRequired | The user-delegated auth path needs a stored token vault entry for the calling user. Redpanda surfaces an authorize_url so the user can complete the consent flow. See User-delegated OAuth. | | OAuthTokenExpired | The user’s stored token has expired and refresh failed. Surface the new authorize URL and have the user re-consent. | | Plain connection error / 502 / 504 | Redpanda couldn’t reach your server. Look at the Connection tab on the detail page for the upstream error and check the connection-status banner. | ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | "Transport mismatch" or empty stream | Your server speaks the other transport. Re-probe with curl per Transport choice and update the Transport field on the server’s edit page. | | TLS errors when registering an https:// URL | Confirm the server’s certificate chains to a public CA (or the CA Redpanda’s egress trusts). Self-signed certs aren’t supported. | | 401 Unauthorized from the upstream | Authentication is misconfigured. For static-key or service-account-OAuth, verify the secret content and header_name. For token passthrough, confirm the caller is sending an Authorization header. | | Tool schema parse errors | The server returned a tools/list response with an invalid JSON schema for one or more tool inputs. Fix the upstream tool’s schema and refresh. | ## [](#next-steps)Next steps - [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) - [Code Mode](https://docs.redpanda.com/agentic-data-plane/gateway/code-mode/) --- # Page 42: Connect Remote MCP Clients **URL**: https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients.md --- # Connect Remote MCP Clients > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect Remote MCP Clients latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: remote-mcp-clients page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: remote-mcp-clients.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/remote-mcp-clients.adoc description: Connect external MCP clients to MCP servers in AI Gateway using client metadata documents, dynamic client registration, or manual registration. page-topic-type: how-to personas: agent_builder, platform_engineer, security_compliance_lead learning-objective-1: Choose a client identification method for an external chat app learning-objective-2: Wire a custom connector in Claude Desktop (or another chat client) to your MCP server learning-objective-3: Walk a user through the OAuth flow that runs end-to-end page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- External MCP clients (Claude Desktop, ChatGPT desktop, Gemini Apps, Cursor) connect to MCP servers hosted in AI Gateway over OAuth. A client can identify itself with a client metadata document, register itself dynamically, or use a client that an administrator registered manually. End-users get the MCP tools inside their preferred chat app, with Redpanda mediating both client-app authentication and upstream-system authentication. After completing this guide, you will be able to: - Choose a client identification method for an external chat app - Wire a custom connector in Claude Desktop (or another chat client) to your MCP server - Walk a user through the OAuth flow that runs end-to-end ![OAuth client onboarding options. Client ID metadata documents let a public client use a metadata URL without a stored registration. Dynamic client registration lets a client create and maintain its own registration in AI Gateway. Manual registration lets an administrator enter redirect URIs and distribute credentials for clients that support neither automated option.](https://docs.redpanda.com/agentic-data-plane/connect/_images/oauth-client-onboarding-options.svg) Figure 1. Choose the lightest client onboarding method the MCP client supports > ❗ **IMPORTANT** > > OAuth clients and OAuth providers govern _opposite directions_ of authentication and are separate resources. > > - An OAuth client (this page) governs _inbound_ authentication: an external chat app (Claude Desktop, ChatGPT, Copilot Studio, Cursor) authenticating to AI Gateway so its users can invoke MCP tools. > > - An OAuth provider governs _outbound_ authentication: AI Gateway authenticating to an upstream system (GitHub, Slack, Salesforce) on a user’s behalf when an MCP server uses user-delegated OAuth. > > > See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) for the outbound authentication side. ## [](#when-to-use-this)When to use this Use a remote MCP client connection when: - You want users to invoke MCP tools from inside Claude Desktop, ChatGPT desktop, Gemini Apps, or Cursor without writing custom integration code. - You already have, or are about to create, an MCP server (managed or self-managed) in AI Gateway. - End-users have accounts with the chat client and the upstream system you’re integrating with. Use a different approach when: - You need programmatic, server-side tool invocation. See [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) for SDK-based access. - You need an in-house chat UI. Build against the AI Gateway’s MCP endpoints directly with the SDK of your choice. ## [](#architecture-how-the-resources-work-together)Architecture: How the resources work together Wiring a remote chat client to an MCP server uses three resources in AI Gateway: | Resource | Role | | --- | --- | | MCP server | The tool surface itself. Managed (Redpanda hosts it) or self-managed (you host it). See Create an MCP Server. | | OAuth Provider | Defines how AI Gateway authenticates against the upstream system on behalf of users (for example, GitHub) when the MCP server uses user-delegated OAuth. See Configure an OAuth Provider. Optional: only needed if the MCP server requires per-user upstream identity. | | OAuth Client | Defines how an external chat app (Claude Desktop, ChatGPT, Gemini, Cursor) authenticates against AI Gateway on behalf of users. The client can use an HTTPS URL as its client_id, register dynamically, or use credentials from a manual registration. This is what makes the chat-client integration possible. | Putting it together with a GitHub example: - The **MCP server** is a managed GitHub MCP, configured to use user-delegated OAuth. - The **OAuth Provider** points at GitHub’s OAuth endpoints; AI Gateway uses it to act as each user against GitHub. - The **OAuth Client** is registered for Claude Desktop; Claude Desktop uses it to act as each user against AI Gateway. When a user invokes a tool, AI Gateway runs both authentication handshakes: Claude to AI Gateway through the OAuth Client, then AI Gateway to GitHub through the OAuth Provider. ![Two-stage OAuth flow. An external MCP client authenticates inbound to AI Gateway through an OAuth client. AI Gateway invokes the MCP server, then authenticates outbound to the upstream system through an OAuth provider when user-delegated OAuth is configured.](https://docs.redpanda.com/agentic-data-plane/connect/_images/remote-mcp-client-authentication.svg) Figure 2. A remote MCP tool call crosses separate inbound and outbound OAuth boundaries ## [](#prerequisites)Prerequisites Before you wire up the chat-client connector, make sure you have: - An MCP server already created in AI Gateway. See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/). - The server’s `Server URL`. Copy it from the server’s **Connection** tab. - For user-delegated MCP servers: An OAuth Provider configured for the upstream system. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). - End-users have accounts with the chat client (Claude, ChatGPT, Gemini, Cursor) and the upstream system the MCP server connects to. ## [](#client-metadata-documents)Let clients use metadata documents Client ID metadata documents let a public OAuth client use an HTTPS URL as its `client_id`. The client hosts a JSON metadata document at that URL instead of creating a saved registration on AI Gateway. The document identifies the client and lists its redirect URIs. When the client starts an authorization request, AI Gateway checks that the document names its own URL and includes the requested redirect URI. Clients that use a metadata document run the authorization-code flow with PKCE and do not use a shared client secret. Use metadata documents when the external MCP client supports them and you want clients to connect without an administrator creating or maintaining a registration. A client that uses a metadata document does not create an entry in the **Inbound clients** list. ### [](#enable-client-metadata-documents)Enable client metadata documents 1. Open **Integrations setup** in the sidebar and select the **Inbound clients** tab. 2. Select **Self-registration**. 3. Turn on **Accept client metadata documents (CIMD)**. 4. Under Trusted document domains, choose one of these options: - **Any domain**: Accept a valid metadata document from any HTTPS domain. - **Only these domains**: Accept documents only from the domains you list. A listed parent domain also covers its subdomains. 5. Under Allowed resources, add the exact MCP URLs these clients may access, or allow any MCP server on the gateway. 6. Click **Save settings**. > 📝 **NOTE** > > If the settings report that client metadata documents are not active on the gateway, clients cannot use a URL as their client ID yet. Contact the gateway operator before testing the connection. > ❗ **IMPORTANT** > > The domain policy controls which clients may identify themselves to the gateway. The allowed-resources policy separately controls which MCP servers those clients may request tokens for. Restrict both lists when only approved clients should reach specific MCP servers. If a client does not support metadata documents, use dynamic client registration or create the client manually. ## [](#register-an-oauth-client-in-ai-gateway)Register an OAuth Client in AI Gateway Register a client by hand when it supports neither client metadata documents nor dynamic client registration, when it needs a confidential `client_secret`, or when you want to pre-provision it before the first connection. For supported public clients, you can skip this manual flow. See [Let clients use metadata documents](#client-metadata-documents) or [Let clients self-register](#self-register-with-dcr). Create an OAuth Client to give the chat app the credentials it needs to authenticate against AI Gateway: 1. Open **Integrations setup** in the sidebar and select the **Inbound clients** tab. The tab lists every external tool registered to request access tokens from this gateway, with its `Name`, `Grant Types`, `Status`, and `Scopes`. (The older `/oauth-clients` path still works but redirects here.) 2. Click **Add external tool**. 3. Pick a well-known client to prefill its redirect URIs and recommended scopes, or choose **Custom Client** to register one from scratch. The catalog includes **Claude.ai Custom Connectors**, **Claude Desktop**, **ChatGPT**, **Cursor**, **Windsurf**, **Visual Studio Code** (which also covers GitHub Copilot in VS Code), **Zed**, **Gemini CLI**, and **mcp-remote bridge** (a stdio-to-HTTP bridge for clients without native remote OAuth). For any client not in the catalog (for example, the Gemini consumer apps or Microsoft Copilot Studio), use **Custom Client** and paste the redirect URIs the tool publishes. 4. Fill in the form: | Field | Notes | | --- | --- | | Display name | Human-readable label shown in the UI and on the consent screen, for example Claude.ai Custom Connectors. | | Name | Machine identifier, and the OAuth client_id. Lowercase letters, numbers, and hyphens only. Immutable after creation. | | Logo URI | Optional HTTPS URL for a logo shown on the consent screen. | | Redirect URIs | The exact URIs the gateway redirects to after a user approves. Each chat client publishes its own; a well-known client prefills them. Multiple URIs are allowed. Claude.ai, for example, uses two: https://claude.ai/api/mcp/auth_callback and https://claude.ai/api/organizations/custom-connectors/oauth/callback; Claude Desktop uses http://127.0.0.1:54545/callback. | | Allowed MCP Resources | Which MCP servers this client may request tokens for. Defaults to * (any MCP server hosted on this gateway). Click Restrict to limit the client to specific MCP resource URLs. | | Grant Types | Authorization Code (RFC 6749 §4.1) and Refresh Token (RFC 6749 §6, rotating single-use). This is the standard combination for browser-based chat clients with long-lived tokens. | | Token Endpoint Auth Method | How the client authenticates to the token endpoint: Client Secret (Basic) (default), Client Secret (POST), or None (PKCE only) for public clients that can’t store a secret. | | Require PKCE | Strongly recommended and enforced by default (RFC 7636). Adds an extra layer of security to the consent flow. | | Enabled | Whether the client can start new consent flows. Disabled clients reject new flows. | 5. Click **Register Client**. On submit, AI Gateway mints the `client_id`. Confidential clients (those using a Client Secret authentication method) also receive a `client_secret`, **shown only once**. Copy it immediately. The OAuth Server Endpoints panel shows the `Authorization Endpoint` and `Token/Refresh Endpoint` (under `/oauth/idp/`) and the `mcp` scope; paste these into the external app’s OAuth configuration. ## [](#self-register-with-dcr)Let clients self-register For clients that do not support metadata documents, you can let spec-conformant MCP clients register themselves. AI Gateway supports the OAuth 2.0 Dynamic Client Registration Protocol (RFC 7591), but dynamic registration is off for each organization by default. After you enable it, the gateway advertises a `registration_endpoint` in its authorization-server metadata at `/.well-known/oauth-authorization-server`. Clients that read this metadata (Claude Code, Cursor, ChatGPT, and other spec-conformant MCP clients) register on their first connection attempt, run the authorization-code flow with PKCE, and obtain tokens without an admin provisioning a `client_id` or `client_secret` first. Enable self-registration before conformant clients can register. ### [](#enable-self-registration)Enable self-registration Run the `rpk ai oauth-client dcr update` command to enable self-registration and choose how callers are admitted: ```bash rpk ai oauth-client dcr update --enabled --admission-mode open ``` > 💡 **TIP** > > You can also review and change self-registration from the UI: on the **Integrations setup** page, open the **Inbound clients** tab and select **Self-registration** to open its settings. Inspect the current policy at any time: ```bash rpk ai oauth-client dcr get ``` The command reports whether self-registration is enabled, the admission mode, and the resource and rate caps, for example `ENABLED: true`, `ADMISSION_MODE: OPEN`, `ALLOWED_RESOURCES: ["*"]`. ### [](#admission-modes)Admission modes The admission mode controls who may register after you enable self-registration: | Mode | Behavior | | --- | --- | | open | Any spec-conformant client registers without credentials. Use this for clients that register anonymously, such as Claude Code. The registration rate limit and client cap still apply. | | initial-access-token | Callers must present an admin-minted, one-shot bearer token (an Initial Access Token) on the registration request. Use this to keep self-registration open only to holders of a token you hand out. | The `software-statement` admission mode is reserved and not yet supported. ### [](#restrict-and-rate-limit-self-registration)Restrict and rate-limit self-registration Pass caps to the same `rpk ai oauth-client dcr update` command to bound what self-registered clients can do: | Flag | Purpose | | --- | --- | | --allowed-resource | MCP server URL that every self-registered client may request tokens for. Repeatable. * allows any MCP server on the gateway. | | --client-cap | Maximum number of concurrent self-registered clients. | | --rate-per-hour | Maximum registrations accepted per hour. | | --inactive-ttl-days | Days of inactivity before AI Gateway removes a self-registered client. 0 keeps clients indefinitely. | AI Gateway disables and then removes inactive self-registered clients according to this policy, so abandoned registrations don’t accumulate. ### [](#mint-initial-access-tokens)Mint initial access tokens In `initial-access-token` mode, mint a one-shot token and give it to the client operator, who presents it on the registration request: ```bash rpk ai oauth-client dcr iat mint --label "Claude handoff" --ttl 24h ``` The token’s plaintext appears only once, on mint, and is consumed on the first successful registration. List and revoke tokens with: ```bash rpk ai oauth-client dcr iat list rpk ai oauth-client dcr iat revoke ``` Replace `` with the ID from the IAT list. ### [](#review-self-registered-clients)Review self-registered clients Self-registered clients appear alongside manually registered ones in the `rpk ai oauth-client list` output and on the **Inbound clients** tab, marked with a DCR badge and a name of the form `dcr-`. They authenticate with PKCE and no client secret. Audit them like any other client, and delete one to revoke its tokens: ```bash rpk ai oauth-client list rpk ai oauth-client delete ``` Replace `` with the `dcr-` name from the list. ## [](#wire-up-claude)Wire up Claude Anthropic supports custom MCP connectors in Claude.ai (web), Claude Desktop, and the Claude organization-settings UI. The setup flow is the same in each: 1. Open **Settings > Connectors** (or **Customize > Connectors** in newer builds; Anthropic surfaces a _Connectors have moved to Customize_ notice during the migration). 2. Click **Add custom connector**. 3. Fill in the connector details: | Field | Value | | --- | --- | | Name | Anything that helps the user identify the connector (for example, Redpanda GitHub). Surfaces in Claude’s tool list. | | Remote MCP server URL | The server’s Server URL from AI Gateway. Format: https://aigw..clusters.rdpa.co/mcp/v1/. | | OAuth client ID (optional, under Advanced settings) | The Client ID from the AI Gateway OAuth Client. Required for any MCP server that requires authentication. Leave blank only for public MCP servers that don’t require authentication. | | OAuth client secret (under Advanced settings) | The Client Secret from the AI Gateway OAuth Client. Required whenever Client ID is set. | 4. Click **Add**. The connector appears in the Connectors list with a `CUSTOM` badge. 5. Click **Connect** on the new connector row. Claude opens a browser tab pointed at AI Gateway’s authorization endpoint. Sign in with your AI Gateway identity provider. After approval, the connector becomes invokable in any conversation. > 📝 **NOTE** > > Anthropic’s modal warns that connectors are user-trust-based; Anthropic doesn’t control which tools developers expose. If you’re publishing a connector for end-users, document the upstream system and scopes clearly so users know what they’re authorizing. ## [](#wire-up-other-chat-clients)Wire up other chat clients The flow mirrors Claude Desktop. **ChatGPT**, **Cursor**, **Windsurf**, **Visual Studio Code**, **Zed**, **Gemini CLI**, and **mcp-remote bridge** have catalog presets in the **Add external tool** flow that prefill their redirect URIs; the others use a **Custom Client**. The exact menu paths and field labels differ by client: - **ChatGPT desktop**: Recent builds support remote MCP custom connectors. Confirm the latest menu path; OpenAI iterates on this surface. - **Cursor**: Supports remote MCP servers in recent builds. - **Windsurf**: Supports remote MCP servers in recent builds. - **Visual Studio Code**: Built-in MCP support, shared with GitHub Copilot in VS Code. Has a catalog preset. - **Zed**: Supports remote MCP server OAuth. Has a catalog preset. - **Gemini CLI**: Supports MCP server OAuth. Has a catalog preset. This is the command-line tool, distinct from the consumer Gemini apps, which use a **Custom Client**. - **mcp-remote bridge**: For stdio-only MCP clients bridged through `npx mcp-remote `. Has a catalog preset. - **Gemini apps**: Recent builds support remote MCP custom connectors. Register through **Custom Client**. - **Microsoft Copilot Studio**: Recent builds support remote MCP custom connectors registered against an external OAuth 2.0 authorization server. Register through **Custom Client**, and add Copilot Studio’s published redirect URIs to the AI Gateway OAuth Client before connecting. The required inputs are the same as Claude Desktop: connector name, MCP URL, Client ID, Client Secret. The chat client’s redirect URIs must be registered on the AI Gateway OAuth Client. ## [](#the-oauth-flow)The OAuth flow When a user calls a tool that needs upstream access, two OAuth handshakes run end-to-end. Most users only see the second one (and only on the very first tool call). ### [](#chat-client-connects-to-ai-gateway)Chat client connects to AI Gateway This handshake runs **once per user** when the connector is first added. 1. The user clicks **Connect** in the chat client. 2. The chat client opens a browser tab at the AI Gateway authorization endpoint, parameterized with the OAuth Client’s `client_id` and one of the registered redirect URIs. 3. AI Gateway authenticates the user against the configured IdP (Auth0 today, Zitadel later) and presents an Authorize access consent screen. The screen shows: - The OAuth Client’s name (for example, _Claude (GitHub Read demo) wants to access your data_). - The **Resource** being authorized: The MCP server name and URL. - The **Requested permissions**: The gateway’s internal scopes for this handshake (`mcp` and `offline_access`). These are **not** the upstream system’s scopes; the upstream’s scopes appear during the next handshake. - A footer reminding the user that they can revoke this access at any time in their Redpanda dashboard. 4. The user clicks **Allow**. AI Gateway redirects the chat client back to the redirect URI with an authorization code. 5. The chat client exchanges the code for an access token and a refresh token, and stores them locally in its credential store. 6. Subsequent calls to AI Gateway send the access token in `Authorization: Bearer …​`. The chat client refreshes the token automatically when it expires. ### [](#ai-gateway-connects-to-the-upstream-system)AI Gateway connects to the upstream system (Only for user-delegated MCP servers.) This handshake runs **once per user, per upstream**. For an MCP server using user-delegated OAuth (GitHub, Slack, Atlassian, Workday, and so on): 1. The user invokes a tool that requires upstream authentication. 2. AI Gateway has no stored upstream token for this user yet. The MCP protocol returns a `FAILED_PRECONDITION` response with an `OAuthConnectionRequired` error detail. The detail carries an `authorize_url` pointing at AI Gateway’s OAuth bridge for the configured upstream provider, for example: `https://aigw..clusters.rdpa.co/oauth/v1/authorize?provider_name=github&scopes=read:user,repo`. 3. The chat client renders the link in its response to the user. Inside Claude this appears as a hyperlinked URL with prose telling the user to authorize the upstream connection (for example, _Authorize the GitHub connection first_) before retrying. 4. The user clicks the link. AI Gateway redirects them to the **upstream system’s own OAuth consent page** (for example, GitHub’s standard authorization UI) listing the requested repositories and scopes. 5. The user clicks **Authorize** on the upstream’s consent page. The upstream redirects back to AI Gateway with an authorization code. AI Gateway exchanges the code for a token and stores it in its token vault under the user’s identity. 6. The user tells the chat client they’ve connected. The chat client retries the original tool call, which now succeeds. Subsequent calls reuse the stored upstream token automatically. After both steps complete, the user can invoke any tool on the MCP server transparently. They re-consent only if scopes change or the refresh tokens expire. > 📝 **NOTE** > > Claude (and other chat clients) layer their own **per-tool consent prompts** on top of the OAuth flow described here. The first time a connector tries to invoke a specific tool, Claude shows a prompt of the form _Claude wants to use {tool\_name} from {connector\_name}_ with **Always allow** / **Deny** buttons. This is the chat client’s own user-trust UX, not an additional AI Gateway authentication step. After a user picks **Always allow** for a tool, Claude won’t prompt again for that tool from that connector. > 📝 **NOTE** > > If the MCP server uses a service-account authentication mode instead of user-delegated OAuth, only Step 1 runs. AI Gateway calls the upstream with one shared identity and the user never sees the upstream consent flow. ## [](#manage-and-rotate)Manage and rotate Maintain registered OAuth Clients without re-creating them: - **List registered clients**: Open **Integrations setup** in the sidebar and select the **Inbound clients** tab. Each row shows the Name, Grant Types, Status, and Scopes. - **Edit a client**: Change the display name, logo URI, redirect URIs, allowed MCP resources, grant types, or token endpoint authentication method. The `client_id` is immutable; delete and recreate the client if you need a new one. - **Reissue the secret**: The Client Secret is shown only once at creation and can’t be regenerated in place. To issue a new secret, delete and recreate the client, then update every chat client that used it. - **Revoke all tokens**: Invalidate every refresh token AI Gateway has issued under this client without deleting the client itself (see [Revoke all tokens](#revoke-all-tokens)). - **Delete a client**: Invalidates every active token issued under it. Token revocation runs automatically as part of delete, so users connected through this client are signed out immediately. Every chat-client connector that depends on this OAuth Client breaks until reconfigured against a replacement. ### [](#revoke-all-tokens)Revoke all tokens Revoking all tokens invalidates every refresh token AI Gateway has issued under an OAuth Client. Use it when you need to force every connected user to sign in again, for example, after a leaked Client Secret, an offboarding event, or a policy change that warrants re-consent. Revoke from the UI: 1. Open **Integrations setup** in the sidebar, select the **Inbound clients** tab, and click the client whose tokens you want to revoke. 2. On the client detail page, click **Revoke all tokens**. 3. Confirm the action. Revoke from the CLI: ```bash rpk ai oauth-client revoke-tokens ``` The command prints the number of tokens it revoked, for example `Revoked 12 tokens for OAuth client claude-desktop`. The action is idempotent: a second invocation returns `Revoked 0 tokens for OAuth client claude-desktop` and exits 0. > 📝 **NOTE** > > Already-issued **short-lived access tokens** may continue to work until their natural expiry, typically a few minutes. The revocation invalidates **refresh tokens**, so after the access token expires, the chat client cannot mint a new one and the user is forced to re-authenticate. When AI Gateway auto-revokes tokens (on Client deletion or on lifecycle events that imply a hard cut), the same flow runs server-side without the explicit CLI or button click. Deletion revokes tokens at both levels. Deleting an OAuth client revokes that client’s refresh tokens. Deleting an upstream OAuth provider revokes every active token issued for that provider across all users before it removes the provider config. ## [](#troubleshooting)Troubleshooting Common symptoms and fixes: | Symptom | What to check | | --- | --- | | Couldn’t connect to MCP server or connector setup fails immediately | The MCP URL is wrong, or the Client ID + Client Secret don’t match an OAuth Client. Confirm the Server URL on the MCP server’s Connection tab and the credentials on the OAuth Client’s detail page. | | redirect_uri_mismatch during the connect flow | The chat client’s callback URL isn’t registered on the OAuth Client. Add the URL the chat client publishes (Claude Desktop has two; check Claude’s docs for the current set). | | Connector authorized but no tools appear | The MCP server has zero tools, or tools/list failed at connection time. Open the server in the Inspector to confirm tools are discovered. See Test a server’s tools. | | Tool call returns an authorize link to the user | First call from a user with no stored upstream token. The user follows the link, completes upstream consent, and the call retries automatically (Step 2 of the flow above). | | scope_upgrade_required from a tool call | The MCP server’s required_scopes was extended after the user consented at the upstream. The user re-consents at the upstream with the higher scope. | | 401 Unauthorized from every call after working previously | The chat client’s access token expired and the refresh token also expired (or the OAuth Client secret was rotated). Disconnect the connector and re-add it to mint fresh tokens. | ## [](#limitations)Limitations This page does not cover: - **Custom desktop or mobile UIs**: Build against the AI Gateway MCP endpoints directly using your platform’s HTTP client; you don’t need an OAuth Client unless you want the same external-app flow. - **Agent-to-agent calls (A2A)**: See the Agents docs; remote MCP clients are end-user-facing. - **MCP server authoring**: See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) for the server side. ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [User-delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) --- # Page 43: Set Up a Self-Managed Agent **URL**: https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents.md --- # Set Up a Self-Managed Agent > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up a Self-Managed Agent latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: self-managed-agents page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: self-managed-agents.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/self-managed-agents.adoc description: Register a self-managed agent, issue it a client credential, and route its LLM and tool calls through the AI Gateway so spend, traces, and transcripts attribute back to the agent. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Choose a self-managed agent over a managed agent for your use case learning-objective-2: Register a self-managed agent and issue it an OAuth client credential learning-objective-3: Route an agent's LLM and MCP calls through the AI Gateway and stream transcripts page-git-created-date: "2026-06-17" page-git-modified-date: "2026-08-12" --- A self-managed agent is an agent you build and run yourself, registered with Redpanda Agentic Data Plane as an identity. You keep your runtime, framework, and hosting; Agentic Data Plane gives the agent a service account and a client credential, and the [AI Gateway](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-gateway) becomes the agent’s LLM and MCP endpoint. Because every model call and tool call flows through the gateway, Agentic Data Plane attributes spend, tokens, latency, and traces back to the agent. To view each session as a transcript, the agent also streams its own OpenTelemetry spans to Agentic Data Plane. Agentic Data Plane does not host or run your agent. After reading this page, you will be able to: - Choose a self-managed agent over a managed agent for your use case - Register a self-managed agent and issue it an OAuth client credential - Route an agent’s LLM and MCP calls through the AI Gateway and stream transcripts ## [](#self-managed-compared-to-managed-agents)Self-managed compared to managed agents The two agent types differ in who runs the agent and how it is defined. They coexist in the same registry, the same governance views, and the same cost-attribution queries. | Question | Self-managed | Managed | | --- | --- | --- | | Who runs the agent? | You do. Agentic Data Plane registers the agent and proxies its LLM and tool calls, but the runtime is yours. | Redpanda deploys, runs, and observes the agent for you. | | How is the agent defined? | It is already coded in your own framework, for example, LangChain, CrewAI, or a custom runtime. | You configure it declaratively through the create form, with no runtime code to maintain. | | What connects it to Agentic Data Plane? | A client credential the agent exchanges for a gateway token. Your code points its LLM and MCP clients at the gateway. | The managed runtime wires the gateway for you. | For the declarative path, see [Create an agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). ## [](#prerequisites)Prerequisites - Access to create an agent, from an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) granting `Action::"Agent.create"`, or from the Admin role. See [Agent management permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#agent-management-permissions). - At least one [LLM provider configured](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) in Agentic Data Plane. The agent calls the model through this provider. - If the agent calls tools: One or more [MCP servers](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) registered in Agentic Data Plane. - An agent built in your own framework. The **Setup** tab generates ready-to-paste samples for ai-sdk-go, LangChain, CrewAI, ADK Java, ADK Go, ADK Python, Vercel AI SDK, and Mastra. ## [](#register-the-agent)Register the agent 1. Open **Agents** in the sidebar. 2. Click **Create agent**. 3. Click **I host it myself**, so Agentic Data Plane registers the agent as an identity and leaves the runtime to you. ![The runtime choice in the create-agent flow, with a Redpanda manages it card for the managed runtime and an I host it myself card for self-managed agents](https://docs.redpanda.com/agentic-data-plane/shared/_images/create-agent-runtime-choice.png) 4. Fill in the identity fields on the **Register self-managed agent** canvas, then click **Create agent**: - **Name**: Required. Human-readable name shown in the agent list and detail header. Agentic Data Plane derives the agent’s resource ID from this name when you register the agent; you don’t enter the ID directly. The derived ID is used in URLs, in cost-attribution queries, and as the agent’s resource identifier, and it can’t be changed after creation. - **Description**: Optional. Up to 1024 characters. An internal note shown on the agent’s detail page. - **Tags**: Optional key/value pairs to organize and filter agents. Expand the **Advanced** area to add them. The agent opens on its detail page with a **Self-managed** type badge. A self-managed agent carries no provider, model, or tool configuration of its own: it is an identity that calls the organization’s shared gateway resources. ## [](#issue-a-client-secret)Issue a client secret Agentic Data Plane provisions a service account for the agent at registration. To authenticate the agent’s calls, issue an OAuth 2.0 client secret on the agent’s **Credentials** tab. 1. Open the agent’s **Credentials** tab. ![The Credentials tab for a self-managed agent, showing the service account Client ID, the authorized scope, and an empty client-secrets list with a Create secret button](https://docs.redpanda.com/agentic-data-plane/connect/_images/self-managed-agent-credentials.png) 2. Note the **Client ID**. It has the form `serviceaccounts/`, where `` is the agent’s identifier. The Client ID is public and stable: every secret on the agent shares it. 3. Click **Create secret**. 4. Optionally enter a **Name** to identify the secret in the list and in audit logs, for example, `production`. If you leave it blank, Agentic Data Plane assigns a default name such as `secret-1`. 5. Click **Generate secret**. Agentic Data Plane shows the plaintext **Client secret** one time. ![The Secret created dialog showing a masked client secret, a Copy secret button, and a warning that the plaintext is shown only at creation](https://docs.redpanda.com/agentic-data-plane/connect/_images/self-managed-agent-secret-created.png) 6. Copy the secret into your secret manager or container environment variables, then click **I’ve saved it**. > ❗ **IMPORTANT** > > The client secret is shown only at creation. Agentic Data Plane stores a hash and cannot show the plaintext again. Each secret expires 90 days after creation. To rotate without downtime, create a new secret, deploy it, and then revoke the old one with **Revoke** on its row in the secrets list. ## [](#connect-your-agent-to-the-ai-gateway)Connect your agent to the AI Gateway The agent’s **Setup** tab generates everything your code needs: the gateway endpoints, an environment-variable block, and a copy-paste SDK sample for your framework. The tab has two parts, shown as cards you switch between: - **Connect through the gateway** (required): Authentication, the gateway endpoints, and the wired code samples that route the agent’s LLM and MCP calls. This is what attributes spend and call records to the agent. - **Stream transcripts** (optional): Instrumentation that exports your agent’s OpenTelemetry spans so its sessions appear on the **Transcripts** tab. If transcripts aren’t enabled for your environment, this card is marked **Not available**. Open the **Setup** tab and start with **Connect through the gateway**. ![The Setup tab for a self-managed agent](https://docs.redpanda.com/agentic-data-plane/connect/_images/self-managed-agent-setup-steps.png) Under **Connect through the gateway**, select your **LLM provider**. Optionally, select the **MCP servers** the sample connects through. Your selection fills in the **Environment variables** block without changing the agent. ![The Setup tab Environment variables block, with exported values for the client ID, token URL, LLM provider URL, and MCP base URL](https://docs.redpanda.com/agentic-data-plane/connect/_images/self-managed-agent-setup-env.png) In the endpoint URLs below, `` is your cluster’s identifier and `` is `https://aigw..clusters.rdpa.co`. Copy the exact values from the **Setup** tab. ### [](#authenticate-with-the-client-credential)Authenticate with the client credential The gateway runs its own OAuth 2.0 identity provider. Exchange the Client ID and client secret for a short-lived access token with the `client_credentials` grant against the token endpoint: ```text /oauth/idp/token ``` Send the resulting token as an `Authorization: Bearer` header on every LLM and MCP request. The gateway authenticates on this token and injects the real upstream provider key itself, so your SDK’s own API-key field is a placeholder. Your client is responsible for refreshing the token before it expires. ### [](#route-llm-calls-through-the-gateway)Route LLM calls through the gateway Point your SDK’s base URL at the provider’s gateway endpoint instead of the upstream API: ```text /llm/v1/providers/ ``` In this URL, `` is the name of an LLM provider you configured in Agentic Data Plane. The gateway forwards each provider’s native API to the upstream, so you keep using the provider’s own SDK. The provider enforces a model allow-list: pick a model the provider serves, or the gateway rejects the call. For the full proxy contract and per-SDK setup, see [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/). ### [](#route-mcp-tool-calls-through-the-gateway)Route MCP tool calls through the gateway Point your MCP client at each server’s gateway URL, with the same bearer token: ```text /mcp/v1/ ``` In this URL, `` is the name of an MCP server registered in Agentic Data Plane. Routing tool calls through the gateway keeps them under the same identity, governance, and observability as the model calls. ### [](#group-a-sessions-calls-with-a-conversation-id)Group a session’s calls with a conversation ID Stamp every request, both the model call and each tool call, with the `X-Redpanda-Genai-Conversation` header set to your framework’s own session identifier, for example, a chat-thread ID or a request ID. Using the same value on every request in a session tags that session’s gateway calls with one conversation ID, so the gateway’s records of the turn group together. The header does not affect authentication or whether calls succeed. ### [](#stream-transcripts)Stream transcripts Agentic Data Plane assembles the agent’s **Transcripts** tab from OpenTelemetry spans your agent exports itself, not from the gateway calls alone. On the **Setup** tab, switch to **Stream transcripts** and follow the instrumentation steps for your framework: point a standard OpenTelemetry exporter at the cluster’s OTLP endpoint, authenticate it by minting short-lived tokens from the same gateway credential your code already uses, and put a stable `gen_ai.conversation.id` attribute on each session’s root span. Use the same identifier you send in the `X-Redpanda-Genai-Conversation` header so the gateway records and the transcript line up. Each distinct `gen_ai.conversation.id` becomes one conversation on the **Transcripts** tab, grouping the exported spans that carry it into one row. The **Turns** column counts the model calls in that conversation, so an agent that loops over several tool calls shows more than one. Streaming transcripts is optional; if it isn’t enabled for your environment, the **Setup** tab marks it **Not available**. ![The Transcripts tab listing conversations, each grouped under one conversation ID, with columns for when it started, its duration, the number of turns, the status, and the token count](https://docs.redpanda.com/agentic-data-plane/connect/_images/self-managed-agent-transcripts.png) ## [](#framework-samples)Framework samples The **Setup** tab generates a ready-to-paste sample for your framework, prefilled with your selected provider and MCP servers. Each sample performs these steps the framework’s way: it runs the `client_credentials` grant, routes the LLM client and every MCP client through one HTTP client that carries the bearer token, and stamps the framework’s own session identifier as the `X-Redpanda-Genai-Conversation` header. The samples read their configuration from these environment variables: | Variable | Where to get it | | --- | --- | | REDPANDA_CLIENT_ID | The Client ID from the Credentials tab. | | REDPANDA_CLIENT_SECRET | A client secret you minted on the Credentials tab. | | REDPANDA_TOKEN_URL | The token endpoint from the Setup tab, ending in /oauth/idp/token. | | REDPANDA_LLM_PROVIDER_URL | The provider-scoped LLM endpoint from the Setup tab, ending in /llm/v1/providers/. | | REDPANDA_LLM_PROVIDER_TYPE | The upstream provider family the SDK builds against: openai, anthropic, or google. | | REDPANDA_LLM_MODEL | A model the provider serves. | | REDPANDA_MCP_BASE_URL | The MCP base endpoint from the Setup tab, ending in /mcp/v1. | | REDPANDA_MCP_SERVERS | A comma-separated list of MCP server names to connect, or empty for none. | Set these variables, then run the sample for your framework. ### ai-sdk-go ```go package main import ( "context" "crypto/rand" "encoding/hex" "fmt" "log" "net/http" "os" "strings" "time" "golang.org/x/oauth2" "golang.org/x/oauth2/clientcredentials" "github.com/redpanda-data/ai-sdk-go/agent" "github.com/redpanda-data/ai-sdk-go/agent/llmagent" "github.com/redpanda-data/ai-sdk-go/llm" "github.com/redpanda-data/ai-sdk-go/providers/anthropic" "github.com/redpanda-data/ai-sdk-go/providers/google" "github.com/redpanda-data/ai-sdk-go/providers/openai" "github.com/redpanda-data/ai-sdk-go/runner" "github.com/redpanda-data/ai-sdk-go/store/session" "github.com/redpanda-data/ai-sdk-go/tool" "github.com/redpanda-data/ai-sdk-go/tool/mcp" ) // convoKey carries the conversation id on the context. type convoKey struct{} // convoTransport stamps the session id (read from the context) as the // conversation header. It sits beneath the oauth2 transport, so one http.Client // carries the bearer AND the conversation id on the LLM call and every MCP tool // call. type convoTransport struct{ base http.RoundTripper } func (t *convoTransport) RoundTrip(r *http.Request) (*http.Response, error) { if id, ok := r.Context().Value(convoKey{}).(string); ok && id != "" { r = r.Clone(r.Context()) r.Header.Set("X-Redpanda-Genai-Conversation", id) } return t.base.RoundTrip(r) } func main() { ctx := context.Background() // OAuth2 client_credentials: x/oauth2 fetches and refreshes the bearer and // its Transport sets it on every request; convoTransport underneath adds the // conversation header. One client instruments the LLM call and every MCP call. cc := clientcredentials.Config{ ClientID: mustEnv("REDPANDA_CLIENT_ID"), ClientSecret: mustEnv("REDPANDA_CLIENT_SECRET"), TokenURL: mustEnv("REDPANDA_TOKEN_URL"), } hc := &http.Client{Transport: &oauth2.Transport{ Source: cc.TokenSource(ctx), Base: &convoTransport{base: http.DefaultTransport}, }} model, err := buildModel(ctx, hc) if err != nil { log.Fatal(err) } // MCP tools ride the SAME client. Each client syncs its server's tools into a // shared registry; Start connects and performs that initial sync. The agent // is then built from the registry, so the model can actually call the tools. registry := tool.NewRegistry(tool.RegistryConfig{}) mcpBase := mustEnv("REDPANDA_MCP_BASE_URL") for _, name := range mcpServers() { factory := mcp.NewStreamableTransport(mcpBase+"/"+name, mcp.WithHTTPClient(hc)) client, err := mcp.NewClient(name, factory, mcp.WithRegistry(registry), // sync this server's tools into the registry mcp.WithToolTimeout(time.Minute)) if err != nil { log.Fatal(err) } if err := client.Start(ctx); err != nil { log.Fatal(err) } defer client.Close() } // WithTools(registry) is what hands the synced MCP tools to the model. ag, err := llmagent.New("assistant", "You are a helpful agent.", model, llmagent.WithTools(registry)) if err != nil { log.Fatal(err) } run, err := runner.New(ag, session.NewInMemoryStore()) if err != nil { log.Fatal(err) } // The runner takes a CALLER-owned conversation id (run.Run keys the session on // it; ai-sdk-go does not mint one). Use your app's own id - a chat thread id, // request id, A2A contextId - reused across the turn; we mint one here. It is // the value convoTransport stamps as X-Redpanda-Genai-Conversation on the model // call and every MCP tool call. const prompt = "What tools can you call?" conversationID := newConversationID() ctx = context.WithValue(ctx, convoKey{}, conversationID) fmt.Printf("> %s\n\n", prompt) msg := llm.NewMessage(llm.RoleUser, llm.NewTextPart(prompt)) for ev, err := range run.Run(ctx, "user-123", conversationID, msg) { if err != nil { log.Fatal(err) } // MessageEvent carries a finished assistant turn (an agentic run may have // several). Print its text so you can see the model actually replied. if m, ok := ev.(agent.MessageEvent); ok { fmt.Println(m.Response.TextContent()) } } } // buildModel constructs the native ai-sdk-go model for the configured provider. // REDPANDA_LLM_PROVIDER_TYPE selects the SDK: "anthropic" and "google" use their // native wire (the gateway forwards /v1/messages and /v1beta/...:generateContent // to the upstream), everything else uses OpenAI chat-completions. All three point // at the same provider-scoped REDPANDA_LLM_PROVIDER_URL. The bearer (set by the // oauth2 transport) is the real auth; the key arg only satisfies the constructor // (the gateway ignores the native x-api-key/x-goog-api-key). func buildModel(ctx context.Context, hc *http.Client) (llm.Model, error) { base := mustEnv("REDPANDA_LLM_PROVIDER_URL") model := mustEnv("REDPANDA_LLM_MODEL") const key = "redpanda-gateway" switch strings.ToLower(os.Getenv("REDPANDA_LLM_PROVIDER_TYPE")) { case "anthropic": p, err := anthropic.NewProvider(key, anthropic.WithBaseURL(base), anthropic.WithHTTPClient(hc)) if err != nil { return nil, err } return p.NewModel(model) case "google", "gemini": p, err := google.NewProvider(ctx, key, google.WithBaseURL(base), google.WithHTTPClient(hc)) if err != nil { return nil, err } return p.NewModel(model) default: // openai (and openai-compatible) p, err := openai.NewProvider(key, openai.WithBaseURL(base), openai.WithHTTPClient(hc)) if err != nil { return nil, err } return p.NewModel(model) } } // mustEnv reads a required env var, exiting with a clear message (not an opaque // downstream panic) when it is unset. Export the values from the Setup tab. func mustEnv(k string) string { v := os.Getenv(k) if v == "" { log.Fatalf("missing env var %s - export it from the Setup tab", k) } return v } // newConversationID mints a fresh conversation id. In a real app, use your own // per-conversation id (chat thread id, request id, A2A contextId) reused across // the turn, not a value generated per call. func newConversationID() string { b := make([]byte, 8) _, _ = rand.Read(b) return "conv-" + hex.EncodeToString(b) } // mcpServers reads the comma-separated REDPANDA_MCP_SERVERS list. Empty is fine // - the agent then runs with no MCP tools. func mcpServers() []string { var out []string for _, p := range strings.Split(os.Getenv("REDPANDA_MCP_SERVERS"), ",") { if p = strings.TrimSpace(p); p != "" { out = append(out, p) } } return out } ``` ### LangChain ```python import asyncio import contextvars import os import httpx from langchain_mcp_adapters.client import MultiServerMCPClient from langgraph.prebuilt import create_react_agent def env(k: str) -> str: """Read a required env var, failing with a clear message (not an opaque KeyError) when it is unset. Export the values from the Setup tab.""" v = os.environ.get(k) if not v: raise SystemExit(f"missing env var {k} - export it from the Setup tab") return v def get_access_token() -> str: """OAuth2 client_credentials grant against the gateway IDP (httpx is already a dep).""" resp = httpx.post( env("REDPANDA_TOKEN_URL"), data={ "grant_type": "client_credentials", "client_id": env("REDPANDA_CLIENT_ID"), "client_secret": env("REDPANDA_CLIENT_SECRET"), }, ) resp.raise_for_status() return resp.json()["access_token"] token = get_access_token() mcp_base = env("REDPANDA_MCP_BASE_URL") # The LangGraph thread_id IS the conversation. Carry it in a contextvar so the # MCP transport reads the current one per request. thread_var: contextvars.ContextVar[str] = contextvars.ContextVar("thread") class GatewayAuth(httpx.Auth): """MCP headers are fixed per connection, so inject per request via httpx.Auth. The default tool mode opens a fresh session per call, so auth_flow re-reads the contextvar and always carries the current thread id. """ def auth_flow(self, request): request.headers["Authorization"] = f"Bearer {token}" request.headers["X-Redpanda-Genai-Conversation"] = thread_var.get() yield request def mcp_servers() -> dict: """Build the MultiServerMCPClient connection map from REDPANDA_MCP_SERVERS.""" servers = {} for name in os.environ.get("REDPANDA_MCP_SERVERS", "").split(","): name = name.strip() if name: servers[name] = { "transport": "streamable_http", "url": f"{mcp_base}/{name}", "auth": GatewayAuth(), } return servers mcp_client = MultiServerMCPClient(mcp_servers()) def build_model(thread_id: str): """Construct the native LangChain chat model for the configured provider. REDPANDA_LLM_PROVIDER_TYPE picks the SDK. Auth is the gateway bearer token, injected on the Authorization header; the SDK's own api_key field is just a non-empty placeholder (the gateway ignores the native x-api-key / x-goog-api-key). The thread id rides along as the conversation header. """ model = env("REDPANDA_LLM_MODEL") base_url = env("REDPANDA_LLM_PROVIDER_URL") headers = { "Authorization": f"Bearer {token}", "X-Redpanda-Genai-Conversation": thread_id, } provider = os.environ.get("REDPANDA_LLM_PROVIDER_TYPE", "openai").lower() if provider == "anthropic": from langchain_anthropic import ChatAnthropic return ChatAnthropic( model=model, base_url=base_url, default_headers=headers, api_key="unused", # gateway authenticates on the bearer header ) if provider in ("google", "gemini"): from langchain_google_genai import ChatGoogleGenerativeAI return ChatGoogleGenerativeAI( model=model, base_url=base_url, api_version="v1beta", # native Gemini wire under the provider URL additional_headers=headers, api_key="unused", # gateway authenticates on the bearer header ) # openai (and openai-compatible) from langchain_openai import ChatOpenAI return ChatOpenAI( model=model, base_url=base_url, api_key=token, ).bind(extra_headers={"X-Redpanda-Genai-Conversation": thread_id}) async def chat(thread_id: str, text: str): thread_var.set(thread_id) # one id per conversation tools = await mcp_client.get_tools() llm = build_model(thread_id) agent = create_react_agent(llm, tools) return await agent.ainvoke( {"messages": [("user", text)]}, config={"configurable": {"thread_id": thread_id}}, ) async def main(): result = await chat("user-123-thread-1", "What tools can you call?") for message in result["messages"]: message.pretty_print() if __name__ == "__main__": asyncio.run(main()) ``` ### CrewAI ```python import os import httpx from crewai import LLM, Agent, Crew, Task from crewai.llms.hooks import BaseInterceptor from crewai_tools import MCPServerAdapter def env(k: str) -> str: """Read a required env var, failing with a clear message (not an opaque KeyError) when it is unset. Export the values from the Setup tab.""" v = os.environ.get(k) if not v: raise SystemExit(f"missing env var {k} - export it from the Setup tab") return v def get_access_token() -> str: """OAuth2 client_credentials grant against the gateway IDP (httpx is already a dep).""" resp = httpx.post( env("REDPANDA_TOKEN_URL"), data={ "grant_type": "client_credentials", "client_id": env("REDPANDA_CLIENT_ID"), "client_secret": env("REDPANDA_CLIENT_SECRET"), }, ) resp.raise_for_status() return resp.json()["access_token"] token = get_access_token() provider_url = env("REDPANDA_LLM_PROVIDER_URL") provider_type = os.environ.get("REDPANDA_LLM_PROVIDER_TYPE", "openai").lower() model = env("REDPANDA_LLM_MODEL") mcp_base = env("REDPANDA_MCP_BASE_URL") # The gateway authenticates on this Bearer token; the native x-api-key / # x-goog-api-key are ignored, so the SDK's api_key is just a placeholder. GATEWAY_API_KEY = "redpanda-gateway" class GatewayInterceptor(BaseInterceptor): """LLM side (OpenAI / Anthropic native clients): stamp the Bearer token and the conversation id on every outbound request via a transport interceptor. The conversation id is carried per instance so one crew.kickoff() groups cleanly. Both the OpenAI and Anthropic native clients build an httpx client around this interceptor; Gemini does not support interceptors and is wired separately (see build_llm). """ def __init__(self, conversation_id: str) -> None: self.conversation_id = conversation_id def on_outbound(self, request: httpx.Request) -> httpx.Request: request.headers["Authorization"] = f"Bearer {token}" request.headers["X-Redpanda-Genai-Conversation"] = self.conversation_id return request def on_inbound(self, response): return response async def aon_outbound(self, request): return self.on_outbound(request) async def aon_inbound(self, response): return response def build_llm(conversation_id: str) -> LLM: """Build the native LLM for REDPANDA_LLM_PROVIDER_TYPE, pointed at the gateway provider URL and carrying the Bearer token + conversation header. """ if provider_type == "anthropic": # Native Anthropic SDK posts to {base_url}/v1/messages. The SDK sends # x-api-key natively; the interceptor adds Authorization: Bearer (which # the gateway authenticates on) plus the conversation header. return LLM( provider="anthropic", model=model, base_url=provider_url, api_key=GATEWAY_API_KEY, interceptor=GatewayInterceptor(conversation_id), ) if provider_type in ("google", "gemini"): # Native google-genai SDK posts to {base_url}/v1beta/models/{model}:generateContent. # It does NOT support transport interceptors, so the Bearer token and the # conversation header are set as fixed client headers via http_options. from google.genai import types return LLM( provider="gemini", model=model, api_key=GATEWAY_API_KEY, client_params={ "http_options": types.HttpOptions( base_url=provider_url, headers={ "Authorization": f"Bearer {token}", "X-Redpanda-Genai-Conversation": conversation_id, }, ), }, ) # openai (and openai-compatible): native OpenAI SDK posts to # {base_url}/chat/completions. The interceptor stamps both headers. return LLM( provider="openai", model=model, base_url=provider_url, api_key=GATEWAY_API_KEY, interceptor=GatewayInterceptor(conversation_id), ) def mcp_server_params(conversation_id: str) -> list: """MCP headers are fixed per connection, so build params per conversation.""" servers = [] for name in os.environ.get("REDPANDA_MCP_SERVERS", "").split(","): name = name.strip() if name: servers.append( { "url": f"{mcp_base}/{name}", "transport": "streamable-http", "headers": { "Authorization": f"Bearer {token}", "X-Redpanda-Genai-Conversation": conversation_id, }, } ) return servers def kickoff(prompt: str, llm: LLM, tools) -> str: agent = Agent(role="Assistant", goal="Help the user", backstory="", llm=llm, tools=tools) crew = Crew( agents=[agent], tasks=[Task(description=prompt, agent=agent, expected_output="A reply")], ) return crew.kickoff() def run_conversation(conversation_id: str, prompt: str) -> str: # One crew.kickoff() == one conversation: build the LLM and MCP clients with # this id so the model call and every tool call carry the same header. llm = build_llm(conversation_id) servers = mcp_server_params(conversation_id) if not servers: return kickoff(prompt, llm, []) with MCPServerAdapter(servers) as tools: return kickoff(prompt, llm, list(tools)) if __name__ == "__main__": print(run_conversation("user-123-conversation-1", "What tools can you call?")) ``` ### ADK Java ```java package com.redpanda.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.google.adk.agents.LlmAgent; import com.google.adk.models.langchain4j.LangChain4j; import com.google.adk.runner.Runner; import com.google.adk.sessions.InMemorySessionService; import com.google.adk.sessions.Session; import com.google.adk.tools.mcp.McpToolset; import com.google.adk.tools.mcp.StreamableHttpServerParameters; import com.google.genai.types.Content; import com.google.genai.types.Part; import dev.langchain4j.model.anthropic.AnthropicChatModel; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.googleai.GoogleAiGeminiChatModel; import dev.langchain4j.model.openai.OpenAiChatModel; import java.net.URI; import java.net.URLEncoder; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.List; import java.util.Locale; import java.util.Map; public final class Main { private Main() {} public static void main(String[] args) throws Exception { String token = accessToken(); String appName = "redpanda-self-managed-agent"; String userId = "user-123"; // ADK owns the session and mints its id - that id IS the conversation, never // a hardcoded constant. langchain4j fixes customHeaders at build time, so we // create the session first (createSession with a null id mints one), then // pass its id to the chat model and the runner. The same id rides the LLM // call and every MCP tool call. One run is one conversation. InMemorySessionService sessions = new InMemorySessionService(); Session session = sessions.createSession(appName, userId).blockingGet(); String sessionId = session.id(); String model = env("REDPANDA_LLM_MODEL"); ChatModel chat = buildChatModel(token, sessionId); // MCP: same bearer + conversation id, fixed per toolset construction. The // agent is built from the toolsets, so the model can call their tools. Map mcpHeaders = Map.of( "Authorization", "Bearer " + token, "X-Redpanda-Genai-Conversation", sessionId); String mcpBase = env("REDPANDA_MCP_BASE_URL"); List tools = new ArrayList<>(); for (String name : mcpServers()) { tools.add( new McpToolset( StreamableHttpServerParameters.builder() .url(mcpBase + "/" + name) .headers(mcpHeaders) .build())); } LlmAgent agent = LlmAgent.builder() .name("assistant") .description("Self-managed agent on the Redpanda AI Gateway.") .instruction("You are a helpful agent.") .model(LangChain4j.builder().chatModel(chat).modelName(model).build()) .tools(tools) .build(); // Build the runner over the SAME session service, so it sees the session we // just minted above. Runner runner = Runner.builder().agent(agent).appName(appName).sessionService(sessions).build(); Content message = Content.fromParts(Part.fromText("What tools can you call?")); runner .runAsync(userId, sessionId, message) .blockingForEach(event -> System.out.println(event.stringifyContent())); } /** * buildChatModel constructs the native langchain4j ChatModel for the configured provider. * *

REDPANDA_LLM_PROVIDER_TYPE selects the wire: "anthropic" speaks /v1/messages and "google" * speaks /v1beta/...:generateContent (the gateway forwards both to the upstream), everything else * speaks OpenAI chat-completions. All three point at the same provider-scoped * REDPANDA_LLM_PROVIDER_URL. * *

Auth is the gateway bearer, sent on the Authorization header via langchain4j's * customHeaders(Map) - fixed at build time, so it also carries the (fixed) conversation id. The * gateway authenticates on that bearer and ignores the native x-api-key/x-goog-api-key, so we * never send a real provider key (OpenAI/Anthropic require a non-empty apiKey, so we pass a dummy * placeholder; Gemini sends no key at all). */ private static ChatModel buildChatModel(String token, String sessionId) { String base = env("REDPANDA_LLM_PROVIDER_URL"); String model = env("REDPANDA_LLM_MODEL"); Map headers = Map.of( "Authorization", "Bearer " + token, "X-Redpanda-Genai-Conversation", sessionId); String type = System.getenv("REDPANDA_LLM_PROVIDER_TYPE"); switch (type == null ? "" : type.toLowerCase(Locale.ROOT)) { case "anthropic": // Native Anthropic Messages API. langchain4j posts to {baseUrl}/messages, so the base URL // carries the version segment: {provider-url}/v1 -> {provider-url}/v1/messages. return AnthropicChatModel.builder() .baseUrl(base + "/v1") .apiKey("redpanda") // dummy; gateway injects the real key and ignores x-api-key .modelName(model) .customHeaders(headers) .build(); case "google": case "gemini": // Native Gemini API. langchain4j posts to {baseUrl}/models/{model}:generateContent, so the // base URL carries the version segment: {provider-url}/v1beta. We do NOT call apiKey(...) - // leaving it null suppresses the x-goog-api-key header; auth rides the Authorization bearer // in customHeaders (requires langchain4j 1.15.0+). return GoogleAiGeminiChatModel.builder() .baseUrl(base + "/v1beta") .modelName(model) .customHeaders(headers) .build(); default: // openai (and openai-compatible) // OpenAI chat-completions. langchain4j posts to {baseUrl}/chat/completions; the provider // URL is the base as-is (the gateway's OpenAI upstream already includes /v1). return OpenAiChatModel.builder() .baseUrl(base) .apiKey("redpanda") // dummy; gateway injects the real key and ignores it .modelName(model) .customHeaders(headers) .build(); } } /** mcpServers reads the comma-separated REDPANDA_MCP_SERVERS list. */ private static List mcpServers() { List out = new ArrayList<>(); String raw = System.getenv("REDPANDA_MCP_SERVERS"); if (raw != null) { for (String name : raw.split(",")) { name = name.trim(); if (!name.isEmpty()) { out.add(name); } } } return out; } /** accessToken runs the OAuth2 client_credentials grant against the gateway IDP. */ private static String accessToken() throws Exception { String form = "grant_type=client_credentials" + "&client_id=" + enc(env("REDPANDA_CLIENT_ID")) + "&client_secret=" + enc(env("REDPANDA_CLIENT_SECRET")); HttpRequest request = HttpRequest.newBuilder(URI.create(env("REDPANDA_TOKEN_URL"))) .header("Content-Type", "application/x-www-form-urlencoded") .POST(HttpRequest.BodyPublishers.ofString(form)) .build(); HttpResponse response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString()); JsonNode node = new ObjectMapper().readTree(response.body()); return node.get("access_token").asText(); } private static String enc(String value) { return URLEncoder.encode(value, StandardCharsets.UTF_8); } /** * env reads a required env var, failing with a clear message (not an opaque downstream NPE) when * it is unset. Export the values from the Setup tab. */ private static String env(String k) { String v = System.getenv(k); if (v == null || v.isEmpty()) { throw new IllegalStateException("missing env var " + k + " - export it from the Setup tab"); } return v; } } ``` ### ADK Go ```go package main import ( "context" "fmt" "log" "net/http" "os" "strings" "golang.org/x/oauth2" "golang.org/x/oauth2/clientcredentials" "github.com/modelcontextprotocol/go-sdk/mcp" "google.golang.org/genai" "google.golang.org/adk/agent" "google.golang.org/adk/agent/llmagent" "google.golang.org/adk/model/gemini" "google.golang.org/adk/runner" "google.golang.org/adk/session" "google.golang.org/adk/tool" "google.golang.org/adk/tool/mcptoolset" ) const appName = "redpanda-self-managed-agent" // convoKey carries the ADK session id on the context. type convoKey struct{} // convoTransport stamps the session id (read from the context) as the // conversation header. It sits beneath the oauth2 transport, so one http.Client // carries the bearer AND the conversation id. ADK threads the ctx you pass to // runner.Run down to both the LLM HTTP call and the MCP tool-call POSTs. type convoTransport struct{ base http.RoundTripper } func (t *convoTransport) RoundTrip(r *http.Request) (*http.Response, error) { if id, ok := r.Context().Value(convoKey{}).(string); ok && id != "" { r = r.Clone(r.Context()) r.Header.Set("X-Redpanda-Genai-Conversation", id) // = ADK session id } return t.base.RoundTrip(r) } func main() { ctx := context.Background() // OAuth2 client_credentials: x/oauth2 fetches and refreshes the bearer and // its Transport sets it on every request; convoTransport underneath adds the // conversation header. One client instruments the LLM call and every MCP call. cc := clientcredentials.Config{ ClientID: mustEnv("REDPANDA_CLIENT_ID"), ClientSecret: mustEnv("REDPANDA_CLIENT_SECRET"), TokenURL: mustEnv("REDPANDA_TOKEN_URL"), } hc := &http.Client{Transport: &oauth2.Transport{ Source: cc.TokenSource(ctx), Base: &convoTransport{base: http.DefaultTransport}, }} // genai refuses to construct the Gemini-API client without a non-empty // APIKey, but the real auth is the bearer the oauth2 transport sets - this // just satisfies the constructor (the gateway ignores the x-goog-api-key). model, err := gemini.NewModel(ctx, mustEnv("REDPANDA_LLM_MODEL"), &genai.ClientConfig{ APIKey: "redpanda-gateway", HTTPClient: hc, HTTPOptions: genai.HTTPOptions{BaseURL: mustEnv("REDPANDA_LLM_PROVIDER_URL")}, }) if err != nil { log.Fatal(err) } // Each MCP server becomes a Toolset over the SAME client; the agent is built // from them via llmagent.Config.Toolsets, so the model can call the tools. mcpBase := mustEnv("REDPANDA_MCP_BASE_URL") var toolsets []tool.Toolset for _, name := range mcpServers() { ts, err := mcptoolset.New(mcptoolset.Config{ Transport: &mcp.StreamableClientTransport{Endpoint: mcpBase + "/" + name, HTTPClient: hc}, }) if err != nil { log.Fatal(err) } toolsets = append(toolsets, ts) } a, err := llmagent.New(llmagent.Config{ Name: "assistant", Model: model, Description: "Self-managed agent on the Redpanda AI Gateway.", Instruction: "You are a helpful agent.", Toolsets: toolsets, }) if err != nil { log.Fatal(err) } sessionService := session.InMemoryService() r, err := runner.New(runner.Config{ AppName: appName, Agent: a, SessionService: sessionService, }) if err != nil { log.Fatal(err) } // ADK owns the session; its id IS the conversation. Create it, put the id on // ctx, and the transport stamps it on the LLM call and every MCP tool call. resp, err := sessionService.Create(ctx, &session.CreateRequest{AppName: appName, UserID: "user-123"}) if err != nil { log.Fatal(err) } sessionID := resp.Session.ID() ctx = context.WithValue(ctx, convoKey{}, sessionID) const prompt = "What tools can you call?" fmt.Printf("> %s\n\n", prompt) msg := genai.NewContentFromText(prompt, genai.RoleUser) for ev, err := range r.Run(ctx, "user-123", sessionID, msg, agent.RunConfig{}) { if err != nil { log.Fatal(err) } if ev.LLMResponse.Content == nil { continue } for _, p := range ev.LLMResponse.Content.Parts { fmt.Print(p.Text) // the assistant's reply, streamed as parts arrive } } fmt.Println() } // mustEnv reads a required env var, exiting with a clear message (not an opaque // downstream panic) when it is unset. Export the values from the Setup tab. func mustEnv(k string) string { v := os.Getenv(k) if v == "" { log.Fatalf("missing env var %s - export it from the Setup tab", k) } return v } // mcpServers reads the comma-separated REDPANDA_MCP_SERVERS list. Empty is fine // - the agent then runs with no MCP tools. func mcpServers() []string { var out []string for _, p := range strings.Split(os.Getenv("REDPANDA_MCP_SERVERS"), ",") { if p = strings.TrimSpace(p); p != "" { out = append(out, p) } } return out } ``` ### ADK Python ```python import asyncio import os import uuid import requests from google.adk.agents import LlmAgent from google.adk.models.lite_llm import LiteLlm from google.adk.runners import InMemoryRunner from google.adk.tools.mcp_tool.mcp_session_manager import StreamableHTTPConnectionParams from google.adk.tools.mcp_tool.mcp_toolset import McpToolset from google.genai import types CONVERSATION_HEADER = "X-Redpanda-Genai-Conversation" APP_NAME = "redpanda-self-managed-agent" USER_ID = "user-123" def env(k: str) -> str: # Required env var, or a clear failure (not an opaque KeyError). Export from the Setup tab. v = os.environ.get(k) if not v: raise SystemExit(f"missing env var {k} - export it from the Setup tab") return v def get_access_token() -> str: # OAuth2 client_credentials grant against the gateway IDP (requests sets the # form content-type automatically when data= is a dict). resp = requests.post( env("REDPANDA_TOKEN_URL"), data={ "grant_type": "client_credentials", "client_id": env("REDPANDA_CLIENT_ID"), "client_secret": env("REDPANDA_CLIENT_SECRET"), }, timeout=30, ) resp.raise_for_status() return resp.json()["access_token"] def model_arg() -> str: # ADK Python drives the gateway through LiteLlm. REDPANDA_LLM_PROVIDER_TYPE # selects the route. Anthropic uses the native Messages wire (LiteLlm appends # /v1/messages to api_base). OpenAI and Google both go through the # OpenAI-compatible /chat/completions route: LiteLlm's gemini/ provider drops # the /v1beta path on a custom api_base, so Google is routed through openai/ too. model = env("REDPANDA_LLM_MODEL") provider = os.environ.get("REDPANDA_LLM_PROVIDER_TYPE", "openai").lower() if provider == "anthropic": return f"anthropic/{model}" return f"openai/{model}" def build_model(token: str, conversation_id: str) -> LiteLlm: # api_key is a placeholder: the gateway authenticates on the bearer, not the # native key. The bearer + conversation id ride extra_headers, which LiteLlm # forwards verbatim to the provider request (the LLM call). return LiteLlm( model=model_arg(), api_base=env("REDPANDA_LLM_PROVIDER_URL"), api_key="redpanda-gateway", extra_headers={ "Authorization": f"Bearer {token}", CONVERSATION_HEADER: conversation_id, }, ) def build_toolsets(token: str, conversation_id: str) -> list[McpToolset]: # One McpToolset per server over MCP Streamable HTTP; the same bearer + # conversation id ride every tool call, matching the LLM call. base = env("REDPANDA_MCP_BASE_URL") headers = {"Authorization": f"Bearer {token}", CONVERSATION_HEADER: conversation_id} names = [n.strip() for n in os.environ.get("REDPANDA_MCP_SERVERS", "").split(",") if n.strip()] return [ McpToolset(connection_params=StreamableHTTPConnectionParams(url=f"{base}/{name}", headers=headers)) for name in names ] async def main() -> None: # Fetched once for this short-lived sample; a long-running agent should # refresh the bearer before its TTL. token = get_access_token() # conversation_id is YOUR conversation id (chat thread, request id, A2A # contextId) - never a shared constant. ADK uses it as the session id, and it # is stamped on the LLM call and every MCP tool call. Mint one here. This demo # runs one conversation; a multi-conversation app rebuilds build_model + # build_toolsets per conversation (the id is baked into both). conversation_id = uuid.uuid4().hex agent = LlmAgent( name="assistant", model=build_model(token, conversation_id), instruction="You are a helpful agent.", tools=build_toolsets(token, conversation_id), ) runner = InMemoryRunner(agent=agent, app_name=APP_NAME) # create_session accepts session_id=, so the caller's id IS the session id. await runner.session_service.create_session( app_name=APP_NAME, user_id=USER_ID, session_id=conversation_id ) message = types.Content(role="user", parts=[types.Part(text="What tools can you call?")]) async for event in runner.run_async(user_id=USER_ID, session_id=conversation_id, new_message=message): if event.is_final_response() and event.content and event.content.parts: print(event.content.parts[0].text) # the assistant's reply if __name__ == "__main__": asyncio.run(main()) ``` ### Vercel AI SDK ```typescript import { createAnthropic } from '@ai-sdk/anthropic'; import { createGoogleGenerativeAI } from '@ai-sdk/google'; import { createOpenAICompatible } from '@ai-sdk/openai-compatible'; import { createMCPClient } from '@ai-sdk/mcp'; import { generateText, stepCountIs, type LanguageModel, type ToolSet } from 'ai'; import { randomUUID } from 'node:crypto'; const CONVERSATION_HEADER = 'X-Redpanda-Genai-Conversation'; // Required env var, or a clear failure (not an opaque 401 later). Export from the Setup tab. function env(key: string): string { const value = process.env[key]; if (!value) { throw new Error(`missing env var ${key} - export it from the Setup tab`); } return value; } // OAuth2 client_credentials grant against the gateway IDP -> the bearer for every call. async function getAccessToken(): Promise { const res = await fetch(env('REDPANDA_TOKEN_URL'), { method: 'POST', headers: { 'content-type': 'application/x-www-form-urlencoded' }, body: new URLSearchParams({ grant_type: 'client_credentials', client_id: env('REDPANDA_CLIENT_ID'), client_secret: env('REDPANDA_CLIENT_SECRET'), }), }); if (!res.ok) { throw new Error(`token request failed: ${res.status}`); } const { access_token } = (await res.json()) as { access_token: string }; return access_token; } // Fetched once for this short-lived sample. A long-running agent should refresh // the bearer before its TTL (for example, mint it inside a custom fetch, as the // Mastra sample does). const token = await getAccessToken(); const mcpBase = env('REDPANDA_MCP_BASE_URL'); // buildModel constructs the native AI SDK model for the configured provider. // REDPANDA_LLM_PROVIDER_TYPE selects the provider: "anthropic" and "google" use // their native wire (the gateway forwards /v1/messages and // /v1beta/...:generateContent to the upstream), everything else uses the // OpenAI-compatible /chat/completions route. All three point at the same // provider-scoped REDPANDA_LLM_PROVIDER_URL. The bearer is the real auth; apiKey // is a placeholder for the ignored native x-api-key / x-goog-api-key. function buildModel(): LanguageModel { const base = env('REDPANDA_LLM_PROVIDER_URL'); const model = env('REDPANDA_LLM_MODEL'); switch ((process.env['REDPANDA_LLM_PROVIDER_TYPE'] ?? 'openai').toLowerCase()) { case 'anthropic': { // baseURL + "/v1" -> the gateway forwards /v1/messages. const provider = createAnthropic({ baseURL: `${base}/v1`, apiKey: 'redpanda-gateway', headers: { Authorization: `Bearer ${token}` }, }); return provider(model); } case 'google': case 'gemini': { // baseURL + "/v1beta" -> /v1beta/models/:generateContent. const provider = createGoogleGenerativeAI({ baseURL: `${base}/v1beta`, apiKey: 'redpanda-gateway', headers: { Authorization: `Bearer ${token}` }, }); return provider(model); } default: { // OpenAI-compatible /chat/completions. apiKey becomes the Authorization: // Bearer the gateway authenticates on, so no placeholder header is needed. const provider = createOpenAICompatible({ name: 'redpanda', apiKey: token, baseURL: base, }); return provider(model); } } } const model = buildModel(); // MCP tool servers over Streamable HTTP. Transport headers are fixed per // connection, so build the clients per conversation: the bearer (gateway auth) // and the SAME conversation id ride every MCP tool call, matching the LLM call. async function connectTools(conversationId: string): Promise<{ tools: ToolSet; close: () => Promise }> { const headers = { Authorization: `Bearer ${token}`, [CONVERSATION_HEADER]: conversationId }; const urls = (process.env['REDPANDA_MCP_SERVERS'] ?? '') .split(',') .map((s) => s.trim()) .filter(Boolean) .map((name) => `${mcpBase}/${name}`); const clients = await Promise.all( urls.map((url) => createMCPClient({ transport: { type: 'http', url, headers } })) ); const toolSets = await Promise.all(clients.map((client) => client.tools())); const tools: ToolSet = Object.assign({}, ...toolSets); const close = async (): Promise => { await Promise.allSettled(clients.map((client) => client.close())); }; return { tools, close }; } // conversationId is YOUR conversation id, passed in by the caller (chat thread, // request id, A2A contextId) - never a shared constant. One chat() == one // conversation; the same id is stamped on the LLM call and every MCP tool call. async function chat(conversationId: string, prompt: string): Promise { const { tools, close } = await connectTools(conversationId); try { const { text } = await generateText({ model, tools, prompt, stopWhen: stepCountIs(10), // Stamp the conversation id so the gateway groups this LLM call with the // MCP tool calls into one transcript. headers: { [CONVERSATION_HEADER]: conversationId }, }); return text; } finally { await close(); } } // In your app the conversation id is the inbound thread/request id; mint one here. const answer = await chat(randomUUID(), 'What tools can you call?'); console.log(answer); ``` ### Mastra ```typescript import { createAnthropic } from '@ai-sdk/anthropic'; import { createGoogleGenerativeAI } from '@ai-sdk/google'; import { createOpenAICompatible } from '@ai-sdk/openai-compatible'; import { Agent } from '@mastra/core/agent'; import { MCPClient } from '@mastra/mcp'; import { AsyncLocalStorage } from 'node:async_hooks'; import { randomUUID } from 'node:crypto'; import type { LanguageModel } from 'ai'; const CONVERSATION_HEADER = 'X-Redpanda-Genai-Conversation'; // Required env var, or a clear failure (not an opaque 401 later). Export from the Setup tab. function env(key: string): string { const value = process.env[key]; if (!value) { throw new Error(`missing env var ${key} - export it from the Setup tab`); } return value; } // Holds the CURRENT conversation id for the in-flight turn. gatewayFetch reads it // so the LLM call and every MCP tool call carry the SAME id. In your app this is // your own id (chat thread, request id, A2A contextId) - never a shared constant. const conversation = new AsyncLocalStorage(); // OAuth2 client_credentials bearer, cached and refreshed before it expires. let token: { value: string; expiresAt: number } | undefined; async function bearer(): Promise { if (token && Date.now() < token.expiresAt - 30_000) { return token.value; } const res = await fetch(env('REDPANDA_TOKEN_URL'), { method: 'POST', headers: { 'content-type': 'application/x-www-form-urlencoded' }, body: new URLSearchParams({ grant_type: 'client_credentials', client_id: env('REDPANDA_CLIENT_ID'), client_secret: env('REDPANDA_CLIENT_SECRET'), }), }); if (!res.ok) { throw new Error(`token request failed: ${res.status}`); } const json = (await res.json()) as { access_token: string; expires_in?: number }; token = { value: json.access_token, expiresAt: Date.now() + (json.expires_in ?? 3600) * 1000 }; return token.value; } // ONE fetch shared by the model provider and every MCP server: it sets the bearer // (gateway auth) and stamps the conversation id (read live from AsyncLocalStorage) // on every request. Sharing it keeps the LLM call and the MCP tool calls in lockstep. const gatewayFetch: typeof fetch = async (input, init) => { const headers = new Headers(init?.headers); headers.set('Authorization', `Bearer ${await bearer()}`); const id = conversation.getStore(); if (id) { headers.set(CONVERSATION_HEADER, id); } return fetch(input, { ...init, headers }); }; const mcpBase = env('REDPANDA_MCP_BASE_URL'); // buildModel builds the AI SDK model Mastra runs on, pointed at the gateway. // REDPANDA_LLM_PROVIDER_TYPE selects the provider: "anthropic" and "google" use // their native wire with the version segment on the base URL (/v1, /v1beta), // everything else uses the OpenAI-compatible /chat/completions route. gatewayFetch // overwrites the Authorization header with the real bearer on every request, so // apiKey is just a placeholder that satisfies each constructor. function buildModel(): LanguageModel { const base = env('REDPANDA_LLM_PROVIDER_URL'); const model = env('REDPANDA_LLM_MODEL'); switch ((process.env['REDPANDA_LLM_PROVIDER_TYPE'] ?? 'openai').toLowerCase()) { case 'anthropic': { const provider = createAnthropic({ baseURL: `${base}/v1`, apiKey: 'redpanda-gateway', fetch: gatewayFetch, }); return provider(model); } case 'google': case 'gemini': { const provider = createGoogleGenerativeAI({ baseURL: `${base}/v1beta`, apiKey: 'redpanda-gateway', fetch: gatewayFetch, }); return provider(model); } default: { const provider = createOpenAICompatible({ name: 'redpanda', apiKey: 'redpanda-gateway', baseURL: base, fetch: gatewayFetch, }); return provider(model); } } } // Each MCP server is ${mcpBase}/ over Streamable HTTP; gatewayFetch carries // the bearer + conversation id on every tool call. Built once, reused per turn. const serverNames = (process.env['REDPANDA_MCP_SERVERS'] ?? '') .split(',') .map((s) => s.trim()) .filter(Boolean); const mcp = new MCPClient({ servers: Object.fromEntries( serverNames.map((name) => [name, { url: new URL(`${mcpBase}/${name}`), fetch: gatewayFetch }]) ), }); const agent = new Agent({ name: 'assistant', instructions: 'You are a helpful agent.', model: buildModel(), tools: await mcp.listTools(), }); // One conversation == one id, scoped with conversation.run so gatewayFetch stamps // it on the model call and every MCP tool call. In your app, pass your own id here. async function chat(conversationId: string, prompt: string): Promise { return conversation.run(conversationId, async () => { const result = await agent.generate(prompt, { maxSteps: 10 }); return result.text; }); } const answer = await chat(randomUUID(), 'What tools can you call?'); console.log(answer); await mcp.disconnect(); ``` > 📝 **NOTE** > > ADK Go ships only Gemini-shaped models, so the ADK Go sample works against a Google provider only. For an OpenAI or Anthropic provider, use one of the other frameworks. ## [](#observe-the-agent)Observe the agent Because the agent’s traffic flows through the gateway, Agentic Data Plane attributes its cost and usage without any instrumentation in your code: spend, tokens, and latency roll up to the agent, from the calls routed through the gateway. See them on the agent’s **Cost & Usage** tab and in [budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/). Transcripts take one more step. Routing calls through the gateway records their cost and usage, but Agentic Data Plane assembles the **Transcripts** tab from the OpenTelemetry spans your agent exports. When the agent streams its own spans (the **Setup** tab’s **Stream transcripts** card, described in [Stream transcripts](#stream-transcripts)), each session appears on the agent’s **Transcripts** tab. See [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). Transcript message text is recorded per LLM provider and is off by default. A transcript always shows token usage, latency, and tool calls; it shows the prompt and response text only when input and output message recording is turned on for the provider. See [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | 401 on the token request | The Client ID or client secret is wrong, or the secret expired or was revoked. The Client ID must be the full serviceaccounts/ value. Issue a new secret on the Credentials tab. | | 403 with model_not_allowed | The model is not on the provider’s allow-list. Pick a model the provider serves. The Setup tab fills in a valid model for you. | | 404 from the LLM endpoint | The provider name in the URL does not match a configured provider. Confirm the segment after /providers/ matches the provider’s name exactly. | | The Transcripts tab stays empty | The agent is not streaming its OpenTelemetry spans. Agentic Data Plane assembles transcripts from the spans the agent exports, so enable Stream transcripts on the Setup tab and put a stable gen_ai.conversation.id on each session’s root span. | | A transcript shows usage but no message text | Message recording is off for the agent’s LLM provider. Turn on input and output message recording in the provider settings. Recording applies to future conversations only. | ## [](#next-steps)Next steps - [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) - [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [Create an agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 44: Write Effective System Prompts **URL**: https://docs.redpanda.com/agentic-data-plane/connect/system-prompts.md --- # Write Effective System Prompts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Write Effective System Prompts latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: system-prompts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: system-prompts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/system-prompts.adoc description: Write system prompts that produce reliable, predictable agent behavior through clear constraints and tool guidance. page-topic-type: best-practices personas: agent_builder learning-objective-1: Identify effective system prompt patterns for agent reliability learning-objective-2: Apply constraint patterns to prevent unintended agent behavior learning-objective-3: Evaluate system prompts for clarity and completeness page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-24" --- Write system prompts that produce reliable, predictable agent behavior. Good prompts define scope, specify constraints, and guide tool usage. After reading this page, you will be able to: - Identify effective system prompt patterns for agent reliability - Apply constraint patterns to prevent unintended agent behavior - Evaluate system prompts for clarity and completeness ## [](#role-definition)Role definition Define what your agent does and the boundaries of its responsibilities. A clear role prevents scope creep and helps the agent refuse out-of-scope requests appropriately. ### [](#be-specific-about-agent-identity)Be specific about agent identity Define what the agent does, not what it is. Do ```text You are an order fulfillment agent for an e-commerce platform. You help customers track orders, update shipping addresses, and process returns. ``` Don’t ```text You are a helpful assistant. ``` ### [](#define-what-the-agent-does-and-doesnt-do)Define what the agent does and doesn’t do Explicitly state boundaries: what tasks the agent handles, what tasks it should refuse or delegate, and when to ask for human assistance. ```text Responsibilities: - Track customer orders - Update shipping addresses - Process returns up to $500 Do not: - Provide product recommendations (redirect to website) - Process refunds above $500 (escalate to manager) - Access orders from other customers ``` ## [](#tool-specification)Tool specification Tell the agent which tools are available and when to use them. Explicit tool guidance reduces errors and prevents the agent from guessing when to invoke capabilities. ### [](#list-available-tools)List available tools Name each tool the agent can use: ```text Available tools: - get_customer_orders: Retrieve order history for a customer - update_order_status: Change order state (shipped, delivered, canceled) - calculate_refund: Compute refund amount based on return policy ``` ### [](#explain-when-to-use-each-tool)Explain when to use each tool Provide decision criteria for tool selection. Do ```text Use get_customer_orders when: - Customer asks about order history - You need order details to answer a question Use update_order_status only when: - Customer explicitly requests a cancellation - You have confirmed the order is eligible for status changes ``` Don’t ```text Use the tools as needed. ``` ## [](#constraints-and-safety)Constraints and safety Set explicit boundaries to prevent unintended agent behavior. ### [](#define-data-boundaries)Define data boundaries Specify what data the agent can access: ```text Data access: - Only orders from the last 90 days - Only data for the authenticated customer - No access to employee records or internal systems ``` ### [](#set-response-guidelines)Set response guidelines Control output format and content: ```text Response guidelines: - Present order details as tables - Always include order numbers in responses - State the analysis time window when showing trends - If you cannot complete a task, explain why and suggest alternatives ``` ## [](#context-and-conversation-management)Context and conversation management Guide the agent on how to handle unclear requests and stay within conversation scope. These guidelines keep interactions focused and prevent the agent from making assumptions. ### [](#handle-ambiguous-requests)Handle ambiguous requests Guide the agent on how to clarify: ```text When request is unclear: 1. Ask clarifying questions 2. Suggest common interpretations 3. Do not guess customer intent ``` ### [](#define-conversation-boundaries)Define conversation boundaries Set limits on conversation scope: ```text Conversation scope: - Answer questions about orders, shipping, and returns - Do not provide product recommendations (redirect to website) - Do not engage in general conversation unrelated to orders ``` ## [](#error-handling)Error handling Guide agents to handle failures gracefully through clear prompt instructions. Agent errors fall into these categories: tool failures (external system issues) and reasoning failures (agent confusion or limits). ### [](#tool-failure-types)Tool failure types Tools can fail for multiple reasons. Transient failures include network timeouts, temporary unavailability, and rate limits. Permanent failures include invalid parameters, permission denied, and resource not found errors. Partial failures occur when tools return incomplete data or warnings. ### [](#graceful-degradation)Graceful degradation Design prompts so agents continue when tools fail: Example prompt guidance for graceful degradation ```text When a tool fails: 1. Attempt an alternative tool if available 2. If no alternative exists, explain the limitation 3. Offer partial results if you retrieved some data before failure 4. Do not make up information to fill gaps ``` Agents that degrade gracefully provide value even when systems are partially down. Implement retries in tools, not in agent prompts. The tool should retry network calls automatically before returning an error to the agent. ### [](#escalation-patterns)Escalation patterns Some failures require human intervention. Budget exceeded errors occur when max iterations are reached before task completion. Insufficient tools means no tool is available for the required action. Ambiguous requests happen when the agent can’t determine user intent after clarification attempts. Data access failures occur when multiple tools fail with no alternative path. Design prompts to recognize escalation conditions: Example prompt guidance for escalation ```text When you cannot complete the task: 1. Explain what you tried and why it didn't work 2. Tell the user what information or capability is missing 3. Suggest how they can help (provide more details, contact support, and so on) ``` ### [](#common-error-scenarios)Common error scenarios Include guidance for specific error types in your system prompt: **Timeout during tool execution:** When a tool takes longer than the agent timeout limit, the agent receives a timeout error in context. The agent should explain the delay to the user and suggest a retry. **Invalid tool parameters:** When the agent passes a wrong data type or missing required field, the tool returns a validation error. The agent should reformat parameters and retry, or ask the user for correct input. **Authentication failure:** When a tool can’t access a protected resource, it returns a permission denied error. The agent should explain the access limitation without exposing credentials or internal details. ## [](#output-formatting)Output formatting Control how the agent presents information to users. Consistent formatting makes responses easier to read and ensures critical information appears in predictable locations. ### [](#specify-structure)Specify structure Define how the agent presents information: ```text Output format: - Use tables for multiple items - Use bulleted lists for steps or options - Use code blocks for tracking numbers or order IDs - Include units (dollars, kilograms) in all numeric values ``` ## [](#template-variables)Template variables The system prompt is a Go template that Redpanda renders on every invocation. Use `{{.Variable}}` expressions to inject the current date and the calling user into the prompt at request time. A prompt with no template expressions renders unchanged, so existing prompts keep working without edits. The same variables are available in agent and subagent system prompts. System prompts support field substitution (such as `{{.User.Email}}`) and the simple conditionals `{{if}}` and `{{with}}`. Loops (`{{range}}`), sub-template definitions (`{{define}}` and `{{block}}`), and template invocation (`{{template}}`) are not supported. Agentic Data Plane rejects a prompt that uses them. ### [](#available-variables)Available variables | Expression | Description | Example output | | --- | --- | --- | | {{.Now.Date}} | Current date, in YYYY-MM-DD form | 2026-06-18 | | {{.Now.Weekday}} | Current day of the week | Thursday | | {{.Now.Month}} | Current month name | June | | {{.Now.Year}} | Current year | 2026 | | {{.Now.Time}} | Raw current timestamp. Apply a Go time layout for full control, for example, {{.Now.Time.Format "15:04 MST"}}. | 15:04 UTC | | {{.User.Email}} | Email of the user the agent acts for. Empty when no user is tied to the request, such as a service-account call. | user@example.com | All times are UTC. ### [](#example-prompt)Example prompt ```text You are a support agent. Today is {{.Now.Weekday}}, {{.Now.Month}} {{.Now.Year}}. You are assisting {{.User.Email}}. Scope every answer to that user's account, and do not reference other users' data. ``` ### [](#keep-prompts-cache-friendly)Keep prompts cache-friendly Model providers can cache an identical prompt prefix between requests to reduce cost and latency. The date fields, `{{.Now.Date}}`, `{{.Now.Weekday}}`, `{{.Now.Month}}`, and `{{.Now.Year}}`, hold the same value all day, so the prompt stays identical and the cache keeps hitting. Formatting a sub-day value from `{{.Now.Time}}`, such as the minute or second, changes the prompt on every request and defeats the cache. Use `{{.Now.Time}}` only when the agent needs the time of day. ### [](#validate-template-changes)Validate template changes When you create or update a managed agent, Agentic Data Plane validates the system prompt and rejects an invalid template with an error that names the field, so a mistake surfaces when you save the agent rather than after deployment. The agent also parses the prompt when it starts. Common mistakes include: - An unclosed `{{ }}` expression - A reference to a variable that does not exist - An unsupported construct, such as `{{range}}` - An undefined function from a stray non-template macro, such as `{{ FQ_START_DATE }}` - A time call that returns more than one value, such as `{{.Now.Time.Clock}}` Test the agent after editing the prompt. ## [](#evaluation-and-testing)Evaluation and testing Test system prompts systematically to verify behavior matches intent. Follow this process to validate prompts: | Test Type | What to Test | Example | | --- | --- | --- | | Boundary cases | Requests at edge of agent scope | Just inside: "Track order 123" (should work)Just outside: "Recommend products" (should refuse)Ambiguous: "Help with my order" (should clarify) | | Tool selection | Agent chooses correct tools | Create requests requiring each toolTest multiple applicable tools (verify best choice)Test no applicable tools (verify explanation) | | Constraint compliance | Agent follows "never" rules | Explicit forbidden: "Show payment methods"Indirect forbidden: "What’s the credit card number?"Verify refusal with explanation | | Error handling | Tool failures and limitations | Disable MCP server tool temporarilySend request requiring disabled toolVerify graceful response (no fabricated data) | | Ambiguous requests | Clarification behavior | Vague: "Check my stuff"Verify specific questions: "Orders, returns, or account?"Ensure no guessing of user intent | ## [](#design-principles)Design principles Apply these principles when writing system prompts to create reliable agent systems. ### [](#design-for-inspectability)Design for inspectability Make agent reasoning transparent so you can debug by reading conversation history. Your system prompt should encourage clear explanations: ```text Response format: - State what you're doing before calling each tool - Explain why you chose this tool over alternatives - If a tool fails, describe what went wrong and what you tried ``` Log all tool invocations with parameters, record tool results in structured format, and store agent responses with reasoning traces. Opaque agents that "just work" are impossible to fix when they break. ### [](#design-for-testability)Design for testability Test agents with boundary cases (requests at the edge of agent capability), error injection (simulate tool failures to verify graceful degradation), context limits (long conversations approaching token limits), and ambiguous requests (unclear user input to verify clarification behavior). Use the systematic testing approach in [Evaluation and testing](#evaluation-and-testing). ### [](#design-for-cost-control)Design for cost control Write clear system prompts that reduce wasted iterations. Vague prompts cause agent confusion and unnecessary tool calls. Each wasted iteration costs tokens. Guide agents to: - Request only needed data from tools (use pagination, filters) - Avoid redundant tool calls (check context before calling) - Stop when the task completes (don’t continue exploring) For cost management strategies including iteration limits and monitoring, see [Understand Agent Concepts](https://docs.redpanda.com/agentic-data-plane/connect/concepts/). ## [](#example-system-prompt-with-best-practices)Example: System prompt with best practices This complete example demonstrates all the patterns described in this guide: ```text You are an order analytics agent for Acme E-commerce. Responsibilities: - Answer questions about customer order trends - Analyze order data from Redpanda topics - Provide insights on order patterns Available tools: - get_customer_orders: Retrieve order history (parameters: customer_id, start_date, end_date) - analyze_recent_orders: Compute order statistics (parameters: time_window, group_by) When to use tools: - Use get_customer_orders for individual customer queries - Use analyze_recent_orders for trend analysis across multiple orders Never: - Expose customer payment information or addresses - Analyze data older than 90 days unless explicitly requested - Make business recommendations without data to support them Data access: - Only orders from the authenticated customer account - Maximum of 90 days of historical data Response guidelines: - Present structured data as tables - Always state the analysis time window - Include order counts in trend summaries - If data is unavailable, explain the limitation When request is unclear: - Ask which time period to analyze - Confirm whether to include canceled orders - Do not assume customer intent ``` ## [](#common-anti-patterns)Common anti-patterns Avoid these patterns that lead to unpredictable agent behavior. ### [](#vague-role-definition)Vague role definition Define specific agent responsibilities and scope. Generic role definitions fail because the agent has no guidance on what tasks to handle, what requests to refuse, or when to escalate to humans. Don’t ```text You are a helpful AI assistant. ``` This doesn’t constrain behavior or set expectations. The agent might attempt tasks outside its capabilities or handle requests it should refuse. Do ```text You are an order fulfillment agent for an e-commerce platform. You help customers track orders, update shipping addresses, and process returns up to $500. Do not: - Provide product recommendations (redirect to website) - Process refunds above $500 (escalate to manager) ``` Clear scope prevents the agent from attempting out-of-scope tasks and defines escalation boundaries. ### [](#missing-constraints)Missing constraints Set explicit boundaries on data access and operations. Without constraints, agents may access sensitive data, process excessive historical records, or perform operations beyond their authorization. Don’t ```text You can access customer data to help answer questions. ``` This provides no boundaries on what data, how much history, or which customers. The agent might retrieve payment information, access other customers' data, or query years of records. Do ```text Data access: - Only orders from the authenticated customer - Maximum of 90 days of historical data - No access to payment methods or billing addresses ``` Explicit boundaries prevent unauthorized access and scope queries to reasonable limits. ### [](#implicit-tool-selection)Implicit tool selection Specify when to use each tool with clear decision criteria. Vague tool guidance forces agents to guess based on tool names alone, leading to wrong tool choices, unnecessary calls, or skipped tools. Don’t ```text Use the available tools to complete tasks. ``` The agent must guess which tool applies when. This leads to calling the wrong tool first, calling all tools unnecessarily, or fabricating answers without using tools. Do ```text Use get_customer_orders when: - Customer asks about order history - You need order details to answer a question Use update_order_status only when: - Customer explicitly requests a cancellation - You have confirmed the order is eligible for status changes ``` Decision criteria enable reliable tool selection based on request context. ## [](#next-steps)Next steps - [Agentic Data Plane Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) --- # Page 45: Test an MCP Server’s Tools with the Inspector **URL**: https://docs.redpanda.com/agentic-data-plane/connect/test-tools.md --- # Test an MCP Server’s Tools with the Inspector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Test an MCP Server’s Tools with the Inspector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: test-tools page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: test-tools.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/test-tools.adoc description: Use the Inspector tab to call tools, inspect resources and prompts, and verify your MCP server works without standing up an agent. page-topic-type: how-to personas: agent_builder learning-objective-1: Run an MCP tool from the Inspector and read its response learning-objective-2: Inspect a server's resources, prompts, and session activity learning-objective-3: Diagnose common errors (missing authentication, scope upgrade required, transport mismatch) before pointing an agent at the server page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- Verify that your MCP server’s [tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#tool), [resources](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#resource), and [prompts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#prompt) work correctly before agents depend on them. The Inspector runs on the same JSON-RPC connection agents use, so a passing test means fewer failed tool calls in production. Use the Inspector after creating a server or whenever you change a tool’s schema. After completing this guide, you will be able to: - Run an MCP tool from the Inspector and read its response - Inspect a server’s resources, prompts, and session activity - Diagnose common errors (missing authentication, scope upgrade required, transport mismatch) before pointing an agent at the server ## [](#when-to-use-the-inspector)When to use the Inspector - After creating or registering a server, to confirm tool discovery completed and tools run correctly. - While iterating on a self-managed server, to test schema changes against a live connection. - When debugging an agent failure, to isolate whether the issue is in the agent or in the MCP server itself. ## [](#open-the-inspector)Open the Inspector 1. Open **MCP Servers** in the sidebar and click into your server. 2. Switch to the **Inspector** tab. The Inspector is a console: a tool list and a tool runner sit side by side, with a status strip along the top. Resource and prompt panels appear underneath only when the server exposes those capabilities. Two more surfaces open from the status strip: **Session details** and the **Protocol panel**. See [Review session and protocol details](#session) for both. ![MCP Inspector flow. The Inspector identifies the server’s MCP version and capabilities, discovers tools, resources, and prompts, sends a selected tool call as JSON-RPC, and shows the formatted result beside the wire frames, timing, progress, and errors.](https://docs.redpanda.com/agentic-data-plane/connect/_images/mcp-inspector-lifecycle.svg) Figure 1. The MCP Inspector follows one request from discovery through the protocol view ## [](#run-a-tool)Run a tool The tool list shows every tool the server returned from `tools/list`. When a server exposes more than one tool, a filter box narrows the list by tool name, title, or description, and shows a count of how many tools match. Long tool names wrap in full instead of being cut off. Select a tool to load it into the runner. 1. Select a tool from the list. The runner shows the tool’s input and, after you run it, its response. 2. Provide the tool’s input using one of the input modes: - **Form**: A field-by-field form generated from the tool’s input JSON schema. - **JSON**: The raw JSON input, for pasting or editing a whole object at once. - **cURL**: A ready-to-run `curl` command for the call, with a copy action. 3. Click **Run tool**, or press Ctrl+Enter (Cmd+Enter on macOS). While the call is in flight, a progress bar reflects any progress the server reports, and you can **Cancel** it. The response appears in a tabbed result card: - **Result**: The tool’s output. Toggle between **Formatted** (pretty-printed JSON) and **Raw** (the exact response text). - **Raw JSON-RPC**: The reconstructed request and response envelopes, with the method and parameters exactly as the server sees them. - **Output schema**: The tool’s declared output schema, shown when the tool provides one. A status indicator reports whether the call succeeded and how long it took. Copy actions let you copy the input, the output, or the JSON-RPC envelope. If the tool returns an error, the result card shows the structured error detail. See [Common errors](#errors) for what each common error means. ### [](#code-mode-tools)Code-mode tools If the server has **Code mode** enabled, the tool list also includes `{name}_search` and `{name}_execute` alongside the server’s regular tools. These are the helpers an agent uses to discover and orchestrate tools through generated Python or JavaScript. Test them like any other tool. ## [](#inspect-resources)Inspect resources When the server exposes resources through `resources/list`, a **Resources** panel appears below the runner. Many MCP servers don’t expose resources at all: if there’s no panel, that’s fine. If your server does expose resources: 1. Click a resource to load its content into the pane. 2. The Inspector renders text, JSON, and binary types differently; for a binary payload, it reports the size and tells you to decode the content with a client SDK. ## [](#inspect-prompts)Inspect prompts When the server exposes prompt templates through `prompts/list`, a **Prompts** panel appears below the runner. Prompt templates are parameterized prompts an agent can invoke by name. As with resources, many servers don’t expose prompts. If your server does expose prompts: 1. Click a prompt to render its template. 2. Fill in any required parameters in the rendered form. 3. Click **Run** to see the rendered prompt content. ## [](#session)Review session and protocol details The status strip along the top of the console shows the connection status and how many tool calls you’ve made this session, along with two controls: **Session details** opens a side panel with the session’s connection health, negotiated capabilities, and protocol version. Use it to confirm the Inspector connected and to see which capabilities (tools, resources, prompts) the server advertised. **Protocol panel** opens a panel next to the runner: - **Wire**: The JSON-RPC request and result frames for your calls, any notifications the server sends, and a timing breakdown of the most recent call. - **Timeline**: A running log of the calls you’ve made this session, each with its result and latency. - **Logs**: Log lines the server sends over MCP logging. A telemetry strip reports the last call’s latency, time to first byte, payload size, and estimated context tokens. Click **Clear** to empty these views. ## [](#errors)Common errors | Error | Meaning and fix | | --- | --- | | OAuthConnectionRequired | User-delegated auth has no stored token for the calling user. Redpanda includes an authorize_url in the error detail; complete the consent flow per User-delegated OAuth. | | OAuthTokenExpired | Stored token is expired and refresh failed. Re-consent through My Connections. | | scope_upgrade_required | The user’s connection lacks one of the server’s required_scopes. Re-consent with the higher scope. | | Transport / connection error | The Inspector can’t reach the upstream. For self-managed servers, check the Connection tab on the detail page; verify the URL and transport choice. | | Tool input validation error | The form’s input doesn’t match the tool’s schema. The error message includes the offending field. Update and re-run. | ## [](#test-from-the-cli)Test from the CLI Use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) for the same tool calls outside the UI, when scripting smoke tests or running checks from CI. ```bash # List every tool exposed by a server rpk ai mcp-server tools list # Call a tool with a JSON arg blob rpk ai mcp-server tools call --input '{"arg1":"value"}' # Get server detail; includes the tool list by default. Add --no-tools # to skip discovery (faster when you only want metadata). rpk ai mcp-server get ``` The command targets the AI Gateway of your selected Agentic Data Plane environment, using credentials from `rpk ai auth login`. See [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) for installation and sign-in. ## [](#next-steps)Next steps - [Code Mode](https://docs.redpanda.com/agentic-data-plane/gateway/code-mode/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 46: Connect an Agent to Microsoft Teams **URL**: https://docs.redpanda.com/agentic-data-plane/connect/triggers/microsoft-teams.md --- # Connect an Agent to Microsoft Teams > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect an Agent to Microsoft Teams latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: triggers/microsoft-teams page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: triggers/microsoft-teams.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/triggers/microsoft-teams.adoc description: Add a Microsoft Teams trigger to a deployed agent so people in your organization can chat with it in a personal Teams chat. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Register a Microsoft Teams bot and configure its Azure app registration for single-tenant access learning-objective-2: Add a Microsoft Teams trigger to an agent and copy its messaging endpoint learning-objective-3: Publish the bot to your organization and verify the agent responds in a Teams chat page-git-created-date: "2026-06-02" page-git-modified-date: "2026-08-10" --- A Microsoft Teams trigger connects a deployed agent to Microsoft Teams so people in your organization can chat with it in a personal Teams chat. Replies stream back into the chat as the agent produces them, and the agent receives the message sender’s verified email address, so it can identify the user. The setup spans three systems: the Microsoft Teams Developer Portal hosts the bot and the Teams app, the Azure portal holds the bot’s authentication settings and permissions, and Agentic Data Plane holds the agent’s trigger. After reading this page, you will be able to: - Register a Microsoft Teams bot and configure its Azure app registration for single-tenant access - Add a Microsoft Teams trigger to an agent and copy its messaging endpoint - Publish the bot to your organization and verify the agent responds in a Teams chat ## [](#prerequisites)Prerequisites - A deployed agent. If you don’t have one, see [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). - Microsoft Entra admin access for your organization. Granting API permission consent and approving the Teams app require an admin. - Access to the [Teams Developer Portal](https://dev.teams.microsoft.com) and the [Teams admin center](https://admin.teams.microsoft.com). > 📝 **NOTE** > > Microsoft and Redpanda each need a value from the other: Redpanda needs the bot credentials before it can show the messaging endpoint URL, and the bot needs that URL to finish its setup. In this guide, you create the bot first without an endpoint, add the trigger in Redpanda to get the URL, and then return to Microsoft to finish. > > The Microsoft steps refer to a third-party product. Specific UI elements may not be up to date. For current instructions, see the [Microsoft Teams Developer Portal documentation](https://learn.microsoft.com/en-us/microsoftteams/platform/concepts/build-and-test/teams-developer-portal). ## [](#create-a-bot-in-the-teams-developer-portal)Create a bot in the Teams Developer Portal 1. Go to the [Teams Developer Portal](https://dev.teams.microsoft.com/tools) and open **Tools** > **Bot management**. 2. Click **New bot**, enter a name, and create it. ![The Bot management page in the Teams Developer Portal](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-devportal-bots.png) 3. Leave the endpoint address blank for now. You get the URL from Redpanda later. ## [](#generate-a-client-secret)Generate a client secret In the bot’s settings: 1. Click **Client secrets**. 2. Click **New client secret**. 3. Copy the generated secret value and store it securely. The value is shown only when you create it. ![The Client secret created dialog in the Teams Developer Portal](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-devportal-secret.png) You now have two values from the Teams side: the bot ID, shown in the bot list, and the client secret you just generated. The bot ID and the application (client) ID of the bot’s app registration are the same value. > 💡 **TIP** > > If you can’t create another client secret in the Teams Developer Portal, open the bot’s app registration in the Azure portal and create one under **Certificates & secrets** instead. ## [](#configure-the-app-registration-in-azure)Configure the app registration in Azure Creating a bot in the Teams Developer Portal also creates a matching app registration in Microsoft Entra. Adjust its settings. ### [](#find-the-app-registration)Find the app registration 1. In the [Azure portal](https://portal.azure.com), search for **App registrations**. 2. Open the **Owned applications** tab and select your bot’s app registration. 3. On the **Overview** page, note the application (client) ID and the directory (tenant) ID. You need both when you add the trigger. ![The Essentials pane of an app registration in the Azure portal](https://docs.redpanda.com/agentic-data-plane/connect/_images/azure-app-overview.png) ### [](#set-the-account-type-to-single-tenant)Set the account type to single-tenant > ❗ **IMPORTANT** > > The app registration must be single-tenant. A multi-tenant configuration causes 401 authorization errors on every inbound message. If the bot stops responding after setup, check this first. Changes can take up to 60 minutes to propagate. 1. In the app registration, open **Authentication**. 2. On the **Supported accounts** tab, select the single-tenant option for the supported account types. Depending on the portal experience, it is labeled **Single tenant only** or **Accounts in this organizational directory only**. ![The Supported accounts tab of the app registration’s Authentication page in the Azure portal](https://docs.redpanda.com/agentic-data-plane/connect/_images/azure-supported-accounts.png) 3. Save. ### [](#add-microsoft-graph-permissions)Add Microsoft Graph permissions The trigger uses the bot’s own credentials to look up the message sender’s email address in Microsoft Graph and passes it to the agent as the user identity. This lookup runs as a background service, so it requires application permissions, not delegated permissions. 1. In the app registration, open **API permissions**. 2. Click **Add a permission** and select **Microsoft Graph**. 3. Select **Application permissions**. 4. Add each of these permissions: | Permission | Purpose | | --- | --- | | User.ReadBasic.All | Resolve the sender’s email address from their Microsoft Entra object ID. | | Chat.Read.All | Read chat messages, as required by Teams. | | Chat.ReadBasic.All | Read chat metadata, as required by Teams. | 5. Click **Grant admin consent**. The button includes your organization’s name. The status of each permission changes from **Not granted** to **Granted**. ![The Configured permissions table on the app registration’s API permissions page](https://docs.redpanda.com/agentic-data-plane/connect/_images/azure-api-permissions.png) > 📝 **NOTE** > > Granting admin consent requires a Microsoft Entra admin role, such as Global Administrator or Privileged Role Administrator. If the button is unavailable, ask an admin to grant consent. ## [](#add-the-microsoft-teams-trigger-to-your-agent)Add the Microsoft Teams trigger to your agent 1. Open your agent’s **Triggers** tab. 2. Click **Add trigger** and select **Microsoft Teams**. 3. Fill in the trigger configuration: | Field | Value | | --- | --- | | Display name | A name for this trigger, for example, Production bot. The trigger’s unique ID is derived from it at creation and can’t be changed afterward. | | Description | Optional. A short description of the trigger, shown in the agent’s trigger list. | | Application (client) ID | The application (client) ID from the app registration Overview page. | | Directory (tenant) ID | The directory (tenant) ID from the app registration Overview page. | | Client secret | A reference to a secret in the secret store that holds the client secret you generated. Select Existing to pick one, or New to name the secret, paste the value, and click Create secret. Only the reference is kept on the trigger. | ![The Add trigger page with Microsoft Teams selected, a display name and description entered, the application and tenant IDs filled in, and the New client secret fields showing a secret name and masked value](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-trigger-form.png) 4. Click **Create trigger**. The new trigger appears on the **Triggers** tab with a **Pending** status. ![A collapsed Microsoft Teams trigger card named Production bot with a Pending status](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-trigger-added.png) ## [](#copy-the-messaging-endpoint)Copy the messaging endpoint Redpanda’s messaging bridge picks up the new trigger, validates the bot credentials with Microsoft, and reports the trigger’s messaging endpoint URL. This takes up to a minute, and the **Triggers** tab refreshes on its own. 1. On the **Triggers** tab, wait for the status to change from **Pending** to **Connected**. The card expands automatically and the **Messaging endpoint** field shows the URL, similar to: ```text https://msteams-bridge..clusters.rdpa.co/teams/v1/ ``` In this URL, `` is the ID of the cluster the agent runs on, and `` is the agent’s ID. 2. Copy the URL exactly as shown. You need it in the next step. ![An expanded Microsoft Teams trigger card with a green Connected badge, showing the application client ID, directory tenant ID, client secret reference, and the messaging endpoint URL with a copy button](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-trigger-connected.png) If the status changes to **Error** instead, the message on the card explains the reason. See [Troubleshooting](#troubleshooting). ## [](#set-the-endpoint-url-on-the-bot)Set the endpoint URL on the bot 1. Go back to the [Teams Developer Portal](https://dev.teams.microsoft.com/tools). 2. Open **Tools** > **Bot management** and select your bot. 3. Paste the messaging endpoint URL into the endpoint address field. ![The bot’s Configure page in the Teams Developer Portal](https://docs.redpanda.com/agentic-data-plane/connect/_images/teams-devportal-endpoint.png) 4. Save. If you manage the bot as an Azure Bot resource instead, paste the URL into the **Messaging endpoint** field on the bot’s **Configuration** page in the Azure portal. ## [](#create-and-publish-the-teams-app)Create and publish the Teams app A Teams app makes the bot visible to people in your organization. 1. In the Teams Developer Portal, click **Apps**, and then create a new app and give it a name. 2. On the **Basic information** tab, fill in the descriptions, developer information, and the application (client) ID from the app registration. 3. Open **App features**, select **Bot**, and choose the bot you created. Under the scopes where people can use the bot, select **Personal**. With only this scope selected, people use the bot in personal chats and can’t add it to channels or group chats, which matches what the trigger is designed for. 4. Click **Publish** > **Publish to your org**. This submits the app for admin approval. ## [](#approve-the-app-in-the-teams-admin-center)Approve the app in the Teams admin center 1. Go to the [Teams admin center](https://admin.teams.microsoft.com). 2. Open **Teams apps** > **Manage apps** and search for your app. 3. Select the app, then click **Publish** to approve it. The app can take up to 24 hours to appear in the Teams app catalog after approval. > 💡 **TIP** > > By default, the app is available to everyone in the organization. To restrict it, assign it to specific users or groups under **Manage apps**. To pin it to the Teams sidebar, add it to a setup policy under **Setup policies**. ## [](#verify)Verify 1. Open the agent’s **Triggers** tab and confirm the trigger status is **Connected**. This confirms that Redpanda validated the bot credentials with Microsoft. It does not confirm that the endpoint URL is set correctly on the bot, so also send a real message. 2. In Microsoft Teams, find the app in your chats and send it a message. While the agent works, Teams shows a progress indicator, and then the reply streams into the chat with an AI-generated label. Tool calls appear as cards that update in place as they finish. ## [](#how-a-message-flows)How a message flows When someone messages the bot, Teams delivers the message to the trigger’s messaging endpoint. Redpanda’s messaging bridge authenticates the request against the trigger’s bot credentials, resolves the sender’s email address through Microsoft Graph, and forwards the message to the agent along with that identity. The reply streams back into the chat as the agent produces it. A reply that exceeds the Teams streaming limits finishes as a regular message. An agent can have several Microsoft Teams triggers, each with its own bot registration and credentials. For example, you can expose one agent through a production bot and a test bot at the same time. Inbound messages are matched to the trigger whose bot they were addressed to. Each trigger’s health is rechecked continuously: about every 30 seconds, the bridge validates the trigger’s credentials with Microsoft and updates the status on the **Triggers** tab. If a client secret expires or is rotated to a wrong value, the trigger flips to **Error** with the reason, without anyone sending a message. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | The bot never responds, or replies with a 401 error | The app registration is most likely set to multi-tenant. In the Azure portal, open the app registration, click Authentication, and select the single-tenant option for the supported account types. Allow up to 60 minutes for the change to propagate. | | The trigger shows Error with a message that the client secret is not set in the secret store | The messaging bridge could not read the secret from this cluster’s secret store. In Secrets Store, confirm that a secret exists whose ID exactly matches the reference on the trigger. If it does and the error persists, contact Redpanda support. | | The trigger shows Error with a message that Microsoft rejected the bot credentials | The application (client) ID, directory (tenant) ID, and client secret don’t match a valid app registration. The Microsoft error code in the message names the exact cause. For example, AADSTS7000215 means the client secret value is invalid. | | Teams shows a Permissions needed message | A Teams admin has not yet approved the app. Approve it in the Teams admin center. | | The Grant admin consent button is unavailable | The signed-in Azure user lacks admin privileges. Ask a Global Administrator to grant consent on the API permissions page. | | The agent responds but doesn’t receive the sender’s email address | The User.ReadBasic.All Graph permission is missing, or admin consent was not granted. Recheck the Microsoft Graph permissions step. | | The trigger status stays on Pending | The messaging bridge has not reported on the trigger yet. The first report normally arrives within a minute. If it doesn’t, refresh the tab; if the status still doesn’t change, contact Redpanda support. | ## [](#next-steps)Next steps - [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) - [How Agents Work](https://docs.redpanda.com/agentic-data-plane/connect/agents-overview/) --- # Page 47: Trigger Agents **URL**: https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview.md --- # Trigger Agents > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Trigger Agents latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: triggers/overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: triggers/overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/triggers/overview.adoc description: Triggers invoke a deployed agent without calling its API directly, from Microsoft Teams or on a schedule. page-topic-type: overview personas: agent_builder, platform_engineer learning-objective-1: Describe what an agent trigger is and how a trigger invokes an agent learning-objective-2: Describe how triggers are added, edited, and disconnected on an agent learning-objective-3: Interpret the status that an agent's Triggers tab reports for each trigger page-git-created-date: "2026-06-02" page-git-modified-date: "2026-08-03" --- A trigger is a configured way to invoke an agent without calling its API directly. Without a trigger, an agent only responds to direct calls against its API. Two trigger types are available: - A Microsoft Teams trigger connects the agent to Microsoft Teams. People in your organization chat with the agent in a personal Teams chat, and replies stream back as the agent produces them. See [Connect an Agent to Microsoft Teams](https://docs.redpanda.com/agentic-data-plane/connect/triggers/microsoft-teams/). - A schedule trigger runs the agent automatically on a recurring schedule. Each run sends a fixed message to the agent and is recorded as a transcript, with no reply path back. See [Run an Agent on a Schedule](https://docs.redpanda.com/agentic-data-plane/connect/triggers/schedule/). Each trigger is its own resource on the agent, created, edited, and disconnected on its own, so you can attach several to the same agent. For example, one agent can serve a production Teams bot and a test Teams bot side by side, each through its own trigger. > 📝 **NOTE** > > Triggers are available for Redpanda-managed agents only. A self-managed agent runs on your own infrastructure and is invoked through it, so it has no **Triggers** tab. To bring a self-managed agent into Agentic Data Plane for observability and governance, see [Set up a self-managed agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/). After reading this page, you will be able to: - Describe what an agent trigger is and how a trigger invokes an agent - Describe how triggers are added, edited, and disconnected on an agent - Interpret the status that an agent’s Triggers tab reports for each trigger ## [](#manage-triggers)Manage triggers Triggers live on the agent’s detail page, on the **Triggers** tab: - Click **Add trigger**, pick the trigger type, and fill in its configuration. A Microsoft Teams trigger takes the bot’s application (client) ID, directory (tenant) ID, and a secret store reference to the bot’s client secret. For the full setup, including the Microsoft side, see [Connect an Agent to Microsoft Teams](https://docs.redpanda.com/agentic-data-plane/connect/triggers/microsoft-teams/). - To change a trigger’s configuration, edit it on its card. The trigger’s unique ID, derived from its display name at creation, and the trigger’s type are fixed; the display name, description, and type-specific configuration stay editable. - To remove a trigger, disconnect or delete it on its card, depending on its type. Removing it stops the agent from being invoked through that trigger and can’t be undone. A Microsoft Teams trigger also exposes a messaging endpoint: a URL that Teams delivers inbound messages to. Redpanda reports this URL on the trigger’s card shortly after you add the trigger, and you register it with your bot so Teams knows where to send messages. ## [](#trigger-status)Trigger status Adding a trigger does not validate it on the spot. For a Microsoft Teams trigger, Redpanda’s messaging bridge checks each trigger in the background, about every 30 seconds, by validating its credentials with Microsoft, and reports the result. The **Triggers** tab shows one of three states: - Pending: No check has been reported yet. A trigger sits here briefly after you add it. - Connected: The latest check passed. For a Microsoft Teams trigger, this means Microsoft accepted the bot credentials. - Error: The latest check failed. The message on the card explains the reason, for example, a rejected client secret. Because the check runs continuously, the status also catches problems that appear later: if a bot’s client secret expires, the trigger flips to Error without anyone sending a message. The tab refreshes the status on its own, and you can also refresh on demand. A schedule trigger reports the outcome of its runs rather than a connection check. Its card shows the last run’s outcome and the next run time. For the outcomes a run can have, see [Run an Agent on a Schedule](https://docs.redpanda.com/agentic-data-plane/connect/triggers/schedule/). ## [](#next-steps)Next steps - [Connect an Agent to Microsoft Teams](https://docs.redpanda.com/agentic-data-plane/connect/triggers/microsoft-teams/) - [Run an Agent on a Schedule](https://docs.redpanda.com/agentic-data-plane/connect/triggers/schedule/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 48: Run an Agent on a Schedule **URL**: https://docs.redpanda.com/agentic-data-plane/connect/triggers/schedule.md --- # Run an Agent on a Schedule > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Run an Agent on a Schedule latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: triggers/schedule page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: triggers/schedule.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/triggers/schedule.adoc description: Add a schedule trigger to a deployed agent so it runs automatically on a recurring cron schedule, in the timezone you choose. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Add a schedule trigger to an agent using a preset or a cron expression learning-objective-2: Choose the timezone and input prompt for scheduled runs learning-objective-3: Describe how a scheduled run invokes the agent and who it runs as page-git-created-date: "2026-07-27" page-git-modified-date: "2026-08-03" --- A schedule trigger runs a deployed agent automatically on a recurring schedule that you define with a cron expression. Each run sends a fixed message, the input prompt, to the agent in the timezone you choose. A schedule trigger is one-directional. It starts the agent on time and records the run as a transcript. This differs from a Microsoft Teams trigger, where a person sends a message and the agent’s reply streams back to them. Each scheduled run invokes the agent on behalf of the person who created the trigger, with that person’s access, and the run is attributed to them in transcripts and cost reporting. Editing the schedule later doesn’t change who it runs as. > 📝 **NOTE** > > Triggers are available for Redpanda-managed agents only. A self-managed agent runs on your own infrastructure and is invoked through it, so it has no **Triggers** tab. To bring a self-managed agent into Agentic Data Plane for observability and governance, see [Set up a self-managed agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/). After reading this page, you will be able to: - Add a schedule trigger to an agent using a preset or a cron expression - Choose the timezone and input prompt for scheduled runs - Describe how a scheduled run invokes the agent and who it runs as ## [](#prerequisites)Prerequisites - A deployed Redpanda-managed agent. If you don’t have one, see [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). ## [](#add-a-schedule-trigger)Add a schedule trigger 1. In Redpanda Cloud, open your agent’s **Triggers** tab. 2. Click **Add trigger** and select **Schedule**. 3. Under Details, enter a display name, for example, `Daily summary`. The trigger’s unique ID is derived from it at creation and can’t be changed afterward. To describe the trigger for other people, add a description. 4. Set the schedule and timezone. See [Set the schedule](#set-the-schedule) and [Choose the timezone](#choose-the-timezone). 5. Under Input, enter an input prompt. This text is the message the agent receives on every run, for example, `Summarize new support tickets from the last 24 hours`. 6. Click **Create trigger**. The trigger is active as soon as you create it. An agent can have up to 10 schedule triggers, so you can run the same agent on several schedules, each with its own input prompt. ## [](#set-the-schedule)Set the schedule Under Schedule, set when the agent runs: - For a common cadence, pick a preset and set the time. The presets are hourly, daily, weekdays, weekly, and monthly. - For a schedule the presets don’t cover, expand the **Advanced** section and enter a standard five-field cron expression. A cron expression has five fields, separated by spaces: minute, hour, day of month, month, and day of week. For example, `0 9 * * 1-5` runs at 9:00 AM on weekdays. The readback below the field restates the schedule in plain language and previews the next run, so you can confirm the expression does what you intend. Two limits apply to the schedule: - Runs must be at least 15 minutes apart. A schedule whose runs fall closer together than that is rejected with the message that the schedule runs too frequently. - Shortcut expressions such as `@daily` and `@hourly` are not supported. Use a five-field expression instead. ## [](#choose-the-timezone)Choose the timezone Select a timezone. The schedule is evaluated in this timezone, and it accounts for daylight saving, so a run set for 9:00 AM stays at 9:00 AM across the change. The field defaults to your browser’s timezone, and you can change it to any IANA timezone, for example, `Europe/Prague` or `America/New_York`. The next run time shown on the form and on the trigger’s card is a preview, computed in the schedule’s timezone. Redpanda remains the authority on the exact time each run fires. ## [](#how-a-scheduled-run-works)How a scheduled run works At each scheduled time, Redpanda invokes the agent with the input prompt as a user message. The agent processes the message as it would a chat, and the work is recorded as a transcript. Because the run is one-directional, no reply is delivered anywhere. Only one run of a trigger is in flight at a time. Redpanda records each run in the trigger’s run history with an outcome: - Succeeded: Redpanda started the run and the agent accepted it. - Failed: The run could not be started, or the agent rejected it. The recorded error explains the reason. - Missed: The scheduled time passed during Redpanda downtime. Missed runs are not retried or caught up later. - Skipped: The previous run of the same trigger was still in progress at the scheduled time. To review past runs, expand the trigger’s card and click **View runs**. Each run is listed with its outcome, and a run that reached the agent produces a transcript, the same as any other agent invocation. ## [](#manage-a-schedule-trigger)Manage a schedule trigger Schedule triggers live on the agent’s **Triggers** tab, each on its own card: - To change a trigger, edit it on its card. The trigger’s unique ID and its type are fixed. The display name, description, schedule, timezone, and input prompt stay editable. - To review past runs, expand the card and click **View runs**. - To stop the schedule, delete the trigger on its card. Deleting stops all future runs and can’t be undone. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | The schedule is rejected as running too frequently | Two runs fall less than 15 minutes apart. Widen the schedule so consecutive runs are at least 15 minutes apart. | | A shortcut expression such as @daily is rejected | Shortcut expressions are not supported. Rewrite the schedule as a five-field cron expression, for example, 0 0 * * * for every day at midnight. | | Runs fire at an unexpected local time | Check the trigger’s timezone. The schedule is evaluated in the timezone set on the trigger, not the viewer’s local time, and it shifts with daylight saving. | | A run is recorded as Missed | The scheduled time passed while Redpanda was unavailable. Missed runs are not caught up. The next scheduled run proceeds as normal. | | A run is recorded as Skipped | The previous run of the same trigger was still in progress at the scheduled time. Give the agent more time between runs, or simplify the input prompt so each run finishes sooner. | ## [](#next-steps)Next steps - [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/) - [Connect an Agent to Microsoft Teams](https://docs.redpanda.com/agentic-data-plane/connect/triggers/microsoft-teams/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 49: Configure User-Delegated OAuth **URL**: https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth.md --- # Configure User-Delegated OAuth > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Configure User-Delegated OAuth latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: user-delegated-oauth page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: user-delegated-oauth.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/connect/pages/user-delegated-oauth.adoc description: Have each end-user authenticate against the MCP server's upstream system with their own credentials. Redpanda stores their token in the vault and injects it at call time. page-topic-type: how-to personas: platform_engineer, agent_builder, security_compliance_lead learning-objective-1: Configure an MCP server to use user-delegated OAuth, with either automatic setup or a provider you registered learning-objective-2: Walk an end-user through the consent flow and verify the connection learning-objective-3: Troubleshoot scope upgrades, token expiry, and refresh failures page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-12" --- User-delegated OAuth means each end-user authenticates against the MCP server’s upstream system (for example, Slack, Jira, Google) with their own credentials. Redpanda stores their token in the token vault and injects it at call time. Contrast with service-account OAuth, where one shared identity is used for every caller. For a self-managed server, Redpanda can set up the OAuth client for you from the server URL, or you can attach an OAuth provider your organization already registered. Managed servers always attach a provider. After completing this guide, you will be able to: - Configure an MCP server to use user-delegated OAuth, with either automatic setup or a provider you registered - Walk an end-user through the consent flow and verify the connection - Troubleshoot scope upgrades, token expiry, and refresh failures ## [](#prerequisites)Prerequisites - For a **managed** MCP server, or to attach a provider your organization already manages on a **self-managed** server: an OAuth provider resource configured on the **Outbound providers** tab of **Integrations setup**. The provider declares the upstream’s `authorize_url`, `token_url`, supported scopes, and client credentials. See [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/). - For automatic setup on a **self-managed** server: nothing, if the server supports dynamic client registration. If it doesn’t, you need an OAuth app registered with the upstream vendor, its client ID, and, for a confidential app, a client-secret reference already created in the secret store in `UPPER_SNAKE_CASE` (proto regex `^[A-Z][A-Z0-9_]*$`). - The scopes the upstream API needs, if you want to request a specific set rather than whatever the server advertises. - For **self-managed** MCP servers: the server URL must be `https://` (proto rule `remote_mcp.user_oauth_requires_https`). HTTP is rejected at create time. - For **managed** MCP servers: the type must support user-delegated OAuth. SQL doesn’t; Slack, Jira, and Google managed types do. Check [Managed catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) before configuring. ## [](#configure-the-server)Configure the server 1. Create your MCP server (see [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/)). 2. Set up the OAuth client for the upstream. What you do depends on the server type: - For a **self-managed** (remote) server, enter the server URL and let Redpanda check what it supports. For a server that authenticates users over OAuth, Redpanda recommends setting up OAuth automatically, which means Redpanda registers an OAuth client and manages each user’s tokens for you. If the server supports dynamic client registration, there is nothing else to configure. If it doesn’t, Redpanda walks you through the parts it can’t do for you: register an OAuth app in the upstream’s developer console, add the authorization callback URL that Redpanda displays to that app, then paste the app’s credentials back into the form. The credentials are the `Client ID` from the developer console and, for a confidential app, a client-secret reference from the secret store in `UPPER_SNAKE_CASE`, for example `GITHUB_CLIENT_SECRET`. Leave the reference empty for a public PKCE app. Either way, Redpanda records the provider it resolved or created for the server, marked with a **Discovered** badge in the **Outbound providers** list so you can tell it from a provider you registered by hand. See [Discover a provider from an MCP server URL](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/#discover-a-provider-from-an-mcp-server-url). To attach an OAuth provider your organization already manages instead, click **Use a different authentication method**, set `Method` to `User-delegated OAuth`, then set `Setup` to `Attach existing provider` and select the provider (`UserOAuthAuth.provider_name`). Click **New** to register a provider without leaving the form. The same `Setup` toggle switches back to `Set up automatically`. - For a **managed** server, select the OAuth provider in the type’s configuration (`UserOAuthAuth.provider_name`). Managed types always attach a provider, so automatic setup doesn’t apply. 3. (Optional) Restrict the scopes Redpanda requests. Expand **Set specific scopes** and list the scopes the upstream API needs (`UserOAuthAuth.required_scopes`). Redpanda enforces them at consent time. Leave it empty to request whatever the server advertises. 4. (Optional) Override token injection through the API or CLI. By default Redpanda sends `Authorization: Bearer `. To use a different header, set `TokenInjection.header_name`. To omit the prefix entirely (for example, an upstream that expects a bare API key as the token), set `TokenInjection.header_prefix` to the empty string. The create form doesn’t expose these fields. 5. Save. > 📝 **NOTE** > > Choosing user-delegated OAuth instead of service-account OAuth **is** the credential-mode decision: there’s no separate field. User-delegated gives each caller a per-user upstream identity; service-account gives every caller one shared identity. Switching between them later requires re-consent for every active user. > 📝 **NOTE** > > Editing an existing server works differently. The edit page doesn’t re-check the server URL, so you set `Method` and `Setup` yourself. Leaving the provider empty there keeps the provider the server already has, so switching an existing server to automatic setup is an explicit choice (`UserOAuthAuth.automatic_setup`): automatic setup re-runs discovery and can register a new upstream client. Setting both a provider name and automatic setup is rejected. > 💡 **TIP** > > From the CLI, `--user-oauth-provider` and `--user-oauth-scopes` on `rpk ai mcp-server create` or `rpk ai mcp-server update` attach a provider you already registered; the flags don’t request automatic setup. To script automatic setup, set `automatic_setup: true` under the server’s `user_oauth` block in a GitOps manifest. See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) and [Manage Resources with GitOps](https://docs.redpanda.com/agentic-data-plane/cli/gitops/). ## [](#the-user-connection-flow)The user connection flow The first time a user calls a tool that needs this server’s authentication, Redpanda doesn’t have a stored token for them. The behavior is: 1. The MCP RPC returns `FAILED_PRECONDITION` with an `OAuthConnectionRequired` error detail. The detail carries an `authorize_url`. 2. Agentic Data Plane surfaces a consent prompt to the user, pointing at the `authorize_url`. 3. The user completes the device-authorization flow with the upstream provider. 4. The upstream provider redirects back to Redpanda with a token. Redpanda stores it in the token vault under that user’s identity. 5. The original tool call retries automatically. Subsequent calls reuse the stored token. After consent, the user can see and revoke their connection under **My Connections** in Agentic Data Plane. ## [](#manage-your-connections)Manage your connections The **My Connections** page in the sidebar is each user’s view of their own third-party connections, so a user can authorize the providers their MCP servers need without waiting for a tool call to trigger the consent flow. It shows connections for your own identity only. The page uses a two-pane layout. The left rail lists providers grouped by status: - **Available to connect**: Providers you haven’t authorized yet. - **Connected**: Providers with a usable connection. - **Needs attention**: Connections that have failed, or whose refresh token is close to expiring, and need to be reconnected. Select a provider to open its detail pane on the right. The pane shows the granted scopes, a status badge (**Connected**, **Needs attention**, **Action required**, or **Not connected**), and the actions for that provider: **Connect** runs the consent flow and stores a token in the vault under your identity, **Reconnect** re-runs consent for an existing connection (for example, to grant additional scopes or recover after a refresh token expires), and **Disconnect** revokes it. A connection that the gateway can renew automatically from a stored refresh token stays **Connected** with no action from you; it moves to **Needs attention** or **Action required** only when it can no longer renew on its own. The pane also lists the MCP servers that use this connection, both managed servers run by Redpanda and remote servers you registered, and flags any scopes those servers require that the connection is missing. When a scope is missing, select **Reconnect** to grant it. Otherwise, the servers' tool calls fail even though the connection reads as **Connected**. > 📝 **NOTE** > > Connections are per-user. Each user authorizes their own accounts, and one user’s connections are not visible to or usable by another. You can also manage your connections from the terminal: run `rpk ai connection list` to see the services you are signed in to, and `rpk ai connection revoke ` to disconnect one. See [rpk ai connection](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-connection/). ## [](#scope-upgrades)Scope upgrades If a user’s stored connection has fewer scopes than the server’s `required_scopes`, the gateway returns a `scope_upgrade_required` error and surfaces a new `authorize_url` requesting the additional scopes. The user re-consents; the connection is updated in place. ## [](#refresh-and-expiry)Refresh and expiry Redpanda transparently refreshes tokens before they expire, using the refresh token returned at consent time. A connection is only considered expired when its access token has lapsed **and** the gateway can no longer renew it automatically. As long as a usable refresh token is stored, the connection stays active across access-token expiry, and the gateway renews access silently on the next call. This matters for upstreams that issue short-lived access tokens with long-lived refresh tokens: a GitHub App, for example, issues 8-hour access tokens backed by refresh tokens that last around six months, so an idle connection stays active for months rather than showing as expired within hours. A connection is reported as expired, meaning the user must reconnect, only when the access token has lapsed and no usable refresh token exists: either none was stored, or the refresh token itself has expired. When that happens, the next tool call returns `OAuthTokenExpired` with a `reason`, and the user must re-consent through the same flow as the initial connection. Each connection reports whether a refresh token backs it, so a client can tell a connection that renews automatically from one that needs reconnecting. The `OAuthConnectionService.ListConnections` and `GetConnection` responses expose this on the `Connection` message through two read-only fields: `refresh_token_present` (a refresh token is stored, so the gateway renews access automatically) and `refresh_token_expires_at` (when the refresh token itself expires, if the upstream reports it, as GitHub Apps do; absent when unknown or non-expiring). ## [](#service-account-oauth-contrast)Service-account OAuth contrast If you want one shared upstream identity for every caller (instead of per-user identities), choose **Service-account OAuth** on the server instead of **User-delegated OAuth**. With service-account OAuth, every caller of every tool sees the same upstream identity; the upstream system has no idea which Agentic Data Plane user invoked the tool. With user-delegated OAuth, the upstream system sees each end-user as themselves and applies their own permissions. For the field-by-field service-account-OAuth setup, see [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/#configure-authentication). ## [](#worked-examples)Worked examples - [Slack](https://docs.redpanda.com/agentic-data-plane/connect/managed/slack/): consumer-facing user-delegated OAuth example. Shows the consent flow against a real Slack workspace. - [Jira](https://docs.redpanda.com/agentic-data-plane/connect/managed/jira/): enterprise user-delegated OAuth example. Atlassian’s OAuth flow differs from Slack’s; this page calls out scope-management gotchas. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | "OAuth provider not found" | The provider name on the server doesn’t match an OAuth provider in Agentic Data Plane. Check spelling and that the provider exists. | | "HTTPS required" on save (self-managed only) | User-delegated OAuth requires https:// URLs on the MCP server (proto rule remote_mcp.user_oauth_requires_https). Switch the server’s URL to HTTPS. | | OAuthConnectionRequired returned even after the user consented | The user’s vault entry might have been revoked or the token expired with no refresh. Have the user re-consent through My Connections. | | scope_upgrade_required returned | The required scopes on the server changed (or the user originally consented with fewer scopes). The user re-consents with the higher scope. | | My Connections shows stale entries | Connections persist until manually revoked. Have the user disconnect and reconnect the connection if upstream credentials changed. | ## [](#next-steps)Next steps - [Configure an OAuth Provider](https://docs.redpanda.com/agentic-data-plane/connect/oauth-providers/) - [Service-account OAuth setup](https://docs.redpanda.com/agentic-data-plane/connect/create-server/#configure-authentication) --- # Page 50: Control & Govern **URL**: https://docs.redpanda.com/agentic-data-plane/control.md --- # Control & Govern > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Control & Govern latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/index.adoc description: Govern agent activity with guardrails, spend limits, and permissions. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-06-10" --- Govern agent activity across your deployment. Set guardrails and spend limits, and control who can do what with roles and permissions. - [Set Up Guardrails](guardrails/) Understand how guardrails work, create them, review blocked requests, and look up the available policies. - [Analyze Cost and Usage](cost-usage/) Break down LLM spend, requests, and tokens by provider, model, token type, agent, user, or tag. Compare periods, export CSV reports, and see which agents are active right now. - [Set Up Budgets](budgets/) Cap LLM spend with per-agent budgets, and see what the Agentic Data Plane records automatically and where to view it. - [Track Spend by Tag](cost-allocation-tags/) Group and filter LLM spend by the tags you set on agents, so you can attribute cost to a department, team, or environment for chargeback and reporting. - [Manage Access](access/) Control who can do what in the Agentic Data Plane with access policies. --- # Page 51: Manage Access Policies **URL**: https://docs.redpanda.com/agentic-data-plane/control/access-policies.md --- # Manage Access Policies > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Manage Access Policies latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: access-policies page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: access-policies.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/access-policies.adoc description: Create Cedar-based access policies that permit or forbid specific actions in the Agentic Data Plane, with conditions over resource tags and ownership. page-topic-type: how-to personas: security_compliance_lead, platform_engineer learning-objective-1: Create an access policy that permits or forbids specific actions learning-objective-2: Write policy conditions that read tags and ownership safely learning-objective-3: Manage access policies from the CLI page-git-created-date: "2026-07-30" page-git-modified-date: "2026-08-07" --- Use access policies to express rules that roles can’t: deny a group access to production-tagged agents, or let only a resource’s creator modify one. Each policy permits or forbids specific actions, with conditions evaluated on every request. After reading this page, you will be able to: - Create an access policy that permits or forbids specific actions - Write policy conditions that read tags and ownership safely - Manage access policies from the CLI > 📝 **NOTE** > > Access policies are a preview capability, available when your organization has them enabled. If you don’t see **Access** in the sidebar, they aren’t enabled for your organization. ## [](#how-access-policies-work)How access policies work Access policies use the [Cedar policy language](https://www.cedarpolicy.com/). A policy is a `permit` or `forbid` statement over three parts: - **Principal**: Who the policy applies to. A user (`User::"[alice@example.com](mailto:alice@example.com)"`) or a group (`Group::"support"`). Principals don’t carry attributes such as department or nationality: model those facts as group membership. - **Action**: What operation the policy covers. An action ID is an entity type and a verb joined by a dot, for example `Action::"Agent.get"` or `Action::"McpServerTool.call"`. See [Action reference](#action-reference). - **Resource**: What the action targets, as a typed entity, for example `resource is Agent` for every agent or `resource == Agent::"support-bot"` for one. Most entity IDs are the resource’s bare name, the same one the API path uses. The policy resources are the exception: `Policy` and `PolicyTemplate` carry their collection prefix, as in `Policy::"policies/deny-prod-reads"`. Always pin the resource type. With a specific action ID such as `Action::"Agent.get"` the pin is redundant, because the action already names its one entity type, but it costs nothing. With an action group a `permit` requires it. Validation also uses the pin to catch conditions that can never match, so pinning is what turns a silent no-op into a save-time error. Every request resolves the same way: - A request is denied unless a `permit` matches it. - A matching `forbid` always wins. This is what makes cross-cutting denials possible: one `forbid` can override any number of grants. Enforcement is fail-closed. A `forbid` that errors during evaluation denies the request rather than being skipped. Policy changes also reach the enforcement points asynchronously, so a successful save is not yet an enforcement guarantee. ### [](#how-roles-compose-with-policies)How roles compose with policies Use roles to get started and to stay coarse: provision the first user, and hand out the Admin role where per-resource precision isn’t the point. Redpanda turns those role bindings into permits for you, so they take part in the same evaluation and you never author them by hand. Among the built-in roles, only Admin carries Agentic Data Plane permissions, so role-based access control (RBAC) gives you one coarse grant and nothing in between. Writer and Reader reach the control plane, Kafka, pipelines, and knowledge bases, but no agent, MCP server, or LLM provider. Policies are how you grant everyone who shouldn’t be an Admin. See [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/). From there, authorization defers to policy evaluation. Every decision that needs a condition, a specific resource, or a denial belongs in a policy, and a `forbid` overrides whatever a role granted. ## [](#action-reference)Action reference Verbs are shared across entity types, and an action ID pairs one verb with one entity. This table lists the verbs each entity type supports, by entity type in alphabetical order. Combine them with a dot to get the action ID: `Budget` plus `update` is `Action::"Budget.update"`. | Entity type | Verbs | | --- | --- | | AdminConnection | list, revoke, rotate | | Agent | get, list, create, update, delete, invoke | | AgentCredential | list, create, delete | | AgentNetwork | get | | AgentTrigger | get, list, create, update, delete | | AgentTriggerRun | list | | AuditLog | list | | Budget | get, list, create, update, delete | | CIMDSettings | get, update | | DCRSettings | get, update | | EffectivePolicySet | get, list | | Guardrail | get, list, create, update, delete | | LLMProvider | get, list, create, update, delete, invoke | | McpServer | get, list, create, update, delete, initialize, ping, tools_list, prompts_get, prompts_list, resources_list, resources_read, resources_templates_list, logging_set_level | | McpServerTool | call | | OAuthClient | get, list, create, update, delete | | OAuthConnection | get, list, authorize, revoke | | OAuthProvider | get, list, create, update, delete, attach | | Policy | get, list, create, update, delete | | PolicyTemplate | get, list, create, update, delete | | Session | get, list, delete | | Spending | get | | SystemPolicy | list | | Transcript | get, list | The `SystemPolicy.list` action gates reading the permits derived from role bindings. That view names every binding’s principal, so it sits on its own action rather than riding along with `Policy.list`. Entity type names are case-sensitive, and the casing doesn’t always follow the prose (`McpServer`, not `MCPServer`). Copy them from this table. Access policies cover the entity types listed here and nothing else. A policy can’t govern anything outside the table, including Redpanda Connect pipelines and knowledge bases. Copy action IDs from this table rather than deriving them from a resource or API name, because the two don’t line up predictably. ### [](#action-groups)Action groups Each verb also exists as an action group, so `action in Action::"get"` matches every `.get` action. > ❗ **IMPORTANT** > > A `permit` that references an action group must pin the resource type with `resource is ` or `resource == ::""`. Saving an unpinned group permit fails, because group membership grows as new resource types adopt the verb, so the stored grant would silently widen on the next deployment. Writing `resource in ::""` doesn’t count as a pin: it bounds the subtree but not the type, so the grant would still widen when a descendant type adopts the verb. `forbid` statements are exempt: a growing deny widens in the safe direction. Use action groups in `forbid`, where one clause can cover every type that adopts a verb. In a `permit` the required type pin narrows the group back down to the actions of that one type, so naming the actions is equivalent and clearer. To grant a verb across every type, name each `.` action explicitly. The Access page does this for you when you pick verbs with the All resources scope. ## [](#attribute-reference)Attribute reference Conditions read the attributes in this table. Every one is optional at the schema level, so each needs a `has` guard before you read it. An unguarded read is an evaluation error, and in a `forbid` that error denies. | Attribute | Entity types that carry it | Required guard | | --- | --- | --- | | Tags | Agent, Budget, Guardrail, LLMProvider, McpServer, OAuthClient, OAuthProvider, Policy | resource.hasTag("") | | created_by, updated_by | Agent, AgentTrigger, Budget, Guardrail, LLMProvider, McpServer, OAuthClient, OAuthProvider, Policy, PolicyTemplate | resource has created_by | | changed_tags | Agent, Budget, Guardrail, LLMProvider, McpServer, OAuthClient, OAuthProvider, on update only | resource has changed_tags | Put the guard first in the same boolean expression, then read the value. [Write conditions](#write-conditions) shows the full form for each. Tags are the attribute half of attribute-based access control (ABAC). Not every type carries them, and the set doesn’t follow a rule you can infer, so check the table rather than assuming. The read-only and derived types carry neither tags nor a creator: sessions, transcripts, agent credentials, OAuth connections, and the derived policy views. The `Policy` type carries tags but no `changed_tags`, so you can condition on a policy’s current tags but not on a tag transition. Both `created_by` and `updated_by` hold `User` references, so compare them to `principal` directly rather than to a string. A condition on any attribute a type doesn’t carry is dead code. It’s valid Cedar, it saves when the resource type isn’t pinned, and it never matches. Pin the type and validation catches it instead. ## [](#prerequisites)Prerequisites - Access policies enabled for your organization. - An account that can author access policies. Admin is the only built-in role that can, because policy authoring is itself an Agentic Data Plane permission. Writer and Reader can’t create, edit, or view them. To let someone author policies without making them an Admin, grant the policy permissions through a custom role, or through a policy that names the `Policy` actions. See [Access policy permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#access-policy-permissions). ## [](#open-the-access-page)Open the Access page Open **Access** in the sidebar. These tabs cover everyday work: - **Policies**: The access policies you author. This is where you create, edit, and delete them. - **Templates**: Reusable policy templates, including the built-in quick-start set. The **Roles** and **System policies** tabs are read-only views of what RBAC already grants. ## [](#create-a-policy)Create a policy 1. On the **Policies** tab, click **Create policy**. 2. Give the policy a name. 3. Choose the effect: Permit grants access, and Forbid takes access away, overriding anything a permit or a role grants. 4. Set the principal: a user from your organization directory, or a group name from your identity provider (IdP). Group membership stays managed in the IdP. 5. Under Permissions, choose how to grant: - Actions: Pick individual verbs for a one-off grant. - Template: Link to a reusable template. The template fixes which actions the policy covers and which effect it carries, so the effect and conditions you set here are ignored. The principal and scope you set still apply, so you decide who gets it and how far it reaches. 6. In Actions mode, work through these steps: 1. Action: Select verbs. The picker offers them in categories: Management (`get`, `list`, `create`, `update`, `delete`), Execution (`invoke`, `call`), Access lifecycle (`attach`, `authorize`, `revoke`, `rotate`), and MCP protocol. A category is a display grouping, not a Cedar action group. 2. Applies to: Choose All resources for every type, All of a type for every resource of one type, or Specific resource for one. Picking a child type under Specific resource scopes to the children of one parent, such as the tools of one MCP server. 3. Conditions: Optional. Add WHEN rows to narrow when the policy applies and UNLESS rows to carve out exceptions. Each row conditions on a resource tag or its creator. See [Attribute reference](#attribute-reference). 7. Click **Create policy**. 8. Confirm the policy appears on the **Policies** tab with the effect, principal, and scope you set. > 💡 **TIP** > > Set the scope before picking verbs if you already know it. The picker only offers verbs the scoped type supports, so narrowing the scope afterward drops any selected verb the new type doesn’t have. The footer summarizes the grant in plain language and lists whatever is still missing before you can save. A `permit` that covers write verbs across every type raises a warning rather than blocking the save. Narrow the scope unless you genuinely want that reach. Every save validates the policy against the current schema, so a policy that stores successfully is well-formed. See [A policy won’t save](#a-policy-wont-save). The **Form** and **Cedar** toggle at the top of the page switches between the form fields and the Cedar text. Edits in either view update the other, and writing Cedar directly produces the same policy. The toggle doesn’t appear when your starting point fixes the principal or the scope, such as an agent’s **Permissions** tab, because raw Cedar could rewrite the locked clause. A policy that grants a whole role bundle opens read-only in this form. To change what it grants, change the role binding in the control plane. ### [](#start-from-a-template)Start from a template The **Templates** tab includes built-in quick-start templates, each a superset of the one before it: | Template | Grants | | --- | --- | | Read only | View-only access across all resources (get, list, and read). | | Sandboxed | Read-only access plus the ability to use agents and MCP servers (invoke, call tools). No management. | | Standard | Sandboxed access plus full management of agents, MCP servers, LLM providers, guardrails, triggers, and budgets. Excludes managing access policies. | | Full access | Every Agentic Data Plane action, including managing access policies. | A template fixes the action set, not the reach. Applies to defaults to All resources, and you narrow it from there: to every resource of one type, or to a single resource. The same Read only template can therefore back an org-wide viewer grant and a one-agent grant, with no second template to maintain. The built-in templates all permit. A template you author yourself can forbid instead, and a policy that links it denies. > 📝 **NOTE** > > A template is a live link, not a copy. Editing a template changes what every policy linked to it grants, so check who links a template before you widen it. ### [](#the-built-in-owner-lifecycle-policy)The built-in Owner lifecycle policy One policy ships with the feature and applies to everyone: users can get, update, and delete the agents, MCP servers, LLM providers, guardrails, and budgets they created, plus create and manage triggers on their own agents. It deliberately stops short in a few places. Creating a resource at the top level isn’t covered, because a create authorizes against a payload that records no creator yet, so ownership can’t grant it. Agent credentials and sessions aren’t covered either, because neither records a creator at all. ## [](#write-conditions)Write conditions Conditions make policies attribute-based: they read resource tags, group membership, and ownership at evaluation time. Follow these rules to write conditions that behave the way you expect. Validation at save time enforces most of them. Put the rule in `when` and the exemption in `unless`. Both clauses can sit on the same policy, and a negated test inside `when` is harder to read than the `unless` that replaces it. ### [](#guard-every-tag-read)Guard every tag read Reading a tag a resource doesn’t have is an evaluation error, not false. Always pair `getTag` with a `hasTag` check on the same key. This policy denies reads of ITAR-tagged agents, and exempts US persons: ```cedar forbid ( principal, action == Action::"Agent.get", resource is Agent ) when { resource.hasTag("itar") && resource.getTag("itar") == "true" } unless { principal in Group::"us-persons" }; ``` Without the `hasTag` guard, the policy errors on every untagged resource, and an erroring `forbid` denies. You would block access to every untagged agent, the opposite of the intent. Validation rejects the unguarded form when you save. A tag condition on a type that carries no tags is dead code: it’s valid Cedar but can never match, and validation flags it when the policy pins the resource type. Pin the type on every policy that reads tags so you get that check. See [Attribute reference](#attribute-reference) for which types carry tags. ### [](#model-people-facts-as-groups)Model people facts as groups Principals carry an email and group memberships, nothing else. Write `principal in Group::"nationality/US"`, not `principal.nationality == "US"`. Validation rejects attribute reads on principals. ### [](#guard-ownership-reads)Guard ownership reads To reference a resource’s creator, guard the read the same way as a tag: ```cedar resource has created_by && resource.created_by == principal ``` Write the attribute on the left of the comparison. The condition builder on the Access page parses this shape back into editable rows, and `principal == resource.created_by` leaves the policy read-only in the UI. ### [](#conditions-on-writes-evaluate-the-resulting-state)Conditions on writes evaluate the resulting state On create and update actions, `resource` is the state the operation produces, not the stored row. A `forbid` on production-tagged agents therefore fires on any update to an agent that carries the tag, even an update that doesn’t touch tags, and stops firing on the update that removes the tag. A write-scoped tag condition therefore behaves as a state invariant. A resource that violates it is frozen for non-exempt principals until an update fixes the violation. Reads and deletes evaluate the stored row, so a resulting-state condition behaves normally on those verbs. To gate the transition itself (adding or removing a specific tag), condition on `changed_tags`, the set of tag keys the update adds, removes, or changes: ```cedar forbid ( principal, action == Action::"Agent.update", resource is Agent ) when { resource has changed_tags && resource.changed_tags.contains("itar") } unless { principal in Group::"itar-cleared" }; ``` The `changed_tags` attribute exists only on the update path, so this policy doesn’t fire on create. To close the create case too, pair the policy with a create-scoped clause that reads the tag directly: ```cedar forbid ( principal, action == Action::"Agent.create", resource is Agent ) when { resource.hasTag("itar") } unless { principal in Group::"itar-cleared" }; ``` ### [](#guard-scalar-attributes-and-only-on-the-true-case)Guard scalar attributes, and only on the true case A scalar field that holds its zero value doesn’t appear on the entity at all, so `resource.enabled == true` errors whenever `enabled` is `false`, and that error denies. Guard scalar reads the same way as tags: ```cedar resource has enabled && resource.enabled == true ``` You can’t condition on a scalar’s zero value. Absent and `false` are indistinguishable, so `resource has enabled && resource.enabled == false` never matches, saves without complaint, and silently does nothing. Write the rule as the positive case and flip it with `unless`. ## [](#grant-an-agent-its-own-permissions)Grant an agent its own permissions Each agent’s detail page includes a **Permissions** tab. The Access policies card lists what the agent itself may do, acting as its own principal. **Create policy** opens the same policy form with the principal locked to the agent’s service account, so the policy targets a `User` principal like any other. A read-only Roles & access card shows the RBAC role bindings that apply to the agent, both direct and inherited. Those are managed in the control plane. An MCP server’s **Access** tab uses the same form from the other end: it locks the resource scope to that server rather than the principal. To control who may act **on** an agent, rather than what the agent may do, author a policy on the **Policies** tab with `resource == Agent::""`. ## [](#manage-policies-from-the-cli)Manage policies from the CLI Run `rpk ai policy` commands to create, get, list, update, and delete access policies, and to apply or diff them from manifest files for GitOps workflows. See [rpk ai policy](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy/). ## [](#examples)Examples These policies cover the shapes that come up most often. Adapt the group names, tag keys, and actions to your environment. ### [](#deny-a-group-access-to-production-agents)Deny a group access to production agents Members of `contractors` can’t read agents tagged `env=prod`, no matter what their roles grant: ```cedar forbid ( principal in Group::"contractors", action == Action::"Agent.get", resource is Agent ) when { resource.hasTag("env") && resource.getTag("env") == "prod" }; ``` ### [](#only-owners-can-modify-their-agents)Only owners can modify their agents Nobody but an agent’s creator can update it: ```cedar forbid ( principal, action == Action::"Agent.update", resource is Agent ) unless { resource has created_by && resource.created_by == principal }; ``` ### [](#restrict-who-can-call-one-mcp-servers-tools)Restrict who can call one MCP server’s tools Only the `support` group may call tools on the `zendesk` MCP server. The resource is the tool child entity, scoped to its parent server: ```cedar permit ( principal in Group::"support", action == Action::"McpServerTool.call", resource is McpServerTool in McpServer::"zendesk" ); ``` ### [](#default-closed-classification-gate)Default-closed classification gate Nothing is readable until it’s explicitly cleared. An agent with no `classification` tag is denied, and only `classification=public` clears the gate: ```cedar forbid ( principal, action == Action::"Agent.get", resource is Agent ) unless { resource.hasTag("classification") && resource.getTag("classification") == "public" }; ``` ## [](#troubleshoot)Troubleshoot Policy problems surface either when you save a policy or when a request hits it. Start from the error you have. ### [](#a-policy-wont-save)A policy won’t save Validation runs in strict mode against the current schema and reports every problem it finds at once. The error text is abbreviated here; search your own error for the quoted fragment. | Error contains | Fix | | --- | --- | | Unknown action | The action ID isn’t one the schema defines. If you pasted a role permission name, the error names the action ID to use instead. Otherwise take the . ID from Action reference; the error suggests a near match when it can find one. | | unrecognized entity type | Check the spelling and the casing. A resource that exists in the API isn’t automatically a policy entity type. | | unable to guarantee safety of access to tag | Add resource.hasTag("") && immediately before the getTag call, on the same key. | | not found on entity type User | Principals carry no attributes. Model the fact as a group and write principal in Group::"us-persons". | | without a resource type pin | A permit names an action group. Add resource is , or name each . action explicitly. | | clause can never match | The condition reads an attribute the pinned type doesn’t carry, such as a tag condition on an untagged type. Check Attribute reference, then retarget the policy or drop the clause. | Validation runs against the policy body, and only when a write includes it, so it never rejects a policy stored before a schema change. To run the current checks against an older policy, re-submit its body, for example by re-applying it with `rpk ai policy apply`. Editing only the name or description re-validates nothing. ### [](#a-request-is-denied)A request is denied A denial caused by access policies surfaces as a permission-denied error: - A `forbid` that matches cleanly returns a generic permission-denied message. Check the **Policies** tab for a `forbid` covering the action and resource. - A `forbid` that errors during evaluation also denies, and the error names the policy. Check that policy for an unguarded tag or attribute read, then re-submit its body to run current validation against it. - A missing `permit` also denies, with the same generic message. Confirm a `permit` covers the action, whether from a policy or from a role binding, and add one if nothing does. On update actions, authorization runs inside the write transaction, after request validation. An unauthorized caller can therefore receive an invalid-argument or not-found error instead of a permission-denied one, and a concurrent-modification conflict can surface as an aborted error. Treat any of them as a failure, and don’t branch on which arrives first. ## [](#next-steps)Next steps - [rpk ai policy](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy/) - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) - [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/) --- # Page 52: Manage Access **URL**: https://docs.redpanda.com/agentic-data-plane/control/access.md --- # Manage Access > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Manage Access latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: access page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: access.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/access.adoc description: Control who can do what in the Agentic Data Plane with access policies. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-08-03" --- Control who can do what in the Agentic Data Plane. Write access policies that permit or forbid specific actions for a user, a group, or a service account, and look up the permissions behind them. - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) Understand how Agentic Data Plane decides who can do what: access policies carry the decision, and RBAC bootstraps the administrators who author them. - [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) Create Cedar-based access policies that permit or forbid specific actions in the Agentic Data Plane, with conditions over resource tags and ownership. - [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/) Look up the permissions Agentic Data Plane enforces across LLM providers, MCP servers, agents, access policies, pipelines, knowledge bases, and governance APIs. --- # Page 53: Set Up Budgets **URL**: https://docs.redpanda.com/agentic-data-plane/control/budgets.md --- # Set Up Budgets > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up Budgets latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: budgets page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: budgets.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/budgets.adoc description: Cap LLM spend with per-agent budgets, and see what the Agentic Data Plane records automatically and where to view it. page-topic-type: how-to personas: platform_engineer, pilot_lead learning-objective-1: Set a per-agent budget that caps LLM spend and warns before the cap learning-objective-2: Identify what spending data the Agentic Data Plane records automatically learning-objective-3: View spend breakdowns by agent, model, and provider page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-30" --- The Agentic Data Plane caps LLM spend with budgets and records every LLM call as a spending event. Set a budget to enforce a hard spending cap per agent, then read what you actually spend on the **Cost & Usage** page under **Governance**, through individual [transcripts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript), and through breakdown queries by provider, model, user, agent, or provider type. After completing these steps, you will be able to: - Set a per-agent budget that caps LLM spend and warns before the cap - Identify what spending data the Agentic Data Plane records automatically - View spend breakdowns by agent, model, and provider ## [](#set-a-spend-limit)Set a budget A **budget** caps LLM spend over a recurring period. When an agent’s spend for the period reaches the budget’s hard limit, AI Gateway rejects that agent’s next LLM request with `HTTP 429` until the period resets. A separate warning threshold fires before the cap, so you can react before AI Gateway cuts the agent off. Redpanda Agentic Data Plane enforces budgets per agent. A budget identifies an _agent_ by its resource name in the form `agents/`, the same identity that appears as `agent_name` in spend data. ### [](#how-a-budget-works)How a budget works | Setting | What it does | | --- | --- | | Limit | The hard cap on per-period spend. You set it in dollars in the UI; the API stores it in USD microcents (1 cent = 1,000,000 microcents). When a matching agent’s accrued spend for the period reaches this value, the agent’s next LLM request through the gateway gets HTTP 429. Spend resets to zero at the start of each period. | | Warning threshold | A spend level, lower than the limit, at which Agentic Data Plane warns: gateway responses carry a SpendLimit-Warning header and Agentic Data Plane records a warning metric. Requests still pass. Must be greater than zero and less than the limit. | | Period | How often the spend pool resets: daily, weekly, or monthly. Periods are calendar-aligned in UTC: daily at 00:00 UTC, weekly at 00:00 UTC Monday, monthly at 00:00 UTC on the first of the month. | | Target agent | Which agent the budget applies to. Leave it unset to create the tenant default budget. Set it to an agent’s resource name (agents/) to create a per-agent override. | ### [](#default-and-per-agent-override-budgets)Default and per-agent override budgets You can have one default budget per tenant and at most one override per agent: - The **default** budget (no target agent) gives every agent its own independent pool of the limit per period. One agent reaching its cap doesn’t affect another. - A **per-agent override** targets a single agent by resource name (`agents/`) and replaces the default for that agent. Use an override to give a specific agent a higher or lower cap than the fleet default. Adding an override partway through a period doesn’t reset the agent’s usage: while a default budget exists, a new override inherits the default’s current spending window instead of starting a fresh one, so spend the agent already accrued this period still counts toward the override’s cap. Deleting and recreating an override behaves the same way, so an agent can’t gain fresh runway by having its override recreated mid-period. Editing a budget’s limit never moves the window either; spend already accrued this period keeps counting. (If no default budget exists, the override starts a new window when you create it.) The target agent is immutable. To move an override to a different agent, delete it and create a new one. An override matches on the agent’s resource name, so recreating an agent with the same name keeps the override pointed at it; the new instance counts as a continuation for budget purposes. When you read a budget, it also reports the current period’s spend, when the period started, when it resets, and (for the default budget) the agent currently closest to its cap. ### [](#set-a-budget-in-the-ui)Set a budget in the UI Open **Budgets** under **Governance** in the sidebar. The page shows the tenant default budget as a card: its cap and period, the warn threshold, and how many agents are doing fine, getting close, or over the limit. Per-agent overrides appear in a table below the card, with each override’s `Agent override` target, `Period`, current `Usage` against the cap, `Warn at` threshold, and when it was last `Updated`. To create the tenant default budget: 1. Click **Create default budget**. 2. Under `Budget`, set `Cap usage at` to a dollar amount and choose a period (day, week, or month). Use the quick-set chips ($25, $100, $500, $1,000) for common values. 3. Drag the `Warn at` slider to set the warning threshold as a percentage of the cap (80% by default). 4. Review the Configuration preview panel, which summarizes the budget, period, and the warn and block thresholds in dollars, then click **Create default budget**. Open the default budget from the **Budgets** page to see its detail view: the per-agent spending limit (each agent gets its own limit, not one shared pool), the reset schedule, the warn threshold, and how much each agent has spent so far toward its limit. To give one agent a different cap, click **Add override**, pick the agent (each agent can have at most one override; agents that already have one are grayed out), then set the `Budget` and `Warn at` controls the same way. The `Resource name` is auto-derived from the picked agent and is immutable; the `Display name` is editable. ### [](#manage-budgets-through-the-api)Manage budgets through the API `BudgetService` exposes standard create, read, update, and delete operations: | Method | Use it to | | --- | --- | | CreateBudget | Create the tenant default or a per-agent override. | | GetBudget, ListBudgets | Read one budget, or list all budgets, each with current-period spend status. | | UpdateBudget | Change the limit, warning threshold, or period. Send a field mask naming the fields you change. | | DeleteBudget | Remove a budget. Deleting the default removes the per-agent pools; deleting an override falls that agent back to the default. | A service account needs the matching `dataplane_adp_budget_*` permission for each operation (`create`, `get`, `list`, `update`, or `delete`). See [Budget permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#budget-permissions). ## [](#what-agentic-data-plane-records-automatically)What Agentic Data Plane records automatically Every LLM call routed through AI Gateway becomes a spending event. Each event captures: - Input tokens, output tokens, and cached tokens. - Total cost (in microcents). - Request count. - The provider, model, user, and organization context the call ran under. No setup required: the gateway captures spending the moment your first agent runs through it. Agentic Data Plane tracks streaming and non-streaming requests the same way, and attributes cache-write tokens (Anthropic 4.x, OpenAI 4.x prompt caches) correctly on streaming responses, so cost rollups stay accurate when an agent reuses long system prompts. > 📝 **NOTE** > > Agentic Data Plane reports cost in **microcents**. 1 cent = 1,000,000 microcents, so $1 = 100,000,000 microcents. Divide `total_cost_microcents` by 100,000,000 to convert to dollars. ### [](#per-request-pricing-variations)Per-request pricing variations A few request- or response-time signals change the rate Agentic Data Plane applies to a single call. You don’t configure these; the spending pipeline picks them up from the upstream response or request and bills accordingly. - **Anthropic fast mode**: Anthropic exposes a fast-mode option on some models (for example, Opus 4.6 fast) that carries a per-token premium over the default rate. Agentic Data Plane reads the `speed` field on each Anthropic response and bills fast-mode calls at the model’s fast-mode rate. Requests without a `speed` field fall back to the default rate. - **Context-tier pricing**: A few models charge a different rate after a request crosses a context-length threshold. Gemini Pro, for example, prices requests above a 128K-token context at a higher tier than shorter requests. Agentic Data Plane uses the call’s context-token count so requests at or above the threshold bill at the tiered rate automatically. ## [](#where-to-view-your-spend)Where to view your spend You don’t view spend on the **Budgets** page. The **Cost & Usage** page, transcripts, and breakdown queries are the read surfaces: | Surface | Use it for | | --- | --- | | Cost & Usage page (Governance sidebar group) | Time-series spend, request, and token charts across providers and models. Use it to group by provider, model, or token type, then filter by provider, model, cost type, token type, user, or agent. See Analyze Cost and Usage. | | Transcripts | Per-call cost on individual executions. Useful when investigating a specific agent run or debugging a cost anomaly. See Read a transcript. | | Breakdown queries | Aggregated spend by provider, model, user, agent, or provider type, available through GetSpendingBreakdown for programmatic access. | Every breakdown and time-series query reads from the same `SpendingFilter` shape: a time range plus optional `provider_name`, `model_id`, `user_email`, `agent_name`, `agent_uid`, or `organization_id` filters. Combine filters to scope a query (for example, "all spend on Anthropic for user `alice` in April"). You can break results down by provider, model, user, agent, or provider type; `organization_id` is a filter only, not a breakdown dimension. For more expressive queries, `SpendingFilter` also accepts an AIP-160 `filter` expression that lets you combine and negate dimensions in a single string (for example, `provider_name="anthropic" AND model_id!="claude-sonnet-4-6"`). The convenience fields and the `filter` expression compose; populate one or both. `user_email` and `organization_id` are populated automatically from the request’s authenticated identity (the caller’s email and organization), so spend is attributed without any setup on your part. ## [](#query-spend-programmatically)Query spend programmatically `SpendingService.GetSpendingBreakdown` is the canonical RPC for pulling spend out of Agentic Data Plane. Use it for chargeback reporting, scheduled emails, internal cost dashboards, or any workflow the built-in UI doesn’t cover. ### [](#authenticate)Authenticate `SpendingService` uses the same OIDC client-credentials grant as the rest of AI Gateway. Mint a service-account access token using the flow in [Authenticate with OIDC client credentials](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/#authenticate-with-oidc-client-credentials), then pass the token in the `Authorization: Bearer ` header on every call. The service account needs `dataplane_adp_spending_get` on the resource you’re querying. See [Spending permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#spending-permissions). ### [](#request-shape)Request shape `GetSpendingBreakdown` takes a `SpendingFilter` plus a `dimension`. The filter accepts: | Field | Meaning | | --- | --- | | start_time, end_time | RFC 3339 timestamps bracketing the window. Required. | | provider_name | Restrict to one LLM provider (matches the Name field on the provider’s detail page). | | model_id | Restrict to one model identifier (claude-sonnet-4-6, gpt-5.2, and so on). | | user_email | Restrict to one identified user, matched on the caller’s email. Anonymous traffic is excluded. | | agent_name | Restrict to one agent by its resource name (agents/), recorded on every call made by or on behalf of an agent. Leave it empty to match every row, including direct user calls; set it to scope spend to a single agent, summed across every instance that has used the name. | | agent_uid | Restrict to one agent instance, identified by an opaque UUID. Only valid when agent_name is also set: setting agent_uid alone is rejected. Use it to exclude spend from a previously deleted agent that reused the same name. | | organization_id | Restrict to one organization. Multi-tenant deployments only. | | filter | AIP-160 expression that combines and negates dimensions in a single string (for example, provider_name="anthropic" AND model_id!="claude-sonnet-4-6"). Composes with the structured fields above; populate one or both. | The `dimension` value chooses the breakdown dimension. Valid values are the `BreakdownDimension` enum: `BREAKDOWN_DIMENSION_PROVIDER`, `BREAKDOWN_DIMENSION_MODEL`, `BREAKDOWN_DIMENSION_USER`, `BREAKDOWN_DIMENSION_AGENT`, `BREAKDOWN_DIMENSION_PROVIDER_TYPE`, and `BREAKDOWN_DIMENSION_TAG`. A breakdown on `BREAKDOWN_DIMENSION_AGENT` keys on the agent’s resource name (`agents/`) and excludes rows with no agent (direct user calls), the same way the other dimensions skip empty keys. Spend is summed across every instance that has used the name, so an agent that was deleted and recreated appears as a single entry. `BREAKDOWN_DIMENSION_TAG` groups spend by the value of a tag you set on your agents (for example, `department`) and takes an extra `tag_key` field. Use it to attribute cost to a department, team, or environment. See [Track Spend by Tag](https://docs.redpanda.com/agentic-data-plane/control/cost-allocation-tags/). ### [](#curl-example)cURL example Pull per-user spend for the last 7 days against an Anthropic provider: ```bash ACCESS_TOKEN="" # from the client_credentials flow DATAPLANE_BASE="https://aigw..clusters.rdpa.co" curl -s --request POST \ --url "${DATAPLANE_BASE}/redpanda.api.adp.v1alpha1.SpendingService/GetSpendingBreakdown" \ --header "Authorization: Bearer ${ACCESS_TOKEN}" \ --header 'Content-Type: application/json' \ --data '{ "filter": { "start_time": "2026-05-17T00:00:00Z", "end_time": "2026-05-24T00:00:00Z", "provider_name": "prod-anthropic" }, "dimension": "BREAKDOWN_DIMENSION_USER" }' | jq ``` The response carries one `entries` row per user in the window. Each entry has a `key` (the user) and a `stats` object with `total_cost_microcents`, `total_requests`, `total_tokens` (server-derived), and per-bucket `input`, `output`, and `cached` usage. Divide `total_cost_microcents` by 100,000,000 to convert to dollars. ### [](#python-example)Python example Generated client code lives in the proto bundle; if your project doesn’t already import it from cloudv2, drive `SpendingService` over plain HTTPS: ```python import os, requests from datetime import datetime, timedelta, timezone token = os.environ["ACCESS_TOKEN"] # from the client_credentials flow base = os.environ["DATAPLANE_BASE"] # https://aigw..clusters.rdpa.co end = datetime.now(timezone.utc) start = end - timedelta(days=7) body = { "filter": { "start_time": start.isoformat().replace("+00:00", "Z"), "end_time": end.isoformat().replace("+00:00", "Z"), "filter": 'provider_name="prod-anthropic"', }, "dimension": "BREAKDOWN_DIMENSION_USER", } r = requests.post( f"{base}/redpanda.api.adp.v1alpha1.SpendingService/GetSpendingBreakdown", headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"}, json=body, ) r.raise_for_status() for entry in r.json().get("entries", []): stats = entry["stats"] dollars = int(stats["total_cost_microcents"]) / 100_000_000 print(f"{entry['key']}: ${dollars:,.2f} ({stats['total_requests']} requests)") ``` The proto-generated client (Connect-Go or grpc-python) is the long-term recommendation; the cURL and `requests` examples are for quick scripting. ### [](#related-methods)Related methods `SpendingService` exposes additional methods that follow the same `SpendingFilter` shape: - `GetSpendingSummary`: Total spend, tokens, and requests for the range, with no breakdown. Also returns the previous comparable period so you can show a trend. - `GetSpendingTimeSeries`: Spend bucketed over the time range (hourly or daily), for chart-style consumers. - `GetSpendingTimeSeriesByDimension`: Time-series buckets split by a breakdown dimension (top-N keys by cost), for stacked charts. Reports `truncated_key_count` when more keys matched than were returned. ## [](#guardrail-cost)Guardrail cost AWS bills guardrail evaluation directly to the AWS account whose credentials the guardrail’s backend uses. This cost does not appear in Agentic Data Plane cost reporting and is not counted against budgets. For current rates, see [AWS Bedrock pricing](https://aws.amazon.com/bedrock/pricing/). For what each policy does, see [How guardrails work](https://docs.redpanda.com/agentic-data-plane/control/guardrails/overview/) and [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/). ## [](#override-per-model-pricing)Override per-model pricing The Agentic Data Plane ships with default per-model pricing per provider, covering input, output, and cache-read prices for every model in the built-in catalog. Cost reporting uses these prices when it computes per-call spend, which is why every dollar value on the **Cost & Usage** page, in transcripts, and in `SpendingService` queries works without any setup. If your organization negotiates non-standard pricing, or you want to track spend against an internal chargeback rate, override the rates as part of configuring an LLM provider. Overrides are scoped to a single provider, where you edit the rate per model. See [Override per-model pricing](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#pricing-overrides). ## [](#next-steps)Next steps - [Read a transcript](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [How guardrails work](https://docs.redpanda.com/agentic-data-plane/control/guardrails/overview/) --- # Page 54: Track Spend by Tag **URL**: https://docs.redpanda.com/agentic-data-plane/control/cost-allocation-tags.md --- # Track Spend by Tag > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Track Spend by Tag latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cost-allocation-tags page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: cost-allocation-tags.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/cost-allocation-tags.adoc description: Group and filter LLM spend by the tags you set on agents, so you can attribute cost to a department, team, or environment for chargeback and reporting. page-topic-type: how-to personas: platform_engineer, pilot_lead learning-objective-1: Tag agents so their LLM spend is attributed to a department, team, or environment learning-objective-2: Group and filter spend by tag in the Cost & Usage page learning-objective-3: Query spend by tag programmatically through SpendingService page-git-created-date: "2026-07-22" page-git-modified-date: "2026-07-30" --- Cost-allocation tags reuse the key/value tags you already set on an agent and give them a second job: grouping and filtering [LLM](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) spend. Tag your agents with a dimension that matches how you report cost (for example, `department`, `team`, or `env`), and the Agentic Data Plane attributes every LLM call that agent makes to those tags. You can then break spend down by `department` on the **Cost & Usage** page, filter a report to `env=prod`, or pull per-team totals through the API for chargeback. You configure no separate tag registry and no spending pipeline. Tag an agent, and its spend is attributed automatically from the next call onward. After reading this page, you will be able to: - Tag agents so their LLM spend is attributed to a department, team, or environment - Group and filter spend by tag in the Cost & Usage page - Query spend by tag programmatically through SpendingService ## [](#prerequisites)Prerequisites - A running agent. If you do not have one, see [Agentic Data Plane Quickstart for Administrators](https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart/). - To query spend through the API, a service account with the `dataplane_adp_spending_get` permission. See [Spending permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#spending-permissions). ## [](#how-tag-based-spend-tracking-works)How tag-based spend tracking works When an agent makes an LLM call through AI Gateway, the Agentic Data Plane records a spending event and stamps it with a snapshot of that agent’s tags at the time of the call. Cost reports then group and filter on those stamped tags. Three behaviors follow from this design: Attribution is point-in-time and immutable The tags recorded on a spending event are the agent’s tags at the moment of the call. Editing an agent’s tags later never rewrites past spend: historical rows keep the tags they were recorded with, and new spend picks up the new tags. A chargeback report for a closed period stays stable even after you retag agents. Tag changes take a short time to take effect After you create or retag an agent, its spend can briefly record under the old tags, or as untagged, until the change takes effect. Unattributable spend is grouped as **Untagged** Calls that carry no value for the tag key you group on collapse into a single **Untagged** bucket. This bucket covers direct user calls that do not run through an agent, calls from an agent that has no tags, and the brief window before a new agent’s tags take effect. ## [](#tag-your-agents-for-cost-tracking)Tag your agents for cost tracking Tags live on the agent. Set them when you create an agent or edit an existing one, either in the agent form (the **Tags** section) or through the `CreateAgent` and `UpdateAgent` methods. See [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). A tag is a key/value pair. Choose keys that match your reporting dimensions and apply them consistently across agents: | Tag | Attributes spend to | | --- | --- | | department=sales | A business unit or cost center. | | team=revops | A team within a department. | | env=prod | An environment, to separate production spend from staging or development. | > ⚠️ **CAUTION** > > Do not put secrets or personally identifiable information (PII) in tag values. Tag values appear in cost reports and API responses. ## [](#track-spend-by-tag-in-the-ui)Track spend by tag in the UI Open **Cost & Usage** under **Governance** in the sidebar. For the page overview, see [Analyze Cost and Usage](https://docs.redpanda.com/agentic-data-plane/control/cost-usage/). Tags add two independent controls: Group by Tag Choose **Tag** as the group-by dimension, then pick a tag key, for example, `department`. Each chart series becomes one tag value, plus an **Untagged** series for spend with no value for that key. Use it to see how spend splits across departments or environments over the selected range. Tag filter Add a tag filter to scope the whole page to specific tag values, independent of what you group by. Selecting several values of the same key matches any of them (OR); selecting values across different keys requires all of them (AND). For example, filter to `department=sales` OR `department=support`, and `env=prod`, to see production spend for those two departments. The filter also applies to the CSV export, so an exported report matches what you see on screen. ## [](#query-spend-by-tag-with-the-api)Query spend by tag with the API The `SpendingService` API exposes tags as a breakdown dimension and a filter. These calls use the same authentication, `SpendingFilter` shape, and `dataplane_adp_spending_get` permission as the rest of the spending API. For authentication and the base request shape, see [Query spend programmatically](https://docs.redpanda.com/agentic-data-plane/control/budgets/#query-spend-programmatically). Reach the methods over the Connect protocol (POST and JSON) at `/redpanda.api.adp.v1alpha1.SpendingService/`. In this path, `` is your cluster’s AI Gateway base URL. Request fields use snake\_case (`start_time`, `tag_key`), and the JSON response uses camelCase (`totalCostMicrocents`, `totalRequests`). ### [](#group-a-breakdown-by-tag)Group a breakdown by tag The `GetSpendingBreakdown` and `GetSpendingTimeSeriesByDimension` methods accept `BREAKDOWN_DIMENSION_TAG` as the `dimension`, together with a `tag_key` that names the key to group on: | Field | Meaning | | --- | --- | | dimension | Set to BREAKDOWN_DIMENSION_TAG to group results by a tag value. | | tag_key | The tag key to group on, for example, department. Required when dimension is BREAKDOWN_DIMENSION_TAG, and ignored for any other dimension. A tag breakdown with an empty tag_key is rejected. | Each entry in the response keys on a tag value, with the same `stats` object (cost in microcents, request count, and token buckets) as the other breakdown dimensions. Spend with no value for the key is returned under the empty-string key, which the UI labels **Untagged**. ### [](#filter-any-query-by-tag)Filter any query by tag Every spending query reads a `SpendingFilter`, which accepts an [AIP-160](https://google.aip.dev/160) `filter` expression. Reference a tag with `tags.`, where `` is the tag key to match: ```none tags.department = "sales" tags.department = "sales" AND tags.env = "prod" ``` Tag predicates compose with the structured `SpendingFilter` fields, such as `provider_name`, `model_id`, `user_email`, and `agent_name`, so you can scope a per-user breakdown to one department. ### [](#curl-example)cURL example Break down the last seven days of spend by `department`: ```bash ACCESS_TOKEN="" # from the client_credentials flow DATAPLANE_BASE="https://aigw..clusters.rdpa.co" curl -s --request POST \ --url "${DATAPLANE_BASE}/redpanda.api.adp.v1alpha1.SpendingService/GetSpendingBreakdown" \ --header "Authorization: Bearer ${ACCESS_TOKEN}" \ --header 'Content-Type: application/json' \ --data '{ "filter": { "start_time": "2026-05-17T00:00:00Z", "end_time": "2026-05-24T00:00:00Z" }, "dimension": "BREAKDOWN_DIMENSION_TAG", "tag_key": "department" }' | jq ``` In this URL, `` is your cluster’s ID, and `` is a token from the client-credentials flow. The response carries one `entries` row per `department` value in the window, plus a row for untagged spend whose `key` is the empty string (omitted from the JSON). Each row has a `key`, the tag value, and a `stats` object whose fields use camelCase JSON names, such as `totalCostMicrocents` and `totalRequests`, plus per-bucket token usage. Divide `totalCostMicrocents` by 100,000,000 to convert to dollars. ## [](#limits-and-constraints)Limits and constraints | Constraint | Value | | --- | --- | | Tags per agent | At most 50 key/value pairs per agent. | | Tag value length | At most 256 characters per value. | | Distinct tag keys | No fixed limit. The keys available for grouping are whatever keys exist across agents that have recorded spend in the selected window. | | Untagged spend | Spend with no value for the grouped key (direct user calls, untagged agents, and spend recorded before an agent’s tags take effect) is grouped under a single Untagged bucket. | ## [](#troubleshooting)Troubleshooting Spend you expect under a tag shows as **Untagged** The agent had no tags when it made the call, or you tagged or retagged it moments earlier and the change had not taken effect yet. Recheck after a short wait. Direct user calls that do not run through an agent are always untagged. Editing an agent’s tags did not change past reports Attribution is point-in-time by design. Only spend recorded after the edit picks up the new tags; historical spend keeps the tags it was recorded with. ## [](#next-steps)Next steps - [Set Up Budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) - [Analyze Cost and Usage](https://docs.redpanda.com/agentic-data-plane/control/cost-usage/) - [Create an Agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) --- # Page 55: Analyze Cost and Usage **URL**: https://docs.redpanda.com/agentic-data-plane/control/cost-usage.md --- # Analyze Cost and Usage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Analyze Cost and Usage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cost-usage page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: cost-usage.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/cost-usage.adoc description: Break down LLM spend, requests, and tokens by provider, model, token type, agent, user, or tag. Compare periods, export CSV reports, and see which agents are active right now. page-topic-type: how-to personas: platform_engineer, pilot_lead learning-objective-1: Break down spend, requests, and tokens by provider, model, token type, agent, user, or tag learning-objective-2: Turn on period comparisons and export the data as CSV learning-objective-3: Identify the agents active right now and open their recent conversations page-git-created-date: "2026-07-30" page-git-modified-date: "2026-08-12" --- The **Cost & Usage** page is the spending and usage report for your Agentic Data Plane: every [LLM](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) call the gateway routes is recorded, priced, and charted here. Use it to see where spend comes from, attribute it to a provider, model, agent, user, or tag, and export the numbers for reporting. After reading this page, you will be able to: - Break down spend, requests, and tokens by provider, model, token type, agent, user, or tag - Turn on period comparisons and export the data as CSV - Identify the agents active right now and open their recent conversations ## [](#prerequisites)Prerequisites This page assumes traffic through at least one [LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/); with no traffic, the charts are empty. > 📝 **NOTE** > > Dollar values come from per-model pricing: the built-in catalog rates, or the overrides you set on a provider. Cost reporting works without any pricing setup. See [Override per-model pricing](https://docs.redpanda.com/agentic-data-plane/control/budgets/#override-per-model-pricing). ## [](#open-the-report)Open the report Open **Cost & Usage** under **Governance** in the sidebar. The page has two tabs: - **Cost & usage**: The spending report. Charts of spend, requests, and tokens over a time window, with grouping, filters, and CSV export. - **Activity**: The live view. The agents active right now, with a drill-down into their recent conversations. ## [](#choose-the-time-window)Choose the time window The date-range picker offers last 24 hours, last 7 days (the default), last 14 days, last 30 days, last 90 days, month to date, quarter to date, year to date, and custom ranges. The chart subtitle shows the selected date range and bucket size. A custom range sets an exact start and end time, to the minute and interpreted in UTC, rather than a relative period. It writes `customStart` and `customEnd` ISO-8601 timestamps to the page URL, so the view is shareable: copy the URL after picking a custom range and any teammate who opens it lands on the same window. To compare against the period before, turn on the **Compare to previous period** switch in the date-range picker. The picker shows which previous window the report compares against, and each series in the report compares against its own numbers from that period. ## [](#break-down-and-filter)Break down and filter Use **Group by** to switch the chart breakdown between providers, models, token type, agents, users, and tags: - Group by provider to see which upstream consumed the most budget. - Group by model to see which model drove spend inside one or more providers. - Group by token type to separate input, output, cached, cache-write, and reasoning usage where those buckets apply. - Group by agent or user to attribute spend to the agent or the identified caller that drove it. - Group by a tag key to attribute spend to a dimension you define, such as `department` or `env`. Tag grouping picks the key from a submenu, and the control shows the active key, for example `Tag: department`. See [Track Spend by Tag](https://docs.redpanda.com/agentic-data-plane/control/cost-allocation-tags/). Each dimension, Provider, Model, Cost type, Token type, User, and Agent, has its own filter button above the charts, and you can combine them freely. For example, filter to one Anthropic provider, drill into `claude-opus-4-7`, then limit the spend view to input tokens. Selecting an agent also narrows the provider options to the providers that agent used. The tag filter is independent of grouping, so you can group by model while filtering to `env=prod`. Deleted providers stay in the report so history remains complete. Turn off **Show deleted providers** to hide them; when everything that matches your filters is from deleted providers, the chart says so instead of rendering blank. ## [](#read-the-charts)Read the charts The report includes these charts: - Spend over time: Estimated spend in USD for the selected range. - Requests over time: Request count for the selected range. - Tokens over time: Token count for the selected range. The chart renders empty buckets in the selected range as zero-height bars rather than gaps, so quiet days line up with their date label and the trend stays readable when traffic is bursty. The chart palette is colorblind-safe. When multiple providers of the same type exist (for example, two OpenAI providers), the chart renders each one with a distinct hatched pattern so the series stay visually distinguishable. The spend chart footer summarizes the selected view by cost bucket, including total, input, output, cached, cache writes, and reasoning when the selected traffic includes those categories. Each chart can be maximized for a closer look, and exported on its own: the chart export downloads the data as currently shown, including any truncation the chart applies to keep the top series readable. ## [](#download-cost-and-usage-report)Download the report as CSV To export the data behind the charts, use the download controls in the toolbar: - Click **Download full report** to download the complete, untruncated dataset for the current filters and date range as a CSV file. Every cost type and token type is a separate column. - Next to **Download full report**, open the report-grouping options to choose how the exported rows are grouped. Under Time bucket, pick Hourly, Daily, Monthly, or Total (no time column). Under Break down by, select any combination of Provider, Model, User, and Agent. Use **Download full report** when you need the whole dataset rather than the charted view. ## [](#see-which-agents-are-active-now)See which agents are active now The **Activity** tab answers a different question from the report: instead of what something cost, it shows what’s running right now. It ranks the agents with traffic over a window of the last hour (the default), the last 24 hours, or the last 7 days, and shows each agent’s change against the previous window. Select an agent to see its recent requests and open a conversation. The Playground’s live view also hands off to this tab: when an agent task is still running after several minutes, the Playground points you here to follow it to completion. ## [](#next-steps)Next steps - [Set Up Budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) - [Track Spend by Tag](https://docs.redpanda.com/agentic-data-plane/control/cost-allocation-tags/) - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) --- # Page 56: Set Up Guardrails **URL**: https://docs.redpanda.com/agentic-data-plane/control/guardrails.md --- # Set Up Guardrails > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up Guardrails latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: guardrails/index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: guardrails/index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/guardrails/index.adoc description: Understand how guardrails work, create them, review blocked requests, and look up the available policies. page-git-created-date: "2026-06-01" page-git-modified-date: "2026-07-08" --- Guardrails inspect and control the requests and responses that flow through your AWS Bedrock LLM providers. Learn how they work, create one, review blocked requests, and look up the available policies. - [How Guardrails Work](overview/) Learn how Agentic Data Plane guardrails use AWS Bedrock policies to inspect and control LLM requests and responses, and how a blocked request surfaces. - [Create a Guardrail](create-guardrail/) Create a guardrail with the wizard, configure its Bedrock authentication and policies, attach it to a Bedrock LLM provider, and verify that it blocks. - [Review Blocked Requests](violations/) Understand how a blocked request surfaces, where guardrail activity is recorded, and what to check when a guardrail blocks too much or too little. - [Guardrail Policy Reference](types-reference/) Reference for the guardrail policy types, their configuration fields, actions, directions, and limits. --- # Page 57: Create a Guardrail **URL**: https://docs.redpanda.com/agentic-data-plane/control/guardrails/create-guardrail.md --- # Create a Guardrail > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Create a Guardrail latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: guardrails/create-guardrail page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: guardrails/create-guardrail.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/guardrails/create-guardrail.adoc description: Create a guardrail with the wizard, configure its Bedrock authentication and policies, attach it to a Bedrock LLM provider, and verify that it blocks. page-topic-type: how-to personas: security_compliance_lead, platform_engineer learning-objective-1: Create a guardrail and configure its backend and policies learning-objective-2: Attach the guardrail to an LLM provider and enable it learning-objective-3: Verify the guardrail blocks a request and trace it through the transcript page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-08" --- Create a guardrail to inspect and control the requests and responses that flow through an AWS Bedrock LLM provider. You configure the guardrail in a wizard (details, authentication, and policies), attach it to a Bedrock provider, and confirm it blocks. After reading this page, you will be able to: - Create a guardrail and configure its backend and policies - Attach the guardrail to an LLM provider and enable it - Verify the guardrail blocks a request and trace it through the transcript ## [](#prerequisites)Prerequisites - An AWS region and Bedrock access. A guardrail reconciles against the AWS Bedrock control plane, so you need either AWS credentials it can use or a Bedrock LLM provider whose credentials it can borrow. - At least one AWS Bedrock LLM provider to attach the guardrail to. Guardrails attach to Bedrock providers only. See [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). - For the Sensitive information, Content filters, Contextual grounding, and Automated reasoning policies, confirm the features you need are available in your chosen Bedrock region. ## [](#add-the-details)Add the details Open the wizard and give the guardrail its name and the messages that callers see when a policy blocks a request. 1. Open **Guardrails** in the sidebar and click **Create guardrail**. 2. On the Details step, set the following: | Field | Notes | | --- | --- | | Name | Required. Lowercase letters, numbers, and hyphens only, 1 to 63 characters. Used in the resource path and can’t be changed after creation. | | Display Name | Optional. A friendlier name shown in the list and detail views. | | Description | Optional. | | Blocked Input Message | Required. The message returned to the caller when a policy blocks a prompt. 1 to 500 characters. | | Blocked Output Message | Required. The message returned instead of the model’s response when a policy blocks it. 1 to 500 characters. | | Enabled | Leave off for now. Enable the guardrail after you configure its policies. | 3. Click **Next**. ## [](#configure-authentication)Configure authentication This step, labeled **Authentication** in the wizard, sets the AWS region and credentials the guardrail uses to reach Bedrock. 1. Select a region. The region decides which Bedrock models and guardrail features you can reach, because not every feature is available in every region. Use the same region as the Bedrock provider or model you intend to protect. 2. Choose a credential source: - Standalone AWS credentials: Supply credentials for this guardrail directly. - Borrow from a Bedrock provider: Reuse the credentials of a Bedrock LLM provider you already configured. 3. If you chose standalone credentials, choose a credential type: - Default provider chain: Use the credentials available in the runtime environment. - Static access keys: Supply an access key ID and secret access key, each stored as a secret reference in the Agentic Data Plane secret store. - Assume IAM role: Supply a role ARN, with an optional external ID and session name. 4. Click **Next**. ## [](#turn-on-policies)Turn on policies Each policy is optional, but a guardrail must enforce at least one. Turn on the policies you need, then configure each. For the full configuration of every policy, see [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/). ### [](#start-from-a-preset)Start from a preset The Quick start row at the top of the Policies step applies a preset, a bundle of policy settings for a common goal: - **Content safety**: Blocks hate speech, insults, sexual content, violence, and misconduct at high strength. - **PII protection**: Masks emails, phone numbers, names, addresses, card numbers, and US Social Security numbers. - **Prompt attack defense**: Blocks jailbreak and prompt-injection attempts before they reach the model. Presets are additive: applying one never removes rules you already configured. Apply a preset, then adjust the individual policies it turned on. ### [](#choose-individual-policies)Choose individual policies Turn on each policy you need and configure its rules: - Content filters: Harmful-content categories (hate, insults, sexual, violence, misconduct, and prompt attacks). - Word filters: Exact words and phrases from your own lists and platform-managed lists such as profanity. - Denied topics: Topics blocked by meaning rather than exact words. - Sensitive information: PII detected by built-in entity types and your own regex patterns, then detected, blocked, or anonymized. - Contextual grounding: Factual grounding and relevance checks for RAG-style output. Output only. - Automated reasoning: Formal Bedrock Automated Reasoning policies. Detect only. Each policy you turn on exposes a separate **Evaluate this policy** toggle. Turn it off to save the policy’s configuration without enforcing it, which lets you stage a policy before it takes effect. > 📝 **NOTE** > > Two separate toggles control evaluation: > > - The guardrail-level **Enabled** toggle, on the Details step, turns the whole guardrail on or off. A disabled guardrail keeps its configuration but evaluates no traffic. > > - The policy-level **Evaluate this policy** toggle controls a single policy within the guardrail. A policy with this toggle off is saved but not evaluated. > > > A policy acts on traffic only when its guardrail is enabled and its own **Evaluate this policy** toggle is on. A policy you turn on needs at least one rule. Agentic Data Plane rejects a guardrail that has no policy turned on, or that turns on a policy with no rules set, such as a content-filter policy with no category configured or a contextual-grounding policy with neither grounding nor relevance enabled. A staged policy does not count toward this minimum, because a policy with its **Evaluate this policy** toggle off is not enforced. A guardrail needs at least one policy that is both turned on and evaluated, so if you stage every policy, Agentic Data Plane rejects the guardrail. When you finish, click **Create guardrail**. ## [](#attach-the-guardrail-to-a-provider)Attach the guardrail to a provider A guardrail takes effect only after a provider references it. On an AWS Bedrock LLM provider, set the guardrail field in the provider’s Bedrock settings to the one you created. Only Bedrock providers expose this setting. A provider references one guardrail, and you can reuse the same guardrail across many providers. See [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). ## [](#enable-the-guardrail)Enable the guardrail After you configure the policies and attach the guardrail, set the guardrail to enabled. A disabled guardrail keeps its configuration but skips evaluation entirely, which is useful while you stage a policy or troubleshoot whether the guardrail is responsible for unexpected blocks. Enabling or disabling a guardrail takes effect within about 30 seconds, because the gateway briefly caches guardrail settings. ## [](#verify-the-guardrail-blocks)Verify the guardrail blocks Send a request through an attached provider that violates a policy. For example, with the sensitive-information policy set to block on input, send a prompt that contains an email address or other PII. The request returns your blocked input message instead of a model response. Open the request’s [transcript](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript) and confirm the guardrail recorded its action. See [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) for the transcript walkthrough and [Review blocked requests](https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations/) for how blocked requests surface. ## [](#edit-disable-or-delete-a-guardrail)Edit, disable, or delete a guardrail From the guardrails list or a guardrail’s detail page: - Edit the guardrail to change its backend or policies. Changes take effect within about 30 seconds, because the gateway briefly caches guardrail settings. - Disable the guardrail to stop evaluation without losing its configuration. - Delete the guardrail to remove it permanently. Deletion asks you to confirm by typing the guardrail’s name. > ❗ **IMPORTANT** > > You cannot delete a guardrail while an LLM provider still references it. Agentic Data Plane blocks the delete and names the providers you must detach first. Open each listed provider, clear its `Guardrail` field, then delete the guardrail. This protects in-flight traffic: if the guardrail were removed while a provider still pointed at it, every request through that provider would fail. ## [](#next-steps)Next steps - [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/) - [Review blocked requests](https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations/) - [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) --- # Page 58: How Guardrails Work **URL**: https://docs.redpanda.com/agentic-data-plane/control/guardrails/overview.md --- # How Guardrails Work > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: How Guardrails Work latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: guardrails/overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: guardrails/overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/guardrails/overview.adoc description: Learn how Agentic Data Plane guardrails use AWS Bedrock policies to inspect and control LLM requests and responses, and how a blocked request surfaces. page-topic-type: overview personas: security_compliance_lead, platform_engineer, agent_builder learning-objective-1: Describe what a guardrail does and how you attach one to an AWS Bedrock LLM provider learning-objective-2: Identify the available policy types and the situations each fits learning-objective-3: Recognize how a blocked request surfaces and which page to read next page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- A guardrail is a set of safety and policy filters that inspect and control how agents and gateways use models. A guardrail can block prompt injection, deny off-topic content, redact or block personally identifiable information (PII), and check responses for factual grounding. You attach a guardrail to an AWS Bedrock LLM provider, and Agentic Data Plane applies it to every request and response that flows through that provider. Agentic Data Plane guardrails run on AWS Bedrock Guardrails. Each guardrail syncs its policy configuration to the Bedrock control plane, so it needs an AWS region and Bedrock credentials. Guardrails attach to AWS Bedrock providers only; other provider types don’t expose a guardrail setting. After reading this page, you will be able to: - Describe what a guardrail does and how you attach one to an AWS Bedrock LLM provider - Identify the available policy types and the situations each fits - Recognize how a blocked request surfaces and which page to read next ## [](#policy-types)Policy types A guardrail bundles a set of policies. Each is optional, but a guardrail must enforce at least one. Turn on the ones you need. | Policy | What it does | | --- | --- | | Content filters | Classify prompts and responses against harmful-content categories: hate, insults, sexual, violence, misconduct, and prompt attacks. Prompt-attack detection applies to input only. | | Word filters | Block or detect exact words and phrases, using your own lists and platform-managed lists such as profanity. | | Denied topics | Block content by meaning rather than exact words, so it catches paraphrases and misspellings. | | Sensitive information | Detect PII using built-in entity types and your own regex patterns, then detect, block, or anonymize it. | | Contextual grounding | For RAG-style applications, check model output for factual grounding against a source and relevance to the query. Output only. | | Automated reasoning | Mathematically verify model output against formal Bedrock Automated Reasoning policies. Detect only: it never blocks, and findings appear in the trace. | For each policy’s full configuration, see [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/). ## [](#where-a-guardrail-runs)Where a guardrail runs A guardrail evaluates both sides of an LLM call: - Input: Agentic Data Plane evaluates the prompt before forwarding it upstream. Use input evaluation to stop sensitive or malicious content from reaching the model. - Output: Agentic Data Plane evaluates the model’s response before returning it to the caller. Use output evaluation to control what the model generates. Some policies are direction-specific. Content-filter prompt-attack detection runs on input only, contextual grounding runs on output only, and automated reasoning runs on output and only reports findings. For streaming responses, Agentic Data Plane skips output evaluation in this release. Input evaluation still applies. ![Guardrail evaluation flow. A caller’s prompt passes through input policies before the Bedrock model. Allowed prompts reach the model; blocked prompts return the configured blocked-input message. For non-streaming calls, model output passes through output policies before an allowed, redacted, or blocked response reaches the caller.](https://docs.redpanda.com/agentic-data-plane/control/_images/guardrail-evaluation-flow.svg) Figure 1. A guardrail can stop a request before the model or control the response after it ## [](#where-you-attach-a-guardrail)Where you attach a guardrail You attach a guardrail by setting an AWS Bedrock LLM provider’s guardrail. The guardrail setting appears in the provider’s Bedrock settings, and only Bedrock providers can reference a guardrail. Each provider references at most one guardrail, and a single guardrail can be reused across many providers. The guardrail’s detail page lists the providers that use it. You create and configure the guardrail first, then attach it from the provider. See [Create a guardrail](https://docs.redpanda.com/agentic-data-plane/control/guardrails/create-guardrail/). Agentic Data Plane enforces this reference in both directions. A provider can only point at a guardrail that exists, and you cannot delete a guardrail while a provider still references it. To remove a guardrail that is in use, detach it from each provider first. ## [](#what-happens-when-a-guardrail-blocks)What happens when a guardrail blocks When a policy blocks an input, Agentic Data Plane stops the request and returns your configured blocked input message to the caller. When a policy blocks an output, Agentic Data Plane returns your configured blocked output message instead of the model’s response. The sensitive-information policy can also anonymize matched PII in place rather than block, replacing each match with its entity type, such as `{EMAIL}`. The two directions differ: on output, Agentic Data Plane delivers the redacted response to the caller. On input, Agentic Data Plane does not forward the redacted prompt to the model. Agentic Data Plane short-circuits the request like a block and returns your configured blocked input message. Agentic Data Plane records guardrail activity as attributes on the request’s OpenTelemetry trace, which you read in [transcripts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript). This release does not include a separate violations dashboard. See [Review blocked requests](https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations/) for how a blocked request surfaces. ## [](#next-steps)Next steps - [Create a guardrail](https://docs.redpanda.com/agentic-data-plane/control/guardrails/create-guardrail/) - [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/) - [Review blocked requests](https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations/) --- # Page 59: Guardrail Policy Reference **URL**: https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference.md --- # Guardrail Policy Reference > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Guardrail Policy Reference latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: guardrails/types-reference page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: guardrails/types-reference.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/guardrails/types-reference.adoc description: Reference for the guardrail policy types, their configuration fields, actions, directions, and limits. page-topic-type: reference personas: security_compliance_lead, platform_engineer page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-08" --- A guardrail bundles a set of policies, each backed by AWS Bedrock Guardrails. Each policy is optional, but a guardrail must enforce at least one. This page documents each policy type’s configuration fields, available actions, direction settings, and regional availability. ## [](#common-settings)Common settings Two settings recur across policies: - Action: what the policy does when it matches. `None` detects and records the match in the trace without intervening. `Block` stops the request and returns the configured blocked message. The sensitive-information policy adds `Anonymize`. - Direction: most policies evaluate input, output, or both, and you set the action per direction. Some policies are fixed to one direction (noted below). > 📝 **NOTE** > > Feature availability varies by AWS region. Choose a region that supports the policies you need, and see the [AWS Bedrock Guardrails documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html) for exhaustive behavior and regional support. ## [](#content-filters)Content filters Classify prompts and responses against harmful-content categories and block or detect them per category. | Field | Description | | --- | --- | | Categories | Hate, Insults, Sexual, Violence, Misconduct, and Prompt attack. Configure each category independently. Prompt-attack detection evaluates input only. | | Strength | Per category and direction. Sets the confidence cutoff for a match: None scores the category in the trace without acting, Low matches only high-confidence content, Medium matches medium-confidence and above, and High matches any non-negligible content. Higher strength is stricter. | | Action | Per category and direction: None (detect) or Block. | | Modality | Text or Image. | ## [](#word-filters)Word filters Block or detect exact words and phrases. | Field | Description | | --- | --- | | Custom words | Your own list of words and phrases to match. | | Managed lists | Platform-managed lists. Profanity is available today. | | Action | Per direction (input and output): None (detect) or Block. Set independently for custom words and for each managed list. | ## [](#denied-topics)Denied topics Block content by meaning rather than exact words, so the policy catches paraphrases and misspellings. A policy holds up to 30 topics. | Field | Description | | --- | --- | | Name | Topic name, 1 to 100 characters. | | Definition | What the topic covers, up to 1000 characters. The definition drives the semantic match, so write it as a clear, self-contained statement. Keep example phrases and negations out of the definition; put concrete examples in the Examples field instead, which improves accuracy. | | Examples | Up to five example phrases, each up to 100 characters, that match the topic. | | Action | Per direction (input and output): None (detect) or Block. | ## [](#sensitive-information)Sensitive information Detect personally identifiable information (PII) by built-in entity type or by your own regular expressions, then detect, block, or anonymize it. | Field | Description | | --- | --- | | Entities | Built-in entity types. Each entity has a per-direction action. | | Regexes | Custom patterns. Each rule has a name (1 to 100 characters), an RE2 pattern (1 to 500 characters; lookaround is not supported), an optional description, and a per-direction action. | | Action | Per direction (input and output): None (detect), Block, or Anonymize. Anonymize replaces each match in place with its entity type, such as {EMAIL}, and applies to text only. The two directions differ: on output, Agentic Data Plane delivers the redacted response to the caller. On input, this release does not forward the redacted prompt to the model. Instead, an anonymize match short-circuits the request like a block: the model is never called, and Agentic Data Plane returns your configured blocked input message rather than the redacted prompt. Block replaces the whole payload with the blocked message. | The built-in entity types are: `ADDRESS`, `AGE`, `NAME`, `EMAIL`, `PHONE`, `USERNAME`, `PASSWORD`, `DRIVER_ID`, `LICENSE_PLATE`, `VEHICLE_IDENTIFICATION_NUMBER`, `CREDIT_DEBIT_CARD_CVV`, `CREDIT_DEBIT_CARD_EXPIRY`, `CREDIT_DEBIT_CARD_NUMBER`, `PIN`, `INTERNATIONAL_BANK_ACCOUNT_NUMBER`, `SWIFT_CODE`, `IP_ADDRESS`, `MAC_ADDRESS`, `URL`, `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, `US_BANK_ACCOUNT_NUMBER`, `US_BANK_ROUTING_NUMBER`, `US_INDIVIDUAL_TAX_IDENTIFICATION_NUMBER`, `US_PASSPORT_NUMBER`, `US_SOCIAL_SECURITY_NUMBER`, `CA_HEALTH_NUMBER`, `CA_SOCIAL_INSURANCE_NUMBER`, `UK_NATIONAL_HEALTH_SERVICE_NUMBER`, `UK_NATIONAL_INSURANCE_NUMBER`, and `UK_UNIQUE_TAXPAYER_REFERENCE_NUMBER`. ## [](#contextual-grounding)Contextual grounding For retrieval-augmented generation (RAG) applications, check model output against a source and the user’s query. This policy evaluates output only and has two independent sub-filters. | Sub-filter | Description | | --- | --- | | Grounding | Checks that the response is factually grounded in the provided source. | | Relevance | Checks that the response is relevant to the user’s query. | Each sub-filter has its own enable toggle, a threshold between `0.0` and `0.99`, and an action. The action fires when the response scores below the threshold, so higher thresholds are stricter. The action is `None` (detect) or `Block`. ## [](#automated-reasoning)Automated reasoning Mathematically verify model output against formal Bedrock Automated Reasoning policies. This policy is detect-only: it never blocks, and its findings appear in the trace. | Field | Description | | --- | --- | | Policy ARNs | One or two Bedrock Automated Reasoning policy Amazon Resource Names (ARNs) to attach. Each ARN must point at a specific numeric version (for example, ending in :1 or :2); the DRAFT version is rejected. Create and publish the policies in the AWS Bedrock console first, then reference their versioned ARNs here. | | Confidence threshold | A value between 0.0 and 1.0. Below this confidence, a finding is reported as non-definitive. | ## [](#next-steps)Next steps - [Create a guardrail](https://docs.redpanda.com/agentic-data-plane/control/guardrails/create-guardrail/) - [Review blocked requests](https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations/) - [How guardrails work](https://docs.redpanda.com/agentic-data-plane/control/guardrails/overview/) --- # Page 60: Review Blocked Requests **URL**: https://docs.redpanda.com/agentic-data-plane/control/guardrails/violations.md --- # Review Blocked Requests > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Review Blocked Requests latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: guardrails/violations page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: guardrails/violations.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/guardrails/violations.adoc description: Understand how a blocked request surfaces, where guardrail activity is recorded, and what to check when a guardrail blocks too much or too little. page-topic-type: how-to personas: security_compliance_lead, platform_engineer, agent_builder learning-objective-1: Review what a caller sees when a guardrail blocks a request learning-objective-2: Find guardrail activity in a request's transcript learning-objective-3: Diagnose a guardrail that blocks too much or doesn't fire page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-30" --- When a guardrail acts on a request, the caller sees a blocked message and Agentic Data Plane records the action on the request’s trace. This release does not include a dedicated violations dashboard, so you review blocked requests through [transcripts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript). After reading this page, you will be able to: - Review what a caller sees when a guardrail blocks a request - Find guardrail activity in a request’s transcript - Diagnose a guardrail that blocks too much or doesn’t fire ## [](#what-the-caller-sees)What the caller sees - Blocked input: Agentic Data Plane stops the request before it reaches the model and returns your configured blocked input message. - Blocked output: Agentic Data Plane returns your configured blocked output message instead of the model’s response. - Anonymized PII: When the sensitive-information policy anonymizes a match on output, Agentic Data Plane delivers the response with each match replaced in place by its entity type, such as `{EMAIL}`. On input, this release does not forward the redacted prompt to the model: Agentic Data Plane short-circuits the request like a block and returns your configured blocked input message, not the redacted prompt. Automated reasoning never blocks. It records findings on the trace for your application to act on. ## [](#where-guardrail-activity-is-recorded)Where guardrail activity is recorded Agentic Data Plane records guardrail activity as attributes on the request’s OpenTelemetry trace, which you read in the request’s transcript alongside the LLM call, tool calls, and cost data. See [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). For streaming responses, Agentic Data Plane skips output evaluation and marks the skip on the trace. Input evaluation still applies to streaming requests. ## [](#troubleshoot-guardrail-behavior)Troubleshoot guardrail behavior When a guardrail blocks legitimate traffic or fails to act when you expect it to, work through the checks in this section to find the cause. ### [](#a-guardrail-blocks-too-much)A guardrail blocks too much If a guardrail blocks legitimate traffic: - Content filters: Lower the strength for the affected category, or set its action to detect while you tune. - Sensitive information: Narrow the entity types or regex patterns to the data you actually care about. Structured payloads such as code or JSON can contain strings that resemble PII. - Denied topics: Tighten each topic’s definition and examples so the semantic match is more specific. - Contextual grounding: Lower the threshold so only clearly ungrounded or irrelevant responses are caught. To stage a change without enforcing it, set the policy’s action to detect and review the results on the trace before you block. For contextual grounding, you can also turn off a filter’s **Evaluate this check** switch to skip evaluating it entirely. ### [](#a-guardrail-doesnt-fire)A guardrail doesn’t fire If you expect a guardrail to act and it doesn’t: - Confirm the guardrail is enabled. A disabled guardrail keeps its configuration but skips evaluation. - Confirm the specific policy is enabled. The guardrail’s detail page shows each policy’s **Enabled** or **Disabled** status; a disabled policy takes no action. For contextual grounding, also confirm the filter’s **Evaluate this check** switch is on. - Confirm the Bedrock provider the request used references this guardrail. A guardrail acts only on providers that reference it. - Confirm the direction. An output-only policy such as contextual grounding never acts on input, and output policies are skipped for streaming responses. - Confirm the request actually went through the Bedrock provider. Direct-to-provider requests that bypass Agentic Data Plane are not evaluated. ## [](#next-steps)Next steps - [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [Guardrail policy reference](https://docs.redpanda.com/agentic-data-plane/control/guardrails/types-reference/) - [Create a guardrail](https://docs.redpanda.com/agentic-data-plane/control/guardrails/create-guardrail/) --- # Page 61: Control Who Can Do What **URL**: https://docs.redpanda.com/agentic-data-plane/control/permissions-overview.md --- # Control Who Can Do What > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Control Who Can Do What latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: permissions-overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: permissions-overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/permissions-overview.adoc description: "Understand how Agentic Data Plane decides who can do what: access policies carry the decision, and RBAC bootstraps the administrators who author them." page-topic-type: concept personas: security_compliance_lead, platform_engineer learning-objective-1: Explain how access policies decide who can do what in Agentic Data Plane learning-objective-2: Explain why Admin is the only built-in role that reaches Agentic Data Plane learning-objective-3: Grant a user, a group, or a service account the access it needs page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-07" --- Redpanda Agentic Data Plane decides who can do what with [access policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). A policy names a principal, an action, and a resource, and either permits or forbids. That’s the authorization model: policies are what you write, and policy evaluation is what every API call resolves against. Role-based access control (RBAC) has one job here. Assign the Admin role to the people who administer the deployment, and they author the policies that grant everyone else. Admin is the only built-in role that carries Agentic Data Plane permissions at all. This page explains how the two fit together and how to grant access to a user, a group, or a service account. After reading this page, you will be able to: - Explain how access policies decide who can do what in Agentic Data Plane - Explain why Admin is the only built-in role that reaches Agentic Data Plane - Grant a user, a group, or a service account the access it needs ## [](#policies-decide-roles-bootstrap)Policies decide, roles bootstrap Every Agentic Data Plane request resolves the same way: - A request is denied unless a `permit` matches it. - A matching `forbid` always wins, over any `permit` and over anything a role granted. Both role bindings and access policies feed that one evaluation. Redpanda compiles your role bindings into permits automatically, so an Admin’s access arrives through the same path as a policy you wrote. There is no separate RBAC check that runs first and no second system to reason about. The division of labor: - **Access policies** express the decision. A policy takes a user or an identity provider group as its principal, conditions on resource tags and ownership, narrows to a single resource, and denies. Policies are how you grant access. - **RBAC** bootstraps and administers. It gets your first administrators in, and it is deliberately coarse: Admin, or nothing. RBAC is not the mechanism for granting day-to-day access in Agentic Data Plane. When you need to express something, express it as a policy. > 📝 **NOTE** > > Access policies are a preview capability, enabled per organization. Until they’re enabled for yours, an Admin binding or a custom role is the only way to reach an Agentic Data Plane API. See [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). > ❗ **IMPORTANT** > > Admin grants every Agentic Data Plane permission on every resource, so it is not a least-privilege answer for day-to-day users. Bind Admin to the people who administer the deployment. Grant everything else with a policy. ## [](#admin-is-the-only-built-in-role-that-reaches-agentic-data-plane)Admin is the only built-in role that reaches Agentic Data Plane Built-in roles span all of Redpanda Cloud. Only Admin reaches Agentic Data Plane. | Role | What it grants in Agentic Data Plane | | --- | --- | | Admin | Every Agentic Data Plane permission, on every resource. Bootstraps a deployment, and covers the operators who configure providers, agents, MCP servers, and pricing, and who author access policies. | | Writer, Reader | Nothing. Both roles keep their control-plane, Kafka, pipeline, and knowledge-base permissions, so a Writer can manage Redpanda Connect pipelines and knowledge bases and still can’t read, create, or invoke an agent, MCP server, or LLM provider. | | PipelineInvoker and the Kafka and Schema Registry roles | Nothing. These cover Redpanda Connect pipelines, topics, consumer groups, and Schema Registry subjects. | There are no Agentic Data Plane Invoker or transcript-reader roles to assign. Runtime-only access for a service account (calling an MCP tool, proxying an LLM request) and read access to conversation transcripts come from policies you write, not roles you pick from a list. Redpanda used to ship narrow built-in roles for exactly those cases and no longer provisions them, because a policy does the same job with a condition and a scope attached. > 📝 **NOTE** > > If your organization predates this change, those roles may still appear in your role list. Don’t build new grants on them. Author the equivalent policy instead. ## [](#grant-access-to-a-user-group-or-service-account)Grant access to a user, group, or service account Write a policy. The principal is a user or a group, so you can grant a whole identity provider group in one statement and let group membership stay managed in your IdP. For example, to let your support team call the tools on one MCP server and nothing else: ```cedar permit ( principal in Group::"support", action == Action::"McpServerTool.call", resource is McpServerTool in McpServer::"zendesk" ); ``` Policies also do the things a role bundle structurally can’t: condition on a resource tag, restrict access to a resource’s creator, or deny an action across every principal at once. See [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) for the authoring workflow, the actions a policy can name, and worked examples. Access policies use their own vocabulary. A policy names an entity type and a verb, such as `Action::"Agent.get"`, not the permission strings on this page. See [Action reference](https://docs.redpanda.com/agentic-data-plane/control/access-policies/#action-reference). A custom role holding Agentic Data Plane permissions still works. Reach for one when you need a permission bundle bound at a control-plane scope, or for the `dataplane_aiagent_a2a_*` permissions, which no policy action covers. ## [](#agentic-data-plane-permissions)Agentic Data Plane permissions Permissions are the RBAC-side vocabulary. You need them to read what Admin grants and to define a custom role. They are not what a policy names. Agentic Data Plane permissions live in these families: - `dataplane_adp_mcpserver_*`: Manage and call MCP servers (CRUD plus runtime operations like `tools_call` and `resources_read`). - `dataplane_adp_llmprovider_*`: Manage LLM providers and proxy LLM requests through AI Gateway. The `_invoke` permission is what your applications need at runtime. - `dataplane_adp_agent_*` and `dataplane_adp_agent_credential_*`: Manage declarative AI agents and the OIDC credentials issued to them. - `dataplane_adp_policy_*` and `dataplane_adp_policytemplate_*`: Author and read access policies and policy templates. - `dataplane_adp_transcript_*`: Read agent conversation transcripts, which carry full conversation content. - `dataplane_adp_auditlog_*`: Read audit log events, which record who attempted an action, on which resource, and whether it was allowed. - `dataplane_adp_spending_*`: Read AI spending data for governance and cost reporting. - `dataplane_aiagent_a2a_*`: Invoke agent-to-agent (A2A) operations against an agent’s own ingress. These are the one Agentic Data Plane family a policy can’t govern; see [A2A runtime permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#a2a-runtime-permissions). - `dataplane_aigateway_*`: Manage enterprise AI Gateway features, including OAuth providers and clients, model providers, rate and spend limits, audit, and pricing. Admin is the only built-in role that holds any of them. The legacy `dataplane_mcpserver_*` namespace, which older proto versions still enforce, is scoped the same way. Two more families belong to Redpanda Connect resources that Agentic Data Plane uses. These are owned outside Agentic Data Plane, so they stay in the Writer and Reader roles and are outside what a policy can govern: - `dataplane_pipeline_*`: Manage and invoke Redpanda Connect pipelines. - `dataplane_knowledgebase_*`: Manage retrieval-augmented generation knowledge bases. For the full list, see [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/). ## [](#identity-and-impersonation)Identity and impersonation Agentic Data Plane API calls authenticate as one of two identity types: - **User identity (OIDC)**: A human user signed in through Redpanda’s OIDC provider. The Agentic Data Plane UI uses this identity when an admin manages Agentic Data Plane resources interactively. - **Service account**: A non-human identity backed by OIDC client credentials. Applications, CI jobs, and `rpk ai` use service accounts. Both are policy principals. A service account is granted access the same way a user is, with a policy naming it as the principal. Account impersonation, which lets Redpanda Console reuse a user’s identity for Kafka API and Schema Registry calls, applies to those two subsystems only. Agentic Data Plane endpoints continue to authenticate clients directly through their OIDC tokens, regardless of how account impersonation is configured at the cluster level. ## [](#where-agentic-data-plane-fits-in-redpanda-cloud-rbac)Where Agentic Data Plane fits in Redpanda Cloud RBAC Agentic Data Plane permissions are part of the same Redpanda Cloud RBAC system that gates control-plane resources such as resource groups, networks, and clusters. They are scoped more tightly than the rest of it. The Writer and Reader roles bundle control-plane permissions with dataplane Kafka permissions, so a Writer binding at the organization scope covers those layers in one grant. It does not reach Agentic Data Plane. A developer who holds Writer on the organization can create a cluster and manage topics and pipelines, and still gets a permission-denied error from every agent, MCP server, and LLM provider API until a policy grants that access. Admin spans every layer, including Agentic Data Plane. ## [](#next-steps)Next steps - [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) - [Roles and Permissions Reference](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/) --- # Page 62: Roles and Permissions Reference **URL**: https://docs.redpanda.com/agentic-data-plane/control/permissions-reference.md --- # Roles and Permissions Reference > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Roles and Permissions Reference latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: permissions-reference page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: permissions-reference.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/control/pages/permissions-reference.adoc description: Look up the permissions Agentic Data Plane enforces across LLM providers, MCP servers, agents, access policies, pipelines, knowledge bases, and governance APIs. page-topic-type: reference personas: security_compliance_lead, platform_engineer learning-objective-1: Look up an Agentic Data Plane permission by namespace learning-objective-2: Identify the operation each permission gates learning-objective-3: Map a denied API call to the permission it enforces page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-06" --- Every Redpanda Agentic Data Plane API call enforces a single permission. This reference lists the Agentic Data Plane-namespaced permissions and the operation each one gates. Use this reference to: - Look up an Agentic Data Plane permission by namespace - Identify the operation each permission gates - Map a denied API call to the permission it enforces > ❗ **IMPORTANT** > > These permission strings are the role-based access control (RBAC) vocabulary. They are not what you write to grant access. > > Agentic Data Plane access is granted with [access policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/), which name their own actions, such as `Action::"Agent.get"`. See [Action reference](https://docs.redpanda.com/agentic-data-plane/control/access-policies/#action-reference) for the list. Use this page to read what the Admin role covers, to map an API operation to the permission it enforces when you’re debugging a denial, or to define a custom role. For how policies and roles fit together, see [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/). ## [](#how-to-read-this-reference)How to read this reference Each table column means the same thing across every namespace. - The Permission column is the exact string the API enforces. - The Operation column is the user-facing action this permission gates. Admin is the only built-in role that grants any of the permissions on this page, and it grants all of them, so there is no per-role column to read. Writer, Reader, and the task-specific built-in roles grant none of them. Two Redpanda Connect namespaces behave differently. [Pipeline](#pipeline-permissions) and [knowledge base](#knowledge-base-permissions) permissions are owned outside Agentic Data Plane, so they remain part of Writer and Reader and keep their role columns. ## [](#mcp-server-permissions)MCP server permissions The `dataplane_adp_mcpserver_*` permissions gate both server management (CRUD) and the MCP protocol calls a client makes against a running server. | Permission | Operation | | --- | --- | | dataplane_adp_mcpserver_create | Register a new MCP server | | dataplane_adp_mcpserver_update | Modify an existing MCP server’s configuration | | dataplane_adp_mcpserver_delete | Delete an MCP server | | dataplane_adp_mcpserver_get | View one MCP server’s configuration | | dataplane_adp_mcpserver_list | List MCP servers | | dataplane_adp_mcpserver_initialize | Initialize an MCP session against a server | | dataplane_adp_mcpserver_ping | Health-check an MCP server | | dataplane_adp_mcpserver_resources_list | List resources a server exposes | | dataplane_adp_mcpserver_resources_templates_list | List resource templates a server exposes | | dataplane_adp_mcpserver_resources_read | Read a resource from a server | | dataplane_adp_mcpserver_prompts_list | List prompts a server exposes | | dataplane_adp_mcpserver_prompts_get | Retrieve a prompt from a server | | dataplane_adp_mcpserver_tools_list | List tools a server exposes | | dataplane_adp_mcpserver_tools_call | Invoke a tool on an MCP server | | dataplane_adp_mcpserver_logging_set_level | Adjust an MCP server’s log level | A legacy `dataplane_mcpserver_*` namespace mirrors these permissions and is still enforced by older proto versions. It is scoped the same way: Admin grants both namespaces, and no other built-in role grants either. ## [](#llm-provider-permissions)LLM provider permissions The `dataplane_adp_llmprovider_*` permissions gate AI Gateway provider configuration and the runtime proxy that forwards LLM requests upstream. | Permission | Operation | | --- | --- | | dataplane_adp_llmprovider_create | Create an LLM provider | | dataplane_adp_llmprovider_get | View one LLM provider’s configuration | | dataplane_adp_llmprovider_list | List LLM providers | | dataplane_adp_llmprovider_update | Modify an LLM provider’s configuration | | dataplane_adp_llmprovider_delete | Delete an LLM provider | | dataplane_adp_llmprovider_invoke | Proxy LLM requests through AI Gateway at runtime | The `dataplane_adp_llmprovider_invoke` permission is all an application needs to send traffic through AI Gateway. It is the narrowest useful grant for a service account, and no built-in role provides it on its own. Grant it with an access policy naming `Action::"LLMProvider.invoke"`. ## [](#agent-management-permissions)Agent management permissions The `dataplane_adp_agent_*` permissions gate declarative agent configuration. | Permission | Operation | | --- | --- | | dataplane_adp_agent_create | Create a declarative agent | | dataplane_adp_agent_get | View one agent’s configuration | | dataplane_adp_agent_list | List agents | | dataplane_adp_agent_update | Modify an agent’s configuration | | dataplane_adp_agent_delete | Delete an agent | ## [](#agent-credential-permissions)Agent credential permissions The `dataplane_adp_agent_credential_*` permissions gate the OIDC client credentials an agent uses for outbound calls. | Permission | Operation | | --- | --- | | dataplane_adp_agent_credential_create | Issue a new OIDC client credential for an agent | | dataplane_adp_agent_credential_list | List an agent’s credentials | | dataplane_adp_agent_credential_delete | Revoke an agent credential | ## [](#agent-trigger-permissions)Agent trigger permissions The `dataplane_adp_agent_trigger_*` permissions gate triggers that start an agent in response to an external event, such as an incoming message or a schedule. See [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/). | Permission | Operation | | --- | --- | | dataplane_adp_agent_trigger_create | Create a trigger on an agent | | dataplane_adp_agent_trigger_get | View a trigger on an agent | | dataplane_adp_agent_trigger_list | List triggers on agents | | dataplane_adp_agent_trigger_update | Modify a trigger on an agent | | dataplane_adp_agent_trigger_delete | Delete a trigger on an agent | A further `dataplane_adp_agent_trigger_report_health` permission lets internal trigger observers report trigger health. Only Admin grants it, and nothing tenant-facing needs it. ## [](#agent-session-permissions)Agent session permissions The `dataplane_adp_agent_session_*` permissions gate access to an agent’s conversation sessions: the persisted conversation threads a managed agent accumulates as it runs. Sessions exist for managed agents only; a self-managed agent keeps no session store. The runtime creates sessions, so there is no create or update permission. | Permission | Operation | | --- | --- | | dataplane_adp_agent_session_list | List an agent’s conversation sessions (summaries only) | | dataplane_adp_agent_session_get | View one conversation session, including its full message transcript | | dataplane_adp_agent_session_delete | Delete an agent’s conversation sessions | > 📝 **NOTE** > > Granting `dataplane_adp_agent_session_get` exposes a session’s full conversation content (prompts, tool inputs and outputs, and model output), the same class of content that transcript access exposes. Treat it as sensitively as the transcript permissions when a policy grants it. ## [](#transcript-permissions)Transcript permissions The `dataplane_adp_transcript_*` permissions gate read access to agent conversation transcripts. Transcripts carry the full content of an agent’s conversations (system prompts, user messages, tool arguments, and model output). See [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) for what a transcript records. | Permission | Operation | | --- | --- | | dataplane_adp_transcript_get | View a single agent conversation transcript | | dataplane_adp_transcript_list | List agent conversation transcripts | > 📝 **NOTE** > > Transcript access is separate from agent read access. A principal that can view an agent’s configuration through `dataplane_adp_agent_get` cannot read that agent’s transcripts. Grant transcript access deliberately, with a policy naming the `Transcript` actions, to the users and service accounts that need to read conversation content. ## [](#audit-log-permissions)Audit log permissions The `dataplane_adp_auditlog_*` permissions gate read access to authorization events. Audit events can include user identities, resource names, network information, configuration changes, and captured request or response data. See [Review the Audit Log](https://docs.redpanda.com/agentic-data-plane/monitor/audit-log/) for how to review them. | Permission | Operation | | --- | --- | | dataplane_adp_auditlog_list | List audit log events and view an event’s details | > 📝 **NOTE** > > The built-in **Read only** template grants `dataplane_adp_auditlog_list`, along with every template built on top of it. To let a principal read resource configuration without reading the audit log, write your own template or policy that omits `AuditLog.list` rather than linking a built-in template. ## [](#access-policy-permissions)Access policy permissions The `dataplane_adp_policy_*` and `dataplane_adp_policytemplate_*` permissions gate the access policies and policy templates themselves. See [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). | Permission | Operation | | --- | --- | | dataplane_adp_policy_create | Create an access policy | | dataplane_adp_policy_get | View one access policy | | dataplane_adp_policy_list | List access policies | | dataplane_adp_policy_update | Modify an access policy | | dataplane_adp_policy_delete | Delete an access policy | | dataplane_adp_policytemplate_create | Create a policy template | | dataplane_adp_policytemplate_get | View one policy template | | dataplane_adp_policytemplate_list | List policy templates | | dataplane_adp_policytemplate_update | Modify a policy template | | dataplane_adp_policytemplate_delete | Delete a policy template | | dataplane_adp_systempolicy_list | List the policies derived from role bindings | | dataplane_adp_effectivepolicyset_get | View the policy set in effect for a principal | | dataplane_adp_effectivepolicyset_list | List effective policy sets | > ❗ **IMPORTANT** > > The permission to author access policies is itself an Agentic Data Plane permission, so Admin is the only built-in role that has it. Whoever manages access for your deployment needs Admin, or a policy that grants the `Policy` actions explicitly. Granting policy authoring is equivalent to granting everything a policy can grant. The `dataplane_adp_systempolicy_list` permission is separate from `dataplane_adp_policy_list` because the derived view names the principal of every role binding in the organization. ## [](#spending-permissions)Spending permissions The `dataplane_adp_spending_*` permissions gate the governance APIs that surface AI spend, request counts, and token volume. See [Set Up Budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) for what spending data Agentic Data Plane records automatically. | Permission | Operation | | --- | --- | | dataplane_adp_spending_get | Read AI spending data for governance and cost reports | ## [](#budget-permissions)Budget permissions The `dataplane_adp_budget_*` permissions gate per-agent LLM spend budgets. See [Set a budget](https://docs.redpanda.com/agentic-data-plane/control/budgets/#set-a-spend-limit). | Permission | Operation | | --- | --- | | dataplane_adp_budget_create | Create a budget | | dataplane_adp_budget_get | View a budget and its current-period spend | | dataplane_adp_budget_list | List budgets | | dataplane_adp_budget_update | Modify a budget | | dataplane_adp_budget_delete | Delete a budget | ## [](#guardrail-permissions)Guardrail permissions The `dataplane_adp_guardrail_*` permissions gate guardrail policies that screen LLM requests and responses. See [How Guardrails Work](https://docs.redpanda.com/agentic-data-plane/control/guardrails/overview/). | Permission | Operation | | --- | --- | | dataplane_adp_guardrail_create | Create a guardrail policy | | dataplane_adp_guardrail_get | View one guardrail policy’s configuration | | dataplane_adp_guardrail_list | List guardrail policies | | dataplane_adp_guardrail_update | Modify a guardrail policy | | dataplane_adp_guardrail_delete | Delete a guardrail policy | ## [](#a2a-runtime-permissions)A2A runtime permissions The `dataplane_aiagent_a2a_*` permissions, along with `dataplane_adp_a2a_invoke`, gate agent-to-agent (A2A) runtime traffic. | Permission | Operation | | --- | --- | | dataplane_adp_a2a_invoke | Invoke an A2A agent | | dataplane_aiagent_a2a_message_send | Send a message to an agent | | dataplane_aiagent_a2a_message_stream | Open a streaming message connection to an agent | | dataplane_aiagent_a2a_tasks_get | Read one A2A task | | dataplane_aiagent_a2a_tasks_list | List A2A tasks | | dataplane_aiagent_a2a_tasks_cancel | Cancel an A2A task | | dataplane_aiagent_a2a_tasks_subscribe | Subscribe to A2A task events | | dataplane_aiagent_a2a_get_extended_agent_card | Read an agent’s extended agent card | The `dataplane_adp_a2a_invoke` permission covers the AI Gateway A2A proxy path, and a policy can grant it by naming `Action::"Agent.invoke"`. The `dataplane_aiagent_a2a_*` permissions are different. The agent’s own ingress enforces them against the caller’s RBAC permission set, and they have no access-policy action, so a policy can’t grant them. A service account that talks to an agent’s A2A endpoint directly needs Admin or a custom role holding this set. ## [](#pipeline-permissions)Pipeline permissions The `dataplane_pipeline_*` permissions gate Redpanda Connect pipelines used by Agentic Data Plane for ingestion and transformation. These are Redpanda Connect permissions rather than Agentic Data Plane permissions, so the Writer, Reader, and PipelineInvoker built-in roles grant them as shown. The PipelineInvoker role grants only the runtime invocation permissions. | Permission | Operation | Writer | Reader | Invoker | | --- | --- | --- | --- | --- | | dataplane_pipeline_create | Create a pipeline | ✓ | | | | dataplane_pipeline_get | View one pipeline’s configuration | ✓ | ✓ | | | dataplane_pipeline_list | List pipelines | ✓ | ✓ | | | dataplane_pipeline_update | Modify a pipeline’s configuration | ✓ | | | | dataplane_pipeline_delete | Delete a pipeline | ✓ | | | | dataplane_pipeline_start | Start a stopped pipeline | ✓ | | | | dataplane_pipeline_stop | Stop a running pipeline | ✓ | | | | dataplane_pipeline_gateway_invoke | Invoke a pipeline through the gateway endpoint | ✓ | | PipelineInvoker | | dataplane_pipeline_otlp_grpc_invoke | Send OTLP traces to a pipeline over gRPC | ✓ | | PipelineInvoker | | dataplane_pipeline_otlp_http_invoke | Send OTLP traces to a pipeline over HTTP | ✓ | | PipelineInvoker | Access policies can’t govern pipelines. The entity types a policy can name are listed in [Action reference](https://docs.redpanda.com/agentic-data-plane/control/access-policies/#action-reference), and pipelines aren’t among them. ## [](#knowledge-base-permissions)Knowledge base permissions The `dataplane_knowledgebase_*` permissions gate retrieval-augmented generation (RAG) knowledge bases. Like the pipeline permissions, these are Redpanda Connect permissions and remain part of Writer and Reader. | Permission | Operation | Writer | Reader | | --- | --- | --- | --- | | dataplane_knowledgebase_create | Create a knowledge base | ✓ | | | dataplane_knowledgebase_get | View one knowledge base’s configuration | ✓ | ✓ | | dataplane_knowledgebase_list | List knowledge bases | ✓ | ✓ | | dataplane_knowledgebase_update | Modify a knowledge base’s configuration | ✓ | | | dataplane_knowledgebase_delete | Delete a knowledge base | ✓ | | ## [](#built-in-roles-summary)Built-in roles summary | Role | Coverage of the permissions on this page | | --- | --- | | Admin | Every Agentic Data Plane permission on this page, plus the pipeline and knowledge-base permissions. Bind it to the operators who administer the deployment. | | Writer | No Agentic Data Plane permissions. Grants full management of pipelines and knowledge bases, alongside its control-plane and Kafka permissions. | | Reader | No Agentic Data Plane permissions. Grants _get and _list on pipelines and knowledge bases, alongside its control-plane and Kafka permissions. | | PipelineInvoker | No Agentic Data Plane permissions. Grants the three pipeline runtime invocation permissions. | Redpanda no longer provisions Agentic Data Plane-specific built-in roles for runtime invocation or transcript reads, and isn’t adding more. Grant that access with an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/). A custom role holding permissions from this page still works. Reach for one when you need a bundle bound at a control-plane scope, or for the `dataplane_aiagent_a2a_*` permissions, which no policy action covers. ## [](#next-steps)Next steps - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) - [Manage Access Policies](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) --- # Page 63: Routing & LLM Settings **URL**: https://docs.redpanda.com/agentic-data-plane/gateway.md --- # Routing & LLM Settings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Routing & LLM Settings latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/index.adoc description: Configure AI Gateway, LLM providers, and routing. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-06-01" --- AI Gateway is the proxy that routes agent traffic to your LLM providers. Learn how it works, configure providers, and connect your agents to the gateway. - [How AI Gateway Works](overview/) AI Gateway is Agentic Data Plane's managed proxy for LLM APIs. Create a provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint, and point your applications at a Redpanda-hosted URL with managed secrets, authentication, and observability. - [Configure an LLM Provider](configure-provider/) Create an LLM provider to proxy requests to OpenAI, Anthropic, Google AI, AWS Bedrock, or any OpenAI-compatible endpoint through Redpanda Agentic Data Plane. - [Connect Your App to AI Gateway](connect-agent/) Point your application or AI agent at an AI Gateway provider's proxy URL. Covers the URL shape, the local development workflow with rpk ai, the OIDC client-credentials flow for CI and application code, and SDK examples for OpenAI, Anthropic, Google AI, AWS Bedrock, and OpenAI-compatible endpoints. --- # Page 64: Set Up AWS Bedrock as an LLM Provider **URL**: https://docs.redpanda.com/agentic-data-plane/gateway/bedrock-setup.md --- # Set Up AWS Bedrock as an LLM Provider > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Set Up AWS Bedrock as an LLM Provider latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: bedrock-setup page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: bedrock-setup.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/bedrock-setup.adoc description: Create the IAM user, policy, and access keys required for AI Gateway to invoke Amazon Bedrock models, then register the provider in Agentic Data Plane. page-topic-type: how-to personas: platform_engineer learning-objective-1: Create an IAM policy that grants AI Gateway permission to invoke Bedrock foundation models and cross-region inference profiles learning-objective-2: Create a dedicated IAM user, attach the policy, and generate access keys for AI Gateway learning-objective-3: Register Bedrock as an LLM provider in Agentic Data Plane and select the models you want to expose page-git-created-date: "2026-05-28" page-git-modified-date: "2026-07-30" --- This guide walks you through the AWS-side setup AI Gateway needs to invoke Amazon Bedrock, then through the Redpanda Agentic Data Plane flow that registers Bedrock as an LLM provider. For background on how Bedrock foundation models, cross-region inference profiles, and IAM patterns map to the provider form, see [AWS Bedrock: Inference profiles and IAM](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#bedrock-inference-profiles) on the main provider configuration page. After completing this guide, you will be able to: - Create an IAM policy that grants AI Gateway permission to invoke Bedrock foundation models and cross-region inference profiles - Create a dedicated IAM user, attach the policy, and generate access keys for AI Gateway - Register Bedrock as an LLM provider in Agentic Data Plane and select the models you want to expose ## [](#prerequisites)Prerequisites - An AWS account with Bedrock model access enabled in the region you plan to call. Model availability varies by region; see [Bedrock models by region](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html). - Access to the AWS CLI configured with credentials that can create IAM users, policies, and access keys. - Access to the Redpanda UI. ## [](#create-the-iam-policy)Create the IAM policy Create a policy that grants the two Bedrock invoke actions on both foundation-model ARNs and cross-region inference-profile ARNs: ```bash aws iam create-policy \ --policy-name RedpandaBedrockInvoke \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Sid": "BedrockInvoke", "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": [ "arn:aws:bedrock:*::foundation-model/*", "arn:aws:bedrock:*:*:inference-profile/*" ] } ] }' ``` The second resource entry enables cross-region inference profiles such as `us.anthropic.claude-sonnet-4-6`, which AI Gateway uses when the model identifier carries a geography prefix. See [AWS Bedrock: Inference profiles and IAM](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#bedrock-inference-profiles) for the full prefix list and pricing implications. > 📝 **NOTE** > > Anthropic Claude 4.6 and later models cannot be invoked with the bare foundation-model ID and require an inference profile. Without the second `Resource` entry, those calls fail with `AccessDenied`. To restrict the policy to specific models and regions for production, replace the wildcard resources with explicit ARNs. For example: ```json { "Resource": [ "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-6", "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-haiku-4-5-20251001" ] } ``` ## [](#create-the-iam-user)Create the IAM user Create a dedicated IAM user for AI Gateway and attach the policy: ```bash aws iam create-user --user-name redpanda-bedrock-invoker aws iam attach-user-policy \ --user-name redpanda-bedrock-invoker \ --policy-arn arn:aws:iam:::policy/RedpandaBedrockInvoke ``` Replace `` with the account ID returned in the `create-policy` output (visible in the policy ARN). > 💡 **TIP** > > Don’t reuse an existing IAM user. A dedicated user makes it easy to rotate credentials or revoke access without affecting other AWS workloads. ## [](#generate-access-keys)Generate access keys Generate the access keys AI Gateway uses: ```bash aws iam create-access-key --user-name redpanda-bedrock-invoker ``` Save the `AccessKeyId` and `SecretAccessKey` from the output. You need both in the next section to register them as Redpanda secrets. > ⚠️ **CAUTION** > > AWS displays the secret access key only at creation. Store it in a password manager or pass it directly into the secret-creation flow in the next section. ## [](#verify-bedrock-access-optional)Verify Bedrock access (optional) Confirm the IAM user can invoke Bedrock before moving to the UI: ```bash aws bedrock-runtime invoke-model \ --model-id us.anthropic.claude-haiku-4-5-20251001-v1:0 \ --region us-east-1 \ --content-type application/json \ --accept application/json \ --body "$(echo -n '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"Hello"}]}' | base64)" \ /tmp/bedrock-test.json \ && jq . /tmp/bedrock-test.json && rm /tmp/bedrock-test.json ``` A successful model response confirms the IAM policy, region, and credentials are correct. If you see `AccessDenied`, check the policy resource list and confirm Bedrock model access is enabled in the target region. ## [](#register-bedrock-as-an-llm-provider)Register Bedrock as an LLM provider 1. Open **LLM Providers** in the sidebar and click **Add provider**. 2. Select **AWS Bedrock** as the provider type. 3. Enter a Name such as `my-bedrock`. Use lowercase letters, digits, and hyphens. The name is immutable and appears in the proxy URL. 4. Select the Region where you want to invoke Bedrock, such as `us-east-1`. 5. For Credential type, select **Static keys**. (This guide uses the access keys you created above. For the default-chain and assume-role options, see [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/).) 6. Configure the credentials: 1. In the Access key ID ref dropdown, type a secret name such as `AWS_ACCESS_KEY_ID`. 2. Paste the `AccessKeyId` value from the IAM user setup and click **Create**. The secret is stored in the Agentic Data Plane secret store, scoped to AI Gateway. 3. Repeat for Secret access key ref. Use a name such as `AWS_SECRET_ACCESS_KEY` and paste the `SecretAccessKey` value. Secret names are normalized to `UPPER_SNAKE_CASE` automatically and get the AI Gateway scope, which makes them usable across the platform: LLM providers, MCP servers, and agents. 7. Select the models you want to expose through this provider, for example: - `anthropic.claude-sonnet-4-6` - `anthropic.claude-haiku-4-5-20251001` - `amazon.nova-pro-v1:0` For Anthropic Claude 4.6 and later, pick the inference profile (for example, `us.anthropic.claude-sonnet-4-6`) rather than the bare foundation-model ID. 8. Click **Create provider**. 9. Verify the provider by clicking **Test connection** on its edit page. A successful response confirms that the credentials, region, and IAM policy are correctly configured. ## [](#cross-region-inference-profile-billing)Cross-region inference profile billing When you call a cross-region inference profile (any model identifier with a `us.`, `eu.`, `apac.`, `au.`, `jp.`, or `global.` prefix), AI Gateway bills at the regional rate for that profile. The regional prefix is preserved end to end so usage on the **Cost & Usage** page under **Governance** reflects the correct per-region price. For example, requests to `eu.anthropic.claude-haiku-4-5` bill at the EU Haiku rate, not the headline foundation-model rate. The `global.` profile shares the headline rate; the geography-specific profiles (`us.`, `eu.`, `apac.`, `au.`, `jp.`) carry approximately a 10% cross-region inference premium. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | AccessDenied from Bedrock | Confirm the IAM policy includes both bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream, and that the resource list covers the model or inference profile you’re calling. For Claude 4.6 and later, the policy must include arn:aws:bedrock:*:*:inference-profile/* or an explicit inference-profile ARN. | | secret "" not found | Confirm the secret exists in the cloud secret store and the reference in the provider configuration matches exactly. Secret names are UPPER_SNAKE_CASE. | | ValidationException: model ID not supported | The model isn’t enabled in the region you chose. Open the AWS Bedrock console, switch to the target region, and enable model access for the foundation models you want to expose. | | Invocation of model ID … with on-demand throughput isn’t supported | You called a Claude 4.6 or later model with a bare foundation-model ID. Switch to an inference profile, for example us.anthropic.claude-sonnet-4-6 instead of anthropic.claude-sonnet-4-6. See AWS Bedrock: Inference profiles and IAM. | ## [](#next-steps)Next steps - [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) - [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) --- # Page 65: Code Mode **URL**: https://docs.redpanda.com/agentic-data-plane/gateway/code-mode.md --- # Code Mode > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Code Mode latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: code-mode page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: code-mode.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/code-mode.adoc description: Turn on code mode for an MCP server to replace a large tool catalog with two tools, search and execute, and cut the token cost of tool-heavy servers. page-topic-type: how-to personas: agent_builder, platform_engineer learning-objective-1: Explain how code mode reduces tool-token usage for an MCP server learning-objective-2: Enable code mode on an MCP server and find its code-mode endpoint learning-objective-3: Use the search and execute tools to find and call tools through code mode page-git-created-date: "2026-06-09" page-git-modified-date: "2026-07-29" --- Enable code mode on an MCP server to cut the token cost of serving a large tool catalog. Instead of loading every tool definition into context, agents search for the tools they need and run them through a lightweight JavaScript sandbox. After reading this page, you will be able to: - Explain how code mode reduces tool-token usage for an MCP server - Enable code mode on an MCP server and find its code-mode endpoint - Use the search and execute tools to find and call tools through code mode > 📝 **NOTE** > > Code mode applies to one MCP server at a time. It is a token-reduction technique for a server with many tools, not a way to combine multiple servers behind one endpoint. ## [](#how-code-mode-works)How code mode works When code mode is enabled on an MCP server, the AI Gateway serves a virtual sibling endpoint alongside the server’s normal one. If the server is reachable at `https://aigw..clusters.rdpa.co/mcp/v1/`, its code-mode endpoint is the same path with a `-code` suffix: ```none https://aigw..clusters.rdpa.co/mcp/v1/-code ``` That endpoint exposes exactly two tools instead of the server’s full catalog: `search` Find tools in the underlying server’s catalog. Takes an optional `query`, a regular expression (Go RE2 syntax) matched against each tool’s name and description. It returns the full schema (name, description, and input schema) of every match, or null when nothing matches. Omit the query to list the entire catalog, which is large, so prefer a narrow regex. `execute` Run JavaScript in a sandbox that is connected to the same MCP server. The value of the last expression in your code is returned as the tool result. Inside the sandbox, two synchronous host functions are available: - `call_tool({name, arguments})`: Invoke a tool on the server and return its output directly. It throws if the tool returns an error. JSON output is parsed into the matching JavaScript value; anything else is returned as a string. - `search_tools(query)`: Search the catalog by regex, the same as the `search` tool. A typical interaction is: call `search` with a narrow regex to find candidate tools, read the returned input schema, then call `execute` with code that invokes one or more of those tools. ### [](#sandbox-constraints)Sandbox constraints The `execute` sandbox is isolated and intentionally limited: - Code is capped at 64 KiB. - A single `execute` call can make at most 50 tool calls. - Each sandbox has a memory limit and a runtime limit. - The `call_tool` and `search_tools` host functions are synchronous, so do not use `await` on them. Top-level `await` and top-level `return` are syntax errors, promises are not awaited (returning one yields an empty result), and `console.log` output is discarded. Calls that code mode makes to the underlying server run with the same identity and authentication as the server itself, including any per-user [token vault](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#token-vault) credentials. Code mode does not widen what the server can reach. ## [](#when-to-use-code-mode)When to use code mode Turn on code mode when: - A server exposes a large tool catalog, so loading every tool definition on each request is expensive or pushes out other context. - An agent frequently performs multi-step tool sequences that you would rather run in one round trip instead of several model turns. Leave it off for servers with only a handful of tools, where the overhead of searching and writing code outweighs the token savings of a small catalog. ## [](#enable-code-mode)Enable code mode Enable code mode on the MCP server, then point your agent or client at the code-mode endpoint. 1. Open **MCP Servers** in the sidebar. 2. Create a server, or open an existing one to edit it. See [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/). 3. Turn on the **Code mode** toggle. 4. Save the server. The server’s detail page shows the code-mode endpoint URL (the primary URL with a `-code` suffix). 5. Configure your agent or MCP client to connect to the code-mode URL instead of the primary URL. ## [](#test-from-the-cli)Test from the CLI The `--code-mode` flag on `rpk ai mcp-server tools` targets the virtual `-code` endpoint instead of the server’s primary endpoint, so you can exercise `search` and `execute` directly. List the code-mode tools: ```bash rpk ai mcp-server tools list --code-mode ``` Search the underlying catalog: ```bash rpk ai mcp-server tools call search --code-mode \ --args '{"query":"(?i)pull.*create"}' ``` Run code through the `execute` tool: ```bash rpk ai mcp-server tools call execute --code-mode \ --args '{"code":"var pulls = call_tool({ name: \"github_pulls_list\", arguments: { owner: \"redpanda-data\", repo: \"redpanda\", state: \"open\" } }); JSON.stringify(pulls.map(function (p) { return { number: p.number, title: p.title }; }));"}' ``` The code calls a tool, shapes the result, and returns the value of its last expression. Because `call_tool` is synchronous, a single `execute` call can search, call several tools, and combine the results without extra model round trips. ## [](#next-steps)Next steps - [Create an MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/create-server/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - [Test MCP Tools](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 66: Configure an LLM Provider **URL**: https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider.md --- # Configure an LLM Provider > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Configure an LLM Provider latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: configure-provider page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: configure-provider.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/configure-provider.adoc description: Create an LLM provider to proxy requests to OpenAI, Anthropic, Google AI, AWS Bedrock, or any OpenAI-compatible endpoint through Redpanda Agentic Data Plane. page-topic-type: how-to personas: platform_engineer, agent_builder learning-objective-1: Create an LLM provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint learning-objective-2: Select the models you want to expose through the provider learning-objective-3: Verify the provider is reachable using the built-in Test connection control page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-11" --- Create an LLM provider to give your applications a managed proxy URL: Redpanda handles the upstream API keys, forwards requests to the provider, and records usage for you. Create a provider for each upstream you use, whether that’s OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint. After reading this page, you will be able to: - Create an LLM provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint - Select the models you want to expose through the provider - Verify the provider is reachable using the built-in Test connection control ## [](#prerequisites)Prerequisites - An API key (or AWS credentials for Bedrock) for the upstream provider you want to configure. - Optional: Secrets already created in your dataplane’s secret store for the provider’s credentials. The create form can also store a new API key inline, so you don’t have to prepare a secret first. Secret identifiers use `UPPER_SNAKE_CASE`. For example: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `AWS_ACCESS_KEY_ID`. ## [](#open-the-create-llm-provider-page)Open the Create LLM provider page 1. Open **LLM Providers** in the sidebar. 2. Click **Add provider**. ## [](#fill-in-the-identity-fields)Fill in the identity fields The Identity section collects the provider’s name. Enter a `Display name`: a human-readable label shown in dashboards and model selectors, up to 253 characters. Agentic Data Plane derives the provider’s resource ID from the display name in the background; the create form doesn’t show it. The resource ID is the machine identifier used in API calls and CLI commands: lowercase letters, numbers, and hyphens only (`^[a-z][a-z0-9-]*$`), up to 63 characters, and immutable after creation. It appears in the proxy URL (`/llm/v1/providers//…​`). ## [](#choose-a-provider-type)Choose a provider type In the Provider type section, pick the type that matches your upstream. | Type | Use when | | --- | --- | | OpenAI | Proxy GPT, o-series, and embeddings through the OpenAI API. Best when you already hold an OpenAI API key or want the broadest GPT model catalog. | | Anthropic | Call Claude Opus, Sonnet, and Haiku directly. Strong at coding, long-context reasoning, and tool use. Supports forwarding client Authorization headers to Anthropic for enterprise and Max-plan subscription passthrough (see Anthropic: Authorization passthrough). | | Google AI | Reach Gemini Pro, Flash, and multimodal models through Google AI Studio. Ideal for long-context workloads and image/video inputs. | | AWS Bedrock | Invoke foundation models (Claude, Llama, Titan, Nova, Mistral, AI21 Jamba, Gemma) hosted inside your AWS account. Requires an AWS region and credentials (static, STS-assumed role, or the default credential chain). Supports the native Bedrock APIs (InvokeModel, Converse) and an OpenAI-compatible Chat Completions endpoint for gpt-oss and Gemma models. See AWS Bedrock: Inference profiles and IAM for picking the right model identifier, and Set up AWS Bedrock as an LLM provider for a step-by-step IAM and access-key walkthrough. | | OpenAI-compatible | Point at any OpenAI-compatible endpoint that ships /v1/chat/completions (vLLM, Ollama, LM Studio, LocalAI, Together, Groq, OpenRouter). Useful for self-hosted models and aggregator gateways. Requires a Base URL. Authentication is optional. | Selecting a type reveals the type-specific configuration fields. ## [](#fill-in-the-type-specific-configuration)Fill in the type-specific configuration Each `API key reference` and credential field points at a secret-store entry, not the secret value itself. On the create form, the Credential section lists the keys already in your dataplane’s secret store: pick one, or select `Bring a new API key reference` to store a new key inline. Enter a `Key name` and the `API key` value, then click **Save key**. Agentic Data Plane stores the key in the secret store under an identifier derived from the key name, shown under the field; the value is stored encrypted and can’t be viewed again after saving. When you edit a provider later, the `API key reference` field offers an **Existing** tab to pick a stored secret and a **New** tab to create one inline. ### OpenAI | Field | Notes | | --- | --- | | Base URL | Optional. Leave empty for the standard OpenAI API (https://api.openai.com/v1). Override for Azure OpenAI or other OpenAI-hosted endpoints. | | API key reference | Required. Secret-store reference for the OpenAI API key. Must be UPPER_SNAKE_CASE, for example OPENAI_API_KEY. | ### Anthropic | Field | Notes | | --- | --- | | Base URL | Optional. Leave empty for the standard Anthropic API (https://api.anthropic.com). | | API key reference | Required unless Authorization passthrough is on. UPPER_SNAKE_CASE, for example ANTHROPIC_API_KEY. | | Authorization passthrough | Optional toggle. When on, AI Gateway forwards the client’s Authorization header to Anthropic instead of using a server-side API key. Used for enterprise and Max-plan OAuth passthrough: each client authenticates with its own Anthropic subscription. Leave the API key reference empty when using passthrough. | ### Google AI | Field | Notes | | --- | --- | | Base URL | Optional. Leave empty for the standard Google AI API (https://generativelanguage.googleapis.com). | | API key reference | Required. Secret-store reference for the Google AI API key. UPPER_SNAKE_CASE, for example GOOGLE_AI_API_KEY. | > ❗ **IMPORTANT** > > Gemini uses the `x-goog-api-key` header for authentication, not `Authorization: Bearer`. This matters when you wire up clients. See [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/). ### AWS Bedrock | Field | Notes | | --- | --- | | Region | Required. AWS region where the Bedrock endpoint is deployed, for example us-east-1. | | Base URL | Optional. Override the default regional Bedrock endpoint. | | Credential type | How AI Gateway authenticates to Bedrock: Default chain, Static keys, or Assume IAM role. The fields below depend on the mode you pick. | | Access key ID reference | Static keys only. Secret-store reference for the AWS access key ID, UPPER_SNAKE_CASE (typically AWS_ACCESS_KEY_ID). | | Secret access key reference | Static keys only. Secret-store reference for the AWS secret access key, UPPER_SNAKE_CASE (typically AWS_SECRET_ACCESS_KEY). | | Role ARN | Assume IAM role only. Required. ARN of the IAM role AI Gateway assumes through AWS STS, for example arn:aws:iam::123456789012:role/BedrockRole. | | External ID | Assume IAM role only. Optional. External ID for cross-account role assumption. Set it only when the role’s trust policy mandates an external ID. | | Session name | Assume IAM role only. Optional. Session name that appears in AWS CloudTrail audit logs, for example redpanda-adp. | | Guardrail | Optional. Name of a guardrail to attach to this provider, or empty for none. Only the Bedrock provider type exposes this setting. AI Gateway validates the name when you save: it rejects a guardrail that doesn’t exist or is being deleted, so set the field to an existing guardrail or leave it empty. See Create a guardrail. | Pick a **Credential type** to control how AI Gateway authenticates to Bedrock: - **Default chain** (default): Leave the credentials unset to use the AWS SDK’s default provider chain (environment variables, shared config, EKS Pod Identity, IRSA, or instance profile). Use this when the gateway already runs with an AWS identity. - **Static keys**: An access key pair stored in the secret store. Use this when no ambient AWS identity is available. This is the path the [Bedrock setup guide](https://docs.redpanda.com/agentic-data-plane/gateway/bedrock-setup/) walks through. - **Assume IAM role**: AI Gateway assumes an IAM role through AWS STS. Use this for cross-account access or when your security policy requires short-lived credentials. ### OpenAI-compatible | Field | Notes | | --- | --- | | Base URL | Required. URL of your OpenAI-compatible endpoint, for example http://vllm.internal:8000/v1, http://ollama.local:11434/v1, or an aggregator like Together / Groq / OpenRouter. | | API key reference | Optional. Leave empty for endpoints with no authentication (common for local runtimes). UPPER_SNAKE_CASE if set. | > 💡 **TIP** > > OpenAI-compatible endpoints can serve any model. Enter the exact model identifiers your upstream server exposes (for example, `meta-llama/Llama-3.3-70B-Instruct` or `qwen3:8b`). > 📝 **NOTE** > > For the OpenAI, Google AI, and AWS Bedrock provider types, AI Gateway validates that the credential references resolve before it accepts the create or update. AI Gateway rejects a missing or empty secret reference at save time instead of failing at first call. The OpenAI-compatible type does not require a credential reference, so it can be created with no authentication for local runtimes such as Ollama or vLLM. ## [](#select-models)Select models Models you select on this form become the catalog the provider exposes. For **OpenAI**, **Anthropic**, **Google AI**, and **AWS Bedrock**, the form shows a picker backed by the provider’s catalog, with the full catalog selected by default. Keep the default, or deselect models to narrow what the provider exposes. Each model in the picker shows its input and output price per million tokens. Pick from the list, or type a model identifier the catalog doesn’t show. For **OpenAI-compatible**, the form takes a freeform list: type the exact identifiers your upstream serves. Redpanda maintains the catalog of available models in the picker. When an upstream provider publishes a new model, it usually appears in the picker within a day or two; admins don’t have to wait for a Redpanda release. New models aren’t enabled automatically: an admin still selects the model in the catalog to make it callable through this provider. For Bedrock, the picker exposes inference profiles, not raw foundation-model IDs. See [AWS Bedrock: Inference profiles and IAM](#bedrock-inference-profiles). > 📝 **NOTE** > > Redpanda stores models as structured `ProviderModel` entries (one entry per model, with the model name as the only required field). Each model can carry custom pricing overrides that replace the catalog rates for that model in cost reporting; see [Override per-model pricing](#pricing-overrides). The legacy flat `models` field still works on writes for backward compatibility. Beyond pricing, the catalog carries each model’s capabilities and context-window limits. The model discovery API (the `ListModels` and `GetModel` methods on `ModelService`) reports `max_input_tokens` (the largest context the model accepts) and `max_output_tokens` (the most it can generate in a single response) for each model. Both are read-only catalog metadata: a limit the catalog doesn’t declare stays unset rather than reported as zero. Clients read `max_input_tokens` to show how full an agent’s context window is, such as the context-window indicator on the agent’s **Playground** tab. ### [](#pricing-overrides)Override per-model pricing Cost reporting prices each call at the catalog rates for the model. If your organization negotiates non-standard rates, or you track spend against an internal chargeback rate, override the rates per model on this provider. In the model picker, each selected model carries a pencil icon (Override pricing). Click it to open the pricing dialog for that model. The dialog lists one field per billing bucket, in the same order as the provider’s published rate card: | Bucket | What it bills | | --- | --- | | Input | Per 1M input tokens. Tool-use input also bills at this rate. | | Output | Per 1M output tokens. Reasoning tokens also bill at this rate. | | Cached input | Per 1M tokens read from prompt cache. | | Cache write (5-minute TTL) | Per 1M tokens written to a 5-minute prompt cache. | | Cache write (1-hour TTL) | Per 1M tokens written to a 1-hour prompt cache. | Enter rates in dollars per million tokens. Each field is independent: - Leave a field blank to keep the catalog rate for that bucket. The catalog rate shows as the field’s placeholder. - Enter a positive value to replace the catalog rate for that bucket only. - Enter `0` to make that bucket explicitly free, which is different from leaving it blank. Cache writes with an unknown TTL always bill at the catalog rate; they have no override field. Use the reset control on a field to clear a single override, or clear every field to drop all overrides for the model. Overrides are scoped to this provider and model, and they change what Agentic Data Plane’s cost reporting computes, not what the upstream provider actually charges you. ## [](#provider-detail-page)Explore the provider detail page After you create the provider, its detail page organizes everything about it into tabs: **Overview**, **Models**, **Connect**, **Playground** (when enabled for your organization), and **Settings**. ### [](#overview)Overview The **Overview** tab opens on a row of stat tiles: **Models enabled** (the models configured on this provider), **Requests** over the last 24 hours, **Spend** over the last 30 days, and **Health** over the last 24 hours, shown with a status indicator that reads healthy, degraded, failing, or idle based on the provider’s recent request and error volume. Below the tiles, a **Gateway traffic** chart plots request and token volume over time next to the **Connection** card. The Connection card shows the provider type, an **Active** or **Disabled** status badge, the authentication passthrough state, the proxy URL, the upstream base URL, the API key secret reference, and any attached guardrail, each with a copy control where relevant. A **Who calls what** card then draws the caller-to-model traffic for a time range you pick with the range selector, and a **Callers** table lists, over the same range, each agent or user that called the provider, with its request count, the models it used, its error count, spend, and when it was last seen. Search the table by caller name, slug, or email, and select a row to open that caller’s details on the **Cost & Usage** page. For analysis across providers, use the **Cost & Usage** page under **Governance** (see [View cost and usage](#view-cost-and-usage)). ### [](#models)Models The **Models** tab lists the chat models available on this provider. Each row shows the model’s name and identifier, icons for the capabilities it supports (such as vision, tools, JSON, or reasoning), its context-window limit, and its input and output prices per million tokens. Hover over a capability icon to read what that capability means. Use the toggle on a row to enable or disable that model on the provider. Select a model to open its detail page (see [View a model’s detail page](#model-detail-page)). Enabled models sort to the top. Use the search box to find a model by display name or identifier, and the **Status** and **Capability** filters to narrow the list. ### [](#connect)Connect The **Connect** tab generates ready-made client configuration for this provider: a gateway-token step, ready-to-run `rpk ai` setup steps, setup instructions for popular clients such as Claude Code, and code examples in several languages, all with the provider’s proxy URL prefilled. For a Bedrock provider, the Claude Code snippets run against the gateway without local AWS credentials, because the gateway signs the upstream AWS requests for you. See [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) for the underlying flow. ### [](#playground)Playground When the **Playground** tab is available, use it to test one of the provider’s models before you connect application code. Choose a model, send a prompt, and optionally adjust response controls such as reasoning effort and maximum output tokens. The response streams into the conversation while the Playground shows the request, response events, status, and timing. Playground conversations are ephemeral. ![LLM Playground test loop. A user selects a model, writes a prompt, and chooses response controls. AI Gateway translates and sends the request to the configured provider, streams the response back, and exposes request details, response events, status, and timing in the Playground.](https://docs.redpanda.com/agentic-data-plane/gateway/_images/llm-playground-test-loop.svg) Figure 1. Playground turns a model test into an inspectable gateway request ### [](#settings)Settings The **Settings** tab is where you edit the provider after you create it. It is organized into sections: - **General**: Edit the display name. The provider type and resource ID are fixed after creation and shown read-only. - **Credential**: Change how the gateway authenticates to the upstream provider. To rotate a stored key’s value for every provider that uses it, follow the link to **Provider keys** in the Secret Store. - **Connection**: The read-only proxy URL and the editable upstream base URL. Use **Test connection** to confirm the provider is reachable before you save. - **Models**: Choose which models this provider exposes to agents and developers. - **Transcripts**: Turn message-body recording on or off (see [Configure transcript logging](#transcript-logging)). When you change a field, a save bar appears with **Save changes** and **Discard**. Below the form, an enable/disable control stops or resumes routing without deleting anything, and a **Delete this provider** danger zone permanently removes the provider after you confirm. ### [](#model-detail-page)View a model’s detail page From the **Models** tab, select a model to open its detail page. The page gathers what the catalog knows about that model on this provider in one place: - An overview strip with the model’s context window (largest input it accepts), maximum output (most it can generate in a single response), and price (input and output rates per million tokens). Each value appears only when the catalog declares it. - Model details: The model ID and the provider’s proxy URL, each with a copy control. - Pricing: A read-only rate card of the model’s effective per-bucket rates, that is, the catalog rates with any per-model overrides applied (see [Override per-model pricing](#pricing-overrides)). This section appears only when rates are known, and links to the upstream provider’s pricing reference. - Capabilities: The model’s capabilities from the catalog, with a link to the upstream provider’s capabilities reference. This section appears only when the catalog declares capabilities. - Usage: Spend, Requests, and Tokens cards for traffic routed to this model through this provider over the last 7 days. Select **View more** on a card to open a metric detail drawer. ## [](#transcript-logging)Configure transcript logging The **Transcripts** section on the provider’s **Settings** tab controls whether AI Gateway records the message bodies this provider proxies. It has two independent toggles, both off by default: | Toggle | What it captures | | --- | --- | | Record inputs | Captures the full request body (prompt content and tool-call arguments) on observability traces. | | Record outputs | Captures the full response body (completion content and tool-call results) on observability traces. | Because both toggles default to off, AI Gateway does not retain message bodies for a new provider until you turn them on. Enable them to power turn-by-turn investigation and per-conversation drill-down in [the Transcripts view](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). Leave them off for workloads where the message body must not be retained, such as regulated PII or customer secrets. These are per-provider settings, not per-request: applications cannot opt in or out at call time. To split sensitive from non-sensitive traffic, create one provider with recording on and another with it off, and route each application to whichever proxy URL matches its data class. Recording settings do not affect cost and usage telemetry. Token counts, latency, and provider/model attribution are always recorded, so the **Cost & Usage** page reports spend for traffic on the provider regardless of these toggles; only the message bodies are withheld when the toggles are off. > 📝 **NOTE** > > Changing a toggle takes effect for new requests. Transcripts already captured under the previous setting are not retroactively redacted; delete or rotate the provider if you need to purge historical content. ## [](#save-and-verify)Save and verify 1. Click **Create provider**. Until the form is complete, the button stays disabled and the hint next to it tells you what’s still needed (for example, saving the new key). 2. After create, you land on the provider’s **Connect** tab, with client setup instructions and code examples that have the proxy URL prefilled. To copy the proxy URL on its own, open the **Overview** tab and use the copy control on the Connection card. The proxy URL is where your applications point. 3. To confirm the provider is reachable, open the **Settings** tab and click **Test connection** in the **Connection** section. The result appears as a message below the button. 4. To wire up an application, use the **Connect** tab or follow [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/). A successful **Test connection** result confirms that the provider’s credentials, region (Bedrock), and network path are all correct. If the call fails, see [Troubleshooting](#troubleshooting). ## [](#bedrock-inference-profiles)AWS Bedrock: Inference profiles and IAM Bedrock has three concepts that affect how you configure a provider: foundation models, cross-region inference profiles, and IAM. Get these right and the **Test connection** check passes. Get them wrong and you see `AccessDenied` or `ValidationException` errors. ### [](#foundation-models-versus-inference-profiles)Foundation models versus inference profiles A **foundation model** is the base model AWS exposes (for example, `anthropic.claude-sonnet-4-6`). It runs in the AWS region you call. A **cross-region inference profile** wraps a foundation model with a geography prefix that routes requests across multiple regions for higher availability and throughput. The prefix tells AWS which geography the request should run in: | Prefix | Geography | | --- | --- | | us. | US regions | | eu. | EU regions | | apac. | Asia-Pacific regions | | au. | Australia regions | | jp. | Japan regions | | global. | Any region; routes for lowest cost | Examples: `us.anthropic.claude-sonnet-4-6` (Claude Sonnet 4.6 routed across US regions), `eu.anthropic.claude-haiku-4-5` (Haiku 4.5 routed across EU regions). > ❗ **IMPORTANT** > > Anthropic Claude 4.6+ models (Sonnet 4.6, Opus 4.6, Opus 4.7) cannot be invoked with the bare foundation-model ID; they require an inference profile. If you try the bare ID, Bedrock returns: > > "Invocation of model ID … with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model." > > Older 4.5 and earlier Claude models still accept bare IDs. Pricing varies by profile. The bare foundation-model ID and the `global.` profile share AWS’s headline rate; geo profiles (`us.`, `eu.`, `apac.`, `au.`, `jp.`) carry approximately a 10% cross-region inference premium. Use `global.` when you want the headline rate and don’t need a specific geography. Use `us.` / `eu.` / `apac.` when data residency matters. AI Gateway preserves the regional prefix end to end when it records spend, so the **Cost & Usage** page attributes usage to the correct regional rate. A call to `eu.anthropic.claude-haiku-4-5` is billed at the EU Haiku rate, not the headline foundation-model rate. ### [](#iam-policy-patterns)IAM policy patterns Bedrock IAM resources have different ARN structures depending on whether you reference a foundation model, a system-defined inference profile, or an account-scoped application inference profile. The provider’s IAM principal needs `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` on every resource it calls. | Resource type | ARN shape | | --- | --- | | Foundation model | arn:aws:bedrock:{region}::foundation-model/{model-id} (no account ID; AWS-owned) | | System-defined inference profile | arn:aws:bedrock:{region}:*:inference-profile/{profile-id} (wildcard account; system-defined) | | Application inference profile (account-scoped) | arn:aws:bedrock:{region}:{account-id}:application-inference-profile/{profile-id} | A minimal policy granting access to all foundation models plus all cross-region profiles: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"], "Resource": [ "arn:aws:bedrock:*::foundation-model/*", "arn:aws:bedrock:*:*:inference-profile/*" ] }] } ``` For production, scope to specific models and regions instead of using wildcards. ## [](#anthropic-authorization-passthrough)Anthropic: Authorization passthrough If you want each client to authenticate against Anthropic with its own subscription (Claude Pro, Max, Team, or enterprise), enable `Authorization passthrough` instead of configuring a server-side API key. In this mode: - Leave the `API key` field empty. - Clients must send their own Anthropic `Authorization` header with every request. AI Gateway forwards it unchanged. - Use this when you want to aggregate individual client subscriptions rather than share a single API account. The provider detail page shows whether Authorization passthrough is enabled in the Connection card. ## [](#browse-providers-in-the-list-view)Browse providers in the list view The **LLM Providers** list page is the at-a-glance home for every provider in your dataplane. Open it from the sidebar’s **LLM Providers** entry. | Column | What it shows | | --- | --- | | Provider | The provider-type icon (OpenAI, Anthropic, Google, AWS Bedrock, or OpenAI-compatible), the display name, and the resource identifier beneath it. Select the provider name to open its detail page. | | Status | Active for an enabled provider. A disabled provider rejects requests to its proxy URL until you enable it again. | | Models | The model identifiers configured on the provider, shown as chips. A provider with no models configured shows a dash. | | 24h requests | Request count over the last 24 hours. | | 30d spend | Spend over the last 30 days. For longer-range or cross-provider analysis, use the Cost & Usage page under Governance (see View cost and usage). | Use the search box to find a provider by name, and the **Filter** button to narrow the list by provider type, model, or status. The **Add provider** button opens the create flow described in [Open the Create LLM provider page](#open-the-create-llm-provider-page). Each row’s actions menu can activate or deactivate the provider, copy its proxy or base URL, or delete it. The list paginates, with a rows-per-page selector in the footer. ## [](#view-cost-and-usage)View cost and usage The **Cost & Usage** page under **Governance** tracks spend, request volume, and token volume over time across providers, models, agents, and users, with period comparisons and CSV export. This report moved to its own page: see [Analyze Cost and Usage](https://docs.redpanda.com/agentic-data-plane/control/cost-usage/). ## [](#edit-disable-or-delete-a-provider)Edit, disable, or delete a provider - **Edit**: Open the **Settings** tab and change any field except the provider type and resource ID, which are immutable after creation. The display name, credential reference, base URL, and model list can all change. When you edit a field, a save bar appears; click **Save changes** to apply your edits. - **Disable**: On the **Settings** tab, click **Disable provider** and confirm. The provider remains in the list, but requests to its proxy URL are rejected until you enable it again. Use this when you want to pause traffic without losing configuration. - **Delete**: On the **Settings** tab, use the **Delete this provider** danger zone, click **Delete provider**, and type `delete` to confirm. The action is permanent. In-flight requests fail and downstream clients receive errors until reconfigured. ## [](#troubleshooting)Troubleshooting | Symptom | What to check | | --- | --- | | secret "" not found | Confirm the secret exists in your dataplane’s secret store and the reference in the provider configuration is spelled identically (UPPER_SNAKE_CASE, no typos). | | Bedrock returns AccessDenied or region errors | Verify the AWS region field matches the region where your Bedrock models are enabled. Bedrock model availability varies by region. Confirm the IAM principal has bedrock:InvokeModel on the foundation-model and inference-profile ARNs you use. See AWS Bedrock: Inference profiles and IAM. | | Bedrock returns "Invocation of model ID … with on-demand throughput isn’t supported" | You called a Claude 4.6+ model with a bare foundation-model ID. Switch to an inference profile (for example, us.anthropic.claude-sonnet-4-6 instead of anthropic.claude-sonnet-4-6). See AWS Bedrock: Inference profiles and IAM. | | Anthropic returns 401 when passthrough is enabled | Confirm the client is sending its own Authorization header and the API key field on the provider is empty. | | Gemini returns 401 | Gemini uses the x-goog-api-key header, not Authorization. If you’re seeing 401s on Gemini, check that the client is sending the correct header. See Connect your app to AI Gateway. | | Provider list empty or 403 | Confirm your account can act on LLM providers. Listing them needs Action::"LLMProvider.list" and creating one needs Action::"LLMProvider.create". Outside the Admin role, both come from an access policy. See LLM provider permissions. | ## [](#limitations)Limitations AI Gateway does not provide these capabilities. For current status, see the [Agentic Data Plane release notes](https://docs.redpanda.com/agentic-data-plane/reference/release-notes/). - **Multi-provider routing, failover, and retries across providers.** A synthetic provider that fans requests to multiple upstreams is not part of AI Gateway. - **Rate limits.** Requests-per-second, per-minute, or per-day limits are not available. To cap spend rather than request rate, use [budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/), which enforce a per-agent hard cap. - **Managed MCP aggregation at the gateway.** Register MCP tool servers separately under **MCP Servers** in Agentic Data Plane. ## [](#next-steps)Next steps - [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) - [Set up budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) --- # Page 67: Connect Your App to AI Gateway **URL**: https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent.md --- # Connect Your App to AI Gateway > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Connect Your App to AI Gateway latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect-agent page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: connect-agent.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/connect-agent.adoc description: Point your application or AI agent at an AI Gateway provider's proxy URL. Covers the URL shape, the local development workflow with rpk ai, the OIDC client-credentials flow for CI and application code, and SDK examples for OpenAI, Anthropic, Google AI, AWS Bedrock, and OpenAI-compatible endpoints. page-topic-type: how-to personas: agent_builder learning-objective-1: Construct the proxy URL for an LLM provider you have configured learning-objective-2: Authenticate to AI Gateway with the rpk ai CLI for local development or with OIDC client credentials for CI and programmatic clients learning-objective-3: Send requests through the proxy URL with the SDK of your choice page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-12" --- This guide shows how to connect your [AI agent](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent) or application to the AI Gateway. You construct the proxy URL for a provider you have already created, authenticate (with the [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) CLI for local development or with OIDC client credentials for CI and application code), and send your first request with the SDK of your choice. > 💡 **TIP** > > The provider’s **Connect** tab in Agentic Data Plane generates this configuration for you: a gateway-token step, setup instructions for popular clients, and code examples with the provider’s proxy URL prefilled. Copy from the tab for a quick start, or follow this page for the full flow. After completing this guide, you will be able to: - Construct the proxy URL for an LLM provider you have configured - Authenticate to AI Gateway with the `rpk ai` CLI for local development or with OIDC client credentials for CI and programmatic clients - Send requests through the proxy URL with the SDK of your choice ## [](#prerequisites)Prerequisites - A configured LLM provider. If you haven’t created one yet, see [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). - For local development, nothing else. You’ll install `rpk ai` in the next section. - For CI or programmatic clients: Your agent’s service account client ID and a client secret, issued from the agent’s **Credentials** tab. See [Service account authorization](https://docs.redpanda.com/agentic-data-plane/connect/concepts/#service-account-authorization). To register an agent and issue credentials, see [Set Up a Self-Managed Agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/). - A development environment with your chosen programming language. ## [](#proxy-url-anatomy)Proxy URL anatomy Every provider you create in AI Gateway gets its own proxy URL: ```text /llm/v1/providers// ``` - ``: The AI Gateway base URL for your dataplane. Cluster-specific subdomain on `clusters.rdpa.co` (for example, `[https://aigw..clusters.rdpa.co](https://aigw.\.clusters.rdpa.co)`). Copy the exact value from the `Proxy URL` field on any provider’s Connection card. - ``: The name you gave the provider when you created it, for example `my-openai` or `prod-anthropic`. - ``: The upstream provider’s native API path (for example, `v1/chat/completions` for OpenAI, `v1/messages` for Anthropic). AI Gateway forwards the request to the upstream provider, attaches the configured credentials, and records the request for observability. Your application never sees the upstream API key. > 💡 **TIP** > > The provider detail page generates ready-to-run snippets pre-filled with the correct proxy URL and paths. When in doubt, copy from the Connect your app section there. ## [](#authenticate-with-rpai)Use `rpk ai` for local development The [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) command is the Redpanda AI CLI. Use it to manage AI Gateway resources (LLM providers, MCP servers, OAuth providers) and call MCP tools from the command line. `rpk ai` is self-contained: it has its own login and its own Agentic Data Plane environment selection, independent of any `rpk cloud` session. 1. [Install `rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-install/): ```bash rpk ai install ``` Update later with [`rpk ai upgrade`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-upgrade/); remove with [`rpk ai uninstall`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-uninstall/). > 📝 **NOTE** > > Upgrading `rpk` does not upgrade the plugin. The plugin is a separate binary that stays at its installed version, even across `rpk` upgrades, until you run `rpk ai upgrade`. New commands and flags in the reference marked "introduced in ai version X" require the plugin at that version or later. 2. Sign in. This runs an OAuth device-authorization flow in your browser, caches credentials in `~/.rpai/credentials` (readable only by you), then lists the Agentic Data Plane environments in your organization so you can select one: ```bash rpk ai auth login ``` 3. Select the Agentic Data Plane environment whose AI Gateway you want to target. The `rpk ai env use` command accepts an environment name or ID and switches the active environment: ```bash rpk ai env list rpk ai env use ``` Inspect the resolved environment and token state at any time with `rpk ai env show` and `rpk ai auth status`. 4. Verify the connection: ```bash rpk ai llm-provider list ``` If the cached token has expired, `rpk ai` returns a 401; rerun `rpk ai auth login` to refresh it. > 📝 **NOTE** > > `rpk ai help`, `rpk ai version`, and unknown subcommands run without prompting for authentication, so you can browse the CLI surface offline before signing in. Authentication is only required for commands that hit AI Gateway. > 💡 **TIP** > > To target a specific AI Gateway URL for a single invocation (for example, a local gateway, or a staging environment the environments list does not include), pass `--rpai-endpoint`: > > ```bash > rpk ai --rpai-endpoint http://localhost:8090 llm-provider list > ``` > > This overrides the selected environment’s AI Gateway URL for that one command, and the flag is not bound to an environment variable. For a manual or local gateway you use repeatedly, define it once as an environment instead: > > ```bash > rpk ai env add local --ai-gateway-url http://localhost:8090 --auth-mode none > rpk ai env use local > ``` ### [](#environment-variables)Environment variables The `rpk ai` command honors the following environment variables: | Variable | Purpose | | --- | --- | | RPAI_TOKEN | Static bearer token for the gateway. rpk ai normally manages its own token through rpk ai auth login; set this (or pass --token) to override, for example in a headless or CI shell. | | RPAI_CONFIG, RPAI_VERBOSE, RPAI_FORMAT | Map to --rpai-config, --rpai-verbose, --format (short flags -c, -v, -o). Long flag names are renamed under rpk ai to avoid collision with rpk’s globals. There is no environment variable for environment selection (use `--rpai-environment or rpk ai env use) or for the AI Gateway URL override (use the --rpai-endpoint flag). | ## [](#authenticate-with-oidc-client-credentials)Authenticate with OIDC client credentials (CI and programmatic) For application code, CI runners, server-side processes, and headless agents, use the OIDC `client_credentials` grant directly. This is the canonical authentication path for SDK-style usage; `rpk ai` is for command-line workflows, not for embedding in application code. Values are surfaced on the provider’s Connection card; defaults at the time of writing are below. | Parameter | Value (today) | | --- | --- | | Discovery URL | https://auth.prd.cloud.redpanda.com/.well-known/openid-configuration. Also surfaced as the Discovery field on the provider’s Connection card. | | Token endpoint | https://auth.prd.cloud.redpanda.com/oauth/token | | Audience | cloudv2-production.redpanda.cloud | | Grant type | client_credentials | #### cURL ```bash AUTH_TOKEN=$(curl -s --request POST \ --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \ --header 'content-type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id= \ --data client_secret= \ --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token) ``` Replace `` and `` with your service account credentials. #### Python (authlib) ```python from authlib.integrations.requests_client import OAuth2Session import requests # Discover token endpoint from OIDC metadata metadata = requests.get( "https://auth.prd.cloud.redpanda.com/.well-known/openid-configuration" ).json() token_endpoint = metadata["token_endpoint"] client = OAuth2Session( client_id="", client_secret="", token_endpoint=token_endpoint, ) token = client.fetch_token( grant_type="client_credentials", audience="cloudv2-production.redpanda.cloud", ) access_token = token["access_token"] ``` Passing `token_endpoint` to the `OAuth2Session` constructor lets `authlib` handle renewal automatically. For `client_credentials` grants, it fetches a new token rather than using a refresh token. #### Node.js (openid-client) ```javascript import { Issuer } from 'openid-client'; const issuer = await Issuer.discover( 'https://auth.prd.cloud.redpanda.com' ); const client = new issuer.Client({ client_id: '', client_secret: '', }); const tokenSet = await client.grant({ grant_type: 'client_credentials', audience: 'cloudv2-production.redpanda.cloud', }); const accessToken = tokenSet.access_token; ``` ### [](#token-lifecycle-management)Token lifecycle management > ❗ **IMPORTANT** > > Your client is responsible for refreshing tokens before they expire. OIDC access tokens have a limited TTL set by the identity provider and are not automatically renewed by AI Gateway. Check the `expires_in` field in the token response for the exact duration. - Proactively refresh at ~80% of the token’s TTL to avoid failed requests. - `authlib` (Python) handles renewal automatically when you pass `token_endpoint` to `OAuth2Session`. - For other languages, cache the token and its expiry, then request a new token before the current one expires. - For SDK code, refresh OIDC client-credentials tokens through your client library (see the `authlib` example above). ## [](#send-requests-with-your-sdk)Send requests with your SDK The examples in this section assume you’ve set: ```bash export PROXY_URL="/llm/v1/providers/" export AUTH_TOKEN="" # from the client_credentials flow above ``` ### OpenAI SDK ```python import os from openai import OpenAI client = OpenAI( base_url=os.environ["PROXY_URL"], # .../llm/v1/providers/my-openai api_key=os.environ["AUTH_TOKEN"], # OIDC access token ) response = client.chat.completions.create( model="gpt-4o", # native OpenAI model ID messages=[{"role": "user", "content": "Hello from AI Gateway"}], ) print(response.choices[0].message.content) ``` The OpenAI SDK calls the proxy’s `/v1/chat/completions` path, which AI Gateway forwards to OpenAI unchanged. Use it with any OpenAI provider and, with a different `base_url`, with any OpenAI-compatible provider (vLLM, Ollama, LM Studio, Together, Groq, OpenRouter). ### Anthropic SDK ```python import os from anthropic import Anthropic client = Anthropic( base_url=os.environ["PROXY_URL"], # .../llm/v1/providers/my-anthropic auth_token=os.environ["AUTH_TOKEN"], # OIDC access token ) message = client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, messages=[{"role": "user", "content": "Hello from AI Gateway"}], ) print(message.content[0].text) ``` The Anthropic SDK hits `v1/messages` on the proxy, which AI Gateway forwards to Anthropic. If the provider is configured with `Auth passthrough`, send your own Anthropic `Authorization` header instead of an `auth_token`. AI Gateway forwards it unchanged. ### Google Gemini SDK ```python import os from google import genai client = genai.Client( api_key=os.environ["AUTH_TOKEN"], # forwarded as x-goog-api-key http_options={"base_url": os.environ["PROXY_URL"]}, # .../llm/v1/providers/my-google ) response = client.models.generate_content( model="gemini-2.0-flash", contents="Hello from AI Gateway", ) print(response.text) ``` > ❗ **IMPORTANT** > > Gemini authenticates with the `x-goog-api-key` header, not `Authorization: Bearer`. Most Google SDKs set `x-goog-api-key` automatically from the `api_key` parameter. If you hand-roll the request, set the header yourself. ### AWS Bedrock Bedrock is different: SigV4 signing is performed **server-side** by AI Gateway using the credentials on the provider. Your client only needs to call the proxy URL with an OIDC access token. ```python import os, httpx # Bedrock 4.6+ Anthropic models require an inference profile (us./eu./apac./global.). # Replace with the inference profile your provider exposes. response = httpx.post( f"{os.environ['PROXY_URL']}/model/us.anthropic.claude-sonnet-4-6/invoke", headers={"Authorization": f"Bearer {os.environ['AUTH_TOKEN']}"}, json={ "anthropic_version": "bedrock-2023-05-31", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 1024, }, ) print(response.json()) ``` See [the Bedrock provider reference](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#bedrock-inference-profiles) for inference-profile selection guidance. > 💡 **TIP** > > Bedrock’s `Converse` API works the same way: send to `/model/{MODEL_ID}/converse` with a Converse-shaped body. Or use the AWS SDK’s `bedrockruntime` client and set its `BaseEndpoint` to the proxy URL; the SDK signs the request, AI Gateway re-signs server-side with the provider’s credentials, and your client never sees AWS keys. ### OpenAI-compatible Use the OpenAI SDK with the proxy URL of the OpenAI-compatible provider and whatever model identifier the upstream exposes: ```python import os from openai import OpenAI client = OpenAI( base_url=os.environ["PROXY_URL"], # .../llm/v1/providers/my-vllm api_key=os.environ["AUTH_TOKEN"], ) response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", # as exposed by your upstream messages=[{"role": "user", "content": "Hello"}], ) ``` > 📝 **NOTE** > > The provider detail page also has client guides for **Claude Code**, **Codex**, and **Gemini** (the desktop client). Open **Connect your app** on the provider’s page to see the per-client setup instructions. ## [](#group-requests-into-transcripts)Group a session’s requests with a conversation ID Stamp every request, the LLM call and each MCP tool call, with the `X-Redpanda-Genai-Conversation` header set to your framework’s session or thread ID, using the same value on every request in the session. This tags the session’s gateway calls with one conversation ID, so the gateway’s records of the turn group together. The header doesn’t affect authentication or whether requests succeed. Set it through your SDK’s default-headers mechanism so it rides along with both the LLM call and each MCP tool call: ```python import os from openai import OpenAI client = OpenAI( base_url=os.environ["PROXY_URL"], api_key=os.environ["AUTH_TOKEN"], default_headers={"X-Redpanda-Genai-Conversation": session_id}, ) ``` Replace `session_id` with the session or thread identifier your framework already tracks, and stamp the same value on the session’s MCP tool calls. Agentic Data Plane assembles the agent’s **Transcripts** tab from OpenTelemetry spans your app exports itself, not from the gateway calls alone. To populate it, stream instrumented spans to the cluster’s OTLP endpoint and put the same identifier on each session’s root span as the `gen_ai.conversation.id` attribute. For a self-managed agent, the **Setup** tab generates this instrumentation under **Stream transcripts**; see [Set up a self-managed agent](https://docs.redpanda.com/agentic-data-plane/connect/self-managed-agents/). For how transcripts read this attribute, see [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). ## [](#streaming-responses)Streaming responses Streaming passes through unchanged. Use the SDK’s native streaming API; the proxy forwards the stream byte-for-byte. ```python response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Write a short poem"}], stream=True, ) for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` ## [](#handle-errors)Handle errors AI Gateway returns standard HTTP status codes. The upstream provider’s error body passes through, so your existing SDK error handling works: | Status | Meaning | | --- | --- | | 400 | Bad request. Invalid parameters or malformed JSON. | | 401 | Authentication failed. Token invalid, expired, or (for Gemini) sent in the wrong header. | | 403 | Forbidden. Nothing grants the service account this action, or the provider is disabled. | | 404 | Provider or model not found. Verify the provider name in the URL and the model identifier. | | 429 | Rate limited by the upstream provider. AI Gateway does not enforce its own rate limits today. Respect Retry-After if present. | | 5xx | Upstream or gateway error. Retry with exponential backoff. | ## [](#best-practices)Best practices - Use environment variables for the proxy URL and token. Never hard-code them. - Refresh OIDC tokens through your client library so refresh is invisible to your SDK code (`authlib` for Python, `openid-client` for Node.js, and so on). - Implement retry with exponential backoff for 5xx and timeout conditions. - Respect `Retry-After` on 429 responses. - Rotate service account credentials on a schedule your organization accepts. - Observe usage in Redpanda Agentic Data Plane on each provider’s detail page. ## [](#troubleshooting)Troubleshooting ### [](#401-unauthorized)401 Unauthorized - If you’re using `rpk ai`: Rerun `rpk ai auth login` to refresh the credentials. Token expiry surfaces as a 401. - If you’re using OIDC client credentials: Check the token hasn’t expired and refresh it. Verify the audience is `cloudv2-production.redpanda.cloud` and the `Authorization` header is formatted `Bearer `. - For Gemini: Ensure the token is sent as `x-goog-api-key`, not `Authorization`. - For Anthropic with passthrough: Ensure the client is sending a valid Anthropic `Authorization` header. ### [](#404-not-found)404 Not found - Re-check the provider name in the proxy URL. The segment after `/providers/` must match the provider’s `Name` exactly. - For model-not-found: Confirm the model identifier is one your provider’s catalog actually serves. OpenAI-compatible endpoints accept whatever model IDs the upstream exposes. ### [](#403-forbidden)403 Forbidden - The service account may lack the required access. Ask an admin for an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) that names the service account as its principal and grants `Action::"LLMProvider.invoke"`, plus `Action::"LLMProvider.get"` if the account also reads provider config. No built-in role short of Admin grants these. See [LLM provider permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#llm-provider-permissions). - The provider may be disabled. Check the `Status` field on its Connection card. ### [](#connection-timeout-or-reset)Connection timeout or reset - Verify the proxy URL is correct (copy directly from the provider’s Connection card). - Check that the provider isn’t pointing at a private base URL your client can’t reach (OpenAI-compatible providers only). - Confirm the upstream provider’s status page. ## [](#next-steps)Next steps - [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) --- # Page 68: How AI Gateway Works **URL**: https://docs.redpanda.com/agentic-data-plane/gateway/overview.md --- # How AI Gateway Works > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: How AI Gateway Works latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/gateway/pages/overview.adoc description: AI Gateway is Agentic Data Plane's managed proxy for LLM APIs. Create a provider for OpenAI, Anthropic, Google AI, AWS Bedrock, or an OpenAI-compatible endpoint, and point your applications at a Redpanda-hosted URL with managed secrets, authentication, and observability. page-topic-type: overview personas: agent_builder, platform_engineer, security_compliance_lead learning-objective-1: Describe what AI Gateway is and how a managed proxy differs from direct upstream calls learning-objective-2: Explain how LLM providers, secrets, and OIDC authentication fit together in AI Gateway learning-objective-3: Identify use cases where AI Gateway fits, and use cases where it does not page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- AI Gateway is Redpanda Agentic Data Plane’s managed proxy for LLM APIs. Instead of giving every application a provider API key and letting it call the upstream directly, you create an **LLM provider** in Agentic Data Plane and point your applications at a Redpanda-hosted proxy URL. Redpanda handles the upstream credentials, forwards the request, and records usage. Your code continues to use the provider’s native SDK. After reading this page, you will be able to: - Describe what AI Gateway is and how a managed proxy differs from direct upstream calls - Explain how LLM providers, secrets, and OIDC authentication fit together in AI Gateway - Identify use cases where AI Gateway fits, and use cases where it does not ## [](#the-problem-ai-gateway-solves)The problem AI Gateway solves Teams adopting LLMs can quickly hit operational problems: - **Credential sprawl:** Every team that touches an LLM gets its own API key. Rotation is manual, offboarding is manual, and it’s hard to know who’s using what. - **SDK lock-in and switching cost:** Each provider has its own SDK, authentication scheme, and model catalog. Swapping OpenAI for Anthropic means a code change, not a configuration change. - **No shared view of usage:** Provider dashboards tell you what a single API key spent. They don’t tell you what your organization spent, broken down by team or application. ![AI Gateway connection flow. An administrator configures an LLM provider with its upstream endpoint, credentials, and allowed models. Local developers connect with rpk ai and deployed applications or self-managed agents use OAuth client credentials. AI Gateway authenticates, routes, applies policy, and records usage before calling the upstream model provider.](https://docs.redpanda.com/agentic-data-plane/gateway/_images/ai-gateway-connection-flow.svg) Figure 1. Configure a provider once, then route every client through one governed AI Gateway path ## [](#what-ai-gateway-gives-you)What AI Gateway gives you AI Gateway consolidates provider access behind the following capabilities. ### [](#traffic-stays-in-your-vpc)Traffic stays in your VPC LLM requests are proxied through your dataplane’s AI Gateway. API keys are stored in your dataplane’s secret store and never leave your infrastructure. Upstream calls leave your VPC only when the LLM provider is third-party (OpenAI, Anthropic, Google AI). Self-hosted OpenAI-compatible endpoints stay entirely inside your network. ### [](#centralized-secrets)Centralized secrets The upstream API key (or AWS credentials for Bedrock) lives in the Redpanda secret store and is attached to the provider at configuration time. Your application never sees it; rotation happens in one place. ### [](#a-managed-proxy-url-per-provider)A managed proxy URL per provider Every provider you create has its own URL of the form `/llm/v1/providers//`. Your application points its SDK at this URL instead of the upstream, continues to use the provider’s native API, and authenticates to Redpanda with a short-lived OIDC access token. The gateway base is a cluster-specific subdomain (for example, `aigw..clusters.rdpa.co`). Copy the exact value from the `Proxy URL` field on any provider’s detail page. ### [](#native-sdk-compatibility)Native SDK compatibility Use the provider’s own SDK: OpenAI, Anthropic, Google AI, AWS Bedrock, or any OpenAI-compatible client (vLLM, Ollama, LM Studio, LocalAI, Together, Groq, OpenRouter). AI Gateway does not require a single unified SDK. It forwards native requests to the native upstream. ### [](#managed-authentication)Managed authentication Applications authenticate to Agentic Data Plane with OIDC service accounts instead of long-lived provider API keys. Service accounts use the same role and audit model as every other Agentic Data Plane resource, and mint short-lived tokens that are easy to revoke. For local command-line workflows, use [`rpk ai`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai/) to sign in (`rpk ai auth login`) and talk to the gateway. CI and programmatic clients use the OIDC client-credentials grant directly. See [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/). ### [](#per-provider-observability)Per-provider observability The provider’s detail page in Agentic Data Plane records spend, request counts, and token counts for the last 7 days. The **Cost & Usage** page under **Governance** expands that view with time-series charts, provider and model grouping, date ranges, and filters for provider, model, cost type, token type, user, and agent. ## [](#whats-in-the-ui)What’s in the UI In Agentic Data Plane ([ai.redpanda.com](https://ai.redpanda.com)) you’ll find these areas: - **Home**: The landing page after sign-in. A snapshot of items that need attention (such as a budget over its cap or a disabled resource, with quick actions to resolve them), recent request and spend activity, budget status, and counts of your Agentic Data Plane resources with quick links into each area. - **Agents**: Create and manage Redpanda-hosted agents and registered self-managed agents. - **Guardrails**: Define content policies, word filters, and PII rules that apply to traffic on a Bedrock provider. - **LLM Providers**: Create, edit, enable, and delete providers. This is the home of AI Gateway configuration. - **MCP Servers**: Register [MCP](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#model-context-protocol-mcp) tool servers for agents. Separate from the AI Gateway proxy URL. - **My Connections**: Each user’s own per-user OAuth connections to the configured providers. - **Integrations setup**: Admin home for the OAuth plumbing behind MCP tools. The **Outbound providers** tab registers upstream identity providers for user-delegated MCP authentication (for example, GitHub or Google); the **Inbound clients** tab registers external tools (Claude.ai, ChatGPT, Cursor) that request access tokens from the gateway. - **Secrets Store**: Create and manage the secrets that providers, MCP servers, and agents reference by name. - **Governance**: The **Cost & Usage** page for spend, request, and token analysis across providers, models, and agents, and the **Budgets** page for per-agent spend caps. LLM Providers is where you configure provider settings. The others are covered by their own docs. ## [](#supported-providers)Supported providers AI Gateway supports the following provider types. The UI labels and short descriptions match the picker on the **Create LLM provider** page. | Type | Typical upstream | | --- | --- | | OpenAI | Proxy GPT, o-series, and embeddings through the OpenAI API. Best when you already hold an OpenAI API key or want the broadest GPT model catalog. | | Anthropic | Call Claude Opus, Sonnet, and Haiku directly. Optionally forwards the client’s Authorization header for enterprise and Max-plan subscription passthrough. | | Google AI | Reach Gemini Pro, Flash, and multimodal models through Google AI Studio. Ideal for long-context workloads and image/video inputs. | | AWS Bedrock | Invoke foundation models (Claude, Llama, Titan, Nova) hosted inside your AWS account. Use when data residency, IAM, or VPC egress matter more than raw feature parity. Signed with SigV4 server-side by AI Gateway. | | OpenAI-compatible | Point at any OpenAI-compatible endpoint (vLLM, Ollama, LM Studio, LocalAI, Together, Groq, OpenRouter). Useful for self-hosted models and aggregator gateways that ship /v1/chat/completions. | See [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) for the full form reference for each type. ## [](#when-to-use-ai-gateway)When to use AI Gateway AI Gateway is a good fit when you want to: - Pull provider API keys out of application code and manage them centrally. - Keep LLM traffic inside your dataplane’s VPC and your secrets out of application code. - Authenticate applications to LLMs using the same OIDC identity you use for other Agentic Data Plane resources. - Run a self-hosted OpenAI-compatible endpoint (vLLM, Ollama, LM Studio) alongside 1P providers behind a single management plane. - Separate operator and developer roles. Operators configure providers and credentials; developers point at proxy URLs. It is not the right fit when you: - Only ever call a single provider with a single API key and are happy managing that key inline. - Need routing, failover, or cross-provider load balancing across providers. AI Gateway does not provide these capabilities. ## [](#out-of-scope)Limitations AI Gateway does not provide these capabilities. For current status, see the [Agentic Data Plane release notes](https://docs.redpanda.com/agentic-data-plane/reference/release-notes/). - **Multi-provider routing, failover, and retries.** A synthetic provider that fans requests to multiple upstreams is not part of AI Gateway. - **Rate limits.** Requests-per-second, per-minute, or per-day caps are not available. To cap spend rather than request rate, use [budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/), which enforce a per-agent hard cap. - **Managed MCP aggregation at the gateway.** Register MCP tool servers separately under **MCP Servers** in Agentic Data Plane. ## [](#next-steps)Next steps 1. [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) 2. [Connect your app to AI Gateway](https://docs.redpanda.com/agentic-data-plane/gateway/connect-agent/) --- # Page 69: Get Started **URL**: https://docs.redpanda.com/agentic-data-plane/get-started.md --- # Get Started > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Get Started latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/get-started/pages/index.adoc description: Stand up Redpanda Agentic Data Plane and ship your first agent. page-git-created-date: "2026-04-29" page-git-modified-date: "2026-07-24" --- Redpanda’s Agentic Data Plane runs and governs AI agents on your streaming data. Get an overview of the platform, then follow the quickstart that matches your role: agent builders create and test their first agent, and administrators set up the LLM provider that makes agent building possible. - [Redpanda Agentic Data Plane Overview](adp-overview/) Enterprise-grade infrastructure for building, deploying, and governing AI agents at scale with compliance-grade audit trails. - [Agentic Data Plane Quickstart for Agent Builders](agent-builder-quickstart/) Sign in to Redpanda Agentic Data Plane, explore the agents your organization already runs, then build, test, and iterate on your own agent backed by MCP tools. - [Agentic Data Plane Quickstart for Administrators](adp-quickstart/) Set up Redpanda Agentic Data Plane for your organization: configure an LLM provider with a key stored in the platform secret store, verify it, and hand off to your agent builders. --- # Page 70: Redpanda Agentic Data Plane Overview **URL**: https://docs.redpanda.com/agentic-data-plane/get-started/adp-overview.md --- # Redpanda Agentic Data Plane Overview > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Redpanda Agentic Data Plane Overview latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: adp-overview page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: adp-overview.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/get-started/pages/adp-overview.adoc description: Enterprise-grade infrastructure for building, deploying, and governing AI agents at scale with compliance-grade audit trails. page-topic-type: overview personas: agent_builder, platform_engineer, security_compliance_lead, pilot_lead, domain_evaluator learning-objective-1: Identify the key components of Redpanda Agentic Data Plane and their purposes learning-objective-2: Describe how each component addresses enterprise governance and reliability requirements learning-objective-3: Determine whether Redpanda Agentic Data Plane fits your organization's requirements for AI agent deployment page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-10" --- Enterprises spent decades building governance for their human workforce: identity, access control, audit trails, and accountability. They never built the same controls for [AI agents](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent). Most agents run with broad credentials, no identity of their own, and no record of what they did, which is why enterprise AI so often stalls before it reaches production. Redpanda Agentic Data Plane closes that gap. Agentic Data Plane is governance infrastructure for enterprise agents: it sits between your agents and your data, gives every agent an identity, mediates every tool call and data access, and records every action so you can replay and audit it. With Agentic Data Plane, you connect agents to all your enterprise data, from real-time event streams to databases and business systems, and run them safely at scale without losing control. Agentic Data Plane builds on open standards, including MCP, OAuth, OpenTelemetry, and the Kafka protocol, so governance sits above your choice of LLM, cloud, or agent framework. You stay free to adopt new models and frameworks as they emerge, without re-platforming or locking into a single vendor. Agentic Data Plane is its own product surface. It runs on Redpanda, so there’s no separate cluster to stand up first: Redpanda provisions the underlying environment when you add Agentic Data Plane. After reading this page, you will be able to: - Identify the key components of Redpanda Agentic Data Plane and their purposes - Describe how each component addresses enterprise governance and reliability requirements - Determine whether Redpanda Agentic Data Plane fits your organization’s requirements for AI agent deployment ## [](#who-runs-each-component)Who runs each component **Managed** and **self-managed** describe runtime ownership, not whether a resource is governed. Redpanda runs managed agents and managed MCP server implementations. You run self-managed agents and MCP servers, while Agentic Data Plane registers their identities and fronts their model or tool traffic through the gateways. An LLM provider is different: it is a gateway connection to an upstream model service, not a model runtime hosted in Agentic Data Plane. ![Resource ownership comparison. Redpanda runs managed agent runtimes and managed MCP server implementations. Customers run self-managed agent runtimes and self-managed MCP endpoints. AI Gateway and MCP Gateway provide identity, policy, and observability across both ownership models. LLM providers remain upstream model services connected through an Agentic Data Plane provider configuration.](https://docs.redpanda.com/agentic-data-plane/get-started/_images/resource-ownership-models.svg) Figure 1. Runtime ownership changes, but gateway identity and governance stay consistent ## [](#architecture)Architecture Agents and clients connect through Agentic Data Plane rather than reaching your systems directly. Agentic Data Plane gives every agent an identity and applies guardrails and spend caps, mediates tool and data access through MCP servers to your systems of record, and routes every LLM call through the AI Gateway. All activity streams to the Redpanda Data Platform as OpenTelemetry traces for replay, audit, and cost reporting. ![Architecture diagram: Redpanda-managed agents](https://docs.redpanda.com/agentic-data-plane/get-started/_images/adp-architecture.png) Figure 2. Redpanda Agentic Data Plane sits between your agents and clients and your external systems and LLM providers ## [](#ai-gateway)AI Gateway The AI Gateway is a managed proxy for LLM APIs. You configure an LLM provider once, and applications point at a Redpanda-hosted proxy URL while continuing to use the provider’s native SDK. Redpanda holds the upstream credentials, so application code never carries a provider API key and rotation happens in one place. Because every LLM call flows through one managed entry point, you get usage and cost visibility broken down by provider, model, and user, across any agent framework or platform, including Redpanda’s own, without locking into any single vendor’s SDK. Applications authenticate with short-lived OIDC service-account tokens, and streaming responses pass through unchanged. To control spend, set caps and per-agent [budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) in one place. For more information, see [How AI Gateway Works](https://docs.redpanda.com/agentic-data-plane/gateway/overview/). ## [](#mcp-servers)MCP servers [MCP servers](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#mcp-server) translate agent intent into connections to databases, queues, HRIS, CRMs, and other business systems. They are the simplest way to give agents context and capabilities without writing glue code. MCP servers are lightweight, support OIDC-based authentication, and enforce deterministic policies at the tool level. You pick a managed type from the marketplace and configure it with your credentials, or register a self-managed server. Policy enforcement programmatically prevents prompt injection, SQL injection, and other agent-based attacks. With real-time debugging capabilities, you reduce integration time while getting enterprise-grade security. You can reuse your existing infrastructure and data sources rather than building new integrations from scratch. For more information, see [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/). ## [](#ai-agents)AI agents With Redpanda AI agents, you declare the agent behavior you want and Redpanda handles execution and orchestration. Instead of writing Python or JavaScript, you configure agents declaratively: set a system prompt, choose a model, and attach the tools the agent can use. You can orchestrate multiple specialized [sub-agents](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#subagent), or bring your own frameworks like LangChain or LlamaIndex. What makes this practical at scale is a broad library of prebuilt tools and integrations with built-in filtering, enrichment, and routing, which give declarative definitions real power. The result is faster time-to-production, lower maintenance (declarative definitions instead of imperative code), and organizational consistency across teams. For more information, see [How Agents Work](https://docs.redpanda.com/agentic-data-plane/connect/agents-overview/). ## [](#built-in-governance-controls)Built-in governance controls Redpanda Agentic Data Plane addresses critical enterprise requirements across all components. - **Security by design**: MCP servers enforce policies at the tool level, programmatically preventing prompt injection, SQL injection, and other agent-based attacks. Policy enforcement is deterministic and controlled. Agents cannot bypass security constraints even through creative prompting. - **Unified authorization**: All components use OIDC-based authentication with an on-behalf-of authorization model. When a user invokes an agent, the agent inherits the intersection of its own permissions and the user’s permissions, so an agent can only do what the user behind it is allowed to do. This ensures proper data access scoping. An embedded identity provider lets any standards-compliant MCP client, such as Claude Desktop, Claude Code, or Microsoft Copilot Studio, connect over OAuth without custom header handling. - **Complete observability**: Redpanda Agentic Data Plane provides two levels of inspection. Execution logs ([transcripts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript)) give a read-only view of every agent run, capturing each prompt, response, and tool call with token and USD cost per conversation, plus status, duration, and an interactive timeline. Traces are emitted as OpenTelemetry data into an immutable Redpanda topic with 100% sampling. Real-time debugging tools let you inspect individual MCP server calls down to individual tool invocations with full timing data. You can view detailed agent actions in Agentic Data Plane and replay data for agent evaluations. For more information, see [How Observability Works](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/). - **Compliance and audit**: Redpanda Agentic Data Plane records every agent action and the data sources used in each decision, which supports audit and compliance efforts (for example, frameworks such as ISO 42001 or the EU AI Act). Execution logs are stored in Redpanda topics and can be materialized to Iceberg tables for longer-term retention and analysis. ## [](#use-cases)Use cases Common Redpanda Agentic Data Plane use cases include: - **Automate financial operations**: Build agents that reconcile invoices across vendors and ERP systems, monitor real-time spending against budget, or review contracts against approved templates and flag out-of-policy clauses for legal review. - **Streamline HR and people operations**: Answer employee questions against benefits, payroll, and HRIS systems; automate onboarding and offboarding workflows; or triage internal support tickets to the right team. - **Improve customer experience**: Route customer inquiries to the right team, summarize prior support history before an agent picks up the case, or surface relevant knowledge-base articles in real time. - **Modernize infrastructure operations**: Detect anomalies in real-time telemetry from cloud, network, or factory equipment, correlate signals across disparate monitoring systems, and auto-create maintenance tickets with the relevant context already attached. - **Extend enterprise productivity tools**: Integrate Microsoft Copilot or other workplace agents with internal data sources and systems that are otherwise inaccessible. ## [](#next-steps)Next steps - [Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) - [Quickstart for Administrators](https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart/) - [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) --- # Page 71: Agentic Data Plane Quickstart for Administrators **URL**: https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart.md --- # Agentic Data Plane Quickstart for Administrators > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Agentic Data Plane Quickstart for Administrators latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: adp-quickstart page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: adp-quickstart.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/get-started/pages/adp-quickstart.adoc description: "Set up Redpanda Agentic Data Plane for your organization: configure an LLM provider with a key stored in the platform secret store, verify it, and hand off to your agent builders." page-topic-type: quickstart personas: platform_engineer learning-objective-1: Configure your first LLM provider in Agentic Data Plane with an Anthropic API key learning-objective-2: Verify the provider is enabled and its connection is active learning-objective-3: Grant your builders access and point them to the agent builder quickstart page-git-created-date: "2026-06-04" page-git-modified-date: "2026-08-07" --- This quickstart sets up Redpanda Agentic Data Plane for your organization. An administrator configures a [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) provider. This is a one-time task. Until a provider exists, no one in your organization can build [agents](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent) against the platform. You get an Anthropic API key, create the provider with the key stored in the platform secret store, confirm the connection is active, and hand off to your builders. This quickstart configures an Anthropic provider, but Agentic Data Plane also supports OpenAI, Google AI, AWS Bedrock, and OpenAI-compatible endpoints. If your organization already has an LLM provider configured and you want to build an agent, go to the [agent builder quickstart](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) instead. After reading this page, you will be able to: - Configure your first LLM provider in Agentic Data Plane with an Anthropic API key - Verify the provider is enabled and its connection is active - Grant your builders access and point them to the agent builder quickstart ## [](#prerequisites)Prerequisites - The [Admin role](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) in Agentic Data Plane. Admin is the only built-in role that carries Agentic Data Plane permissions. - An Anthropic account. If you don’t have one, the [Get an Anthropic API key](#get-an-anthropic-api-key) section walks through signup. ## [](#what-you-set-up)What you set up 1. **Anthropic API key**. Created in the Anthropic console and stored in the Agentic Data Plane secret store, so the raw key never has to live on a builder’s machine. 2. **Anthropic LLM provider**. An organization-level resource that routes Claude API calls through Agentic Data Plane so credentials, usage, and [transcripts](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript) stay on the platform. Configure it once, and every builder can select it when they create agents. When you sign in to Agentic Data Plane at [ai.redpanda.com](https://ai.redpanda.com), you land on the Home page: a snapshot of items that need your attention, recent gateway traffic, month-to-date spend and token totals, budget status, and quick actions for common tasks. Start the tasks in this guide from the sidebar. ## [](#get-an-anthropic-api-key)Get an Anthropic API key Anthropic requires a payment method before it issues an API key. Add billing first, then create the key. 1. Open [platform.claude.com](https://platform.claude.com) and sign in, or create a new account. 2. Add a payment method to your account if you haven’t already. Anthropic requires a payment method before it issues API keys. (Optional: add a small amount of prepaid credit so the first calls don’t fail on a fresh account.) 3. In the sidebar, select a workspace other than `Claude Code`. The `Claude Code` workspace uses a separate key-creation flow that doesn’t fit this quickstart. 4. In the sidebar, click **API keys**. 5. Click **Create key**. In the Create API key dialog, name the key `adp-quickstart` and click **Add**. 6. Copy the key (it starts with `sk-ant-`). > ❗ **IMPORTANT** > > Anthropic shows the API key value once. If you close the dialog without copying, you can’t recover it. Paste the key into a secure note before continuing. The next section creates the secret in Agentic Data Plane’s secret store as part of the provider form, so you don’t have to keep the key on your local machine. 7. (Optional) On the **API keys** page, confirm the new key appears in the list. ## [](#configure-your-llm-provider)Configure your LLM provider Now point Agentic Data Plane at Anthropic. Agentic Data Plane references provider credentials by secret name, not by raw value, and you create the secret inline as part of this form. 1. Sign in to Agentic Data Plane at [ai.redpanda.com](https://ai.redpanda.com). 2. In the sidebar, open **LLM Providers**. 3. Click **Add provider**. 4. `Display name`: Enter `anthropic-quickstart`. This label appears in dashboards and model selectors. Agentic Data Plane derives the provider’s resource ID from it in the background, using lowercase letters, numbers, and hyphens; the ID is used in API calls and CLI commands and can’t be changed after creation. 5. `Provider type`: Select `Anthropic`. 6. In the Credential section, select `Bring a new API key reference` and add the key inline: - `Key name`: Enter `Anthropic API key`. Agentic Data Plane stores the key in the platform secret store under an identifier derived from this name (`ANTHROPIC_API_KEY`), shown under the field. The identifier can’t be changed later. - `API key`: Paste the `sk-ant-` key you copied from Anthropic. - Click **Save key**. The key is stored encrypted and can’t be viewed again after saving, so if you mistype it, save a new key. New keys get the AI Gateway scope automatically, which makes them usable across the platform: LLM providers, MCP servers, and agents. To create secrets ahead of time or manage them later, open **Secrets Store** in the sidebar. 7. Leave `Authorization passthrough` off. This quickstart uses one shared key, not per-user OAuth. 8. Leave the default `Base URL` in place. The form prefills the standard Anthropic endpoint. 9. In the Models section, keep the default selection and confirm `claude-sonnet-4-6` is selected. It’s a good default for agents and handles multi-step tool use reliably. The catalog of available models is maintained by Redpanda and starts fully selected. The models you leave selected become the catalog this provider exposes to agents and applications; you can change the list later from the provider detail page. 10. For Transcripts, turn on `Record inputs` and `Record outputs`. Both are off by default, which limits transcripts to token counts, latency, and tool calls. Turning them on lets your builders replay full conversations, including prompts, responses, and tool arguments. This setting applies to every agent on the provider, so leave it off for providers that carry regulated or sensitive data. See [Configure transcript logging](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#transcript-logging). 11. Click **Create provider**. 12. On the provider detail page, select the **Overview** tab. Confirm the badge next to the provider name shows `Enabled` and the `Status` in the Connection card shows `Active`. If the Status badge stays in a failure state, see the [provider troubleshooting table](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#troubleshooting). The most common cause is a typo in the secret reference or a key that wasn’t issued in the workspace Agentic Data Plane can read. For the full provider field reference (transcript logging, Bedrock IAM, Anthropic authentication passthrough, OpenAI-compatible endpoints), see [Configure an LLM provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). ## [](#hand-off-to-your-builders)Hand off to your builders With the provider active, your organization is ready for agents. To get builders started: 1. Write an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) that lets your builders create MCP servers and agents. Point it at the identity provider group they already belong to and grant the whole team in one statement. Don’t reach for a role here: Admin is the only built-in role that touches Agentic Data Plane, and it grants far more than a builder needs. See [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/). Access policies are a preview capability. If they aren’t enabled for your organization yet, a custom role holding `dataplane_adp_agent_*` and `dataplane_adp_mcpserver_*` is the interim path. 2. Send them the [agent builder quickstart](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/). It assumes exactly what you just set up: a signed-in user who can create agents, and an active LLM provider. Builders pick a model in the agent form’s **Model** section, where the provider’s display name (`anthropic-quickstart`) appears next to the models it serves. They don’t see or handle the API key. ## [](#clean-up)Clean up If you set this provider up only as a test, delete it so it doesn’t continue to count against your account. 1. Delete the LLM provider (`anthropic-quickstart`). Open the provider detail page, scroll to the Delete this provider section, click **Delete**, and type `delete` to confirm. Note that the underlying secret (`ANTHROPIC_API_KEY`) is not deleted automatically. 2. (Optional) Delete the secret. Open **Secrets Store**, click `ANTHROPIC_API_KEY`, then **Delete**. 3. (Optional, on the Anthropic side) Revoke the API key. Open [platform.claude.com](https://platform.claude.com), navigate to **Manage** > **API keys** in the workspace where you created the key, find `adp-quickstart` in the list, and revoke it from the row’s action menu. If you want to keep the provider but rotate the key, edit the provider and change the `API key reference` to a new secret. ## [](#next-steps)Next steps - [Agentic Data Plane Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/) - [Configure an LLM Provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/) - [Set Up Budgets](https://docs.redpanda.com/agentic-data-plane/control/budgets/) - [Set Up Guardrails](https://docs.redpanda.com/agentic-data-plane/control/guardrails/) - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) --- # Page 72: Agentic Data Plane Quickstart for Agent Builders **URL**: https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart.md --- # Agentic Data Plane Quickstart for Agent Builders > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Agentic Data Plane Quickstart for Agent Builders latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: agent-builder-quickstart page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: agent-builder-quickstart.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/get-started/pages/agent-builder-quickstart.adoc description: Sign in to Redpanda Agentic Data Plane, explore the agents your organization already runs, then build, test, and iterate on your own agent backed by MCP tools. page-topic-type: quickstart personas: agent_builder learning-objective-1: Find and evaluate the agents that already run in your organization learning-objective-2: Build an agent that combines an LLM provider with tools from an MCP server learning-objective-3: Test, iterate on, and verify agent behavior with the Playground and Transcripts tabs page-git-created-date: "2026-07-20" page-git-modified-date: "2026-08-10" --- This quickstart takes you from zero to a working agent: you sign in, explore the agents your organization already runs, then build your own that turns a natural-language question into a real API call. By the end, you have an agent you can test, tune, and share with your team. Redpanda Agentic Data Plane is Redpanda’s platform for running and governing [AI agents](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent) on your streaming data. For the bigger picture, see the [Agentic Data Plane overview](https://docs.redpanda.com/agentic-data-plane/get-started/adp-overview/). After reading this page, you will be able to: - Find and evaluate the agents that already run in your organization - Build an agent that combines an LLM provider with tools from an MCP server - Test, iterate on, and verify agent behavior with the Playground and Transcripts tabs ## [](#prerequisites)Prerequisites > ❗ **IMPORTANT** > > Before you build, your organization needs a [large language model (LLM)](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) provider configured. If your administrator has already set one up, continue. If not, or you’re not sure, complete the [administrator quickstart](https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart/) first, then come back. It’s a one-time setup. - Access to Agentic Data Plane at [ai.redpanda.com](https://ai.redpanda.com). If you can’t sign in, ask your administrator for an invitation. - Access to create agents and MCP servers. Your administrator grants this with an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/), or with a custom role where policies aren’t enabled yet. The Writer and Reader roles don’t cover Agentic Data Plane. If a create button is disabled or a save returns a permission error, ask them to check what applies to you or your group. - At least one LLM provider configured in your organization. ## [](#what-you-build)What you build You create the following Agentic Data Plane resources, in order: 1. **PetStore [MCP server](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#mcp-server)**. Exposes the public Swagger PetStore API as a set of [MCP tools](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#mcp-tool) the agent can call. PetStore is a public demo API with no authentication, so you can focus on the flow instead of credentials. 2. **Pet store assistant agent**. Combines your organization’s LLM provider and the MCP server so a natural-language question becomes a real API call against PetStore. After the agent works, you replay one of its runs turn-by-turn on its **Transcripts** tab, then learn how to swap the demo API for a system your team actually uses. ## [](#sign-in-and-look-around)Sign in and look around Sign in to Agentic Data Plane at [ai.redpanda.com](https://ai.redpanda.com). You land on the Home page: a snapshot of items that need your attention, recent gateway traffic, month-to-date spend and token totals, budget status, and quick actions for common tasks. Start every task in this guide from the sidebar. ## [](#explore-existing-agents)Explore existing agents Before you build anything, see what already exists. Your organization may run agents you can use or learn from. 1. In the sidebar, open **Agents**. The agent registry lists every agent in your organization with its status. 2. Select an agent to open its detail page. The tabs give you a complete picture of the agent: - **Overview**: The agent’s full configuration, including its model, system prompt, and tools. - **Triggers**: Schedules and event sources that invoke the agent. - **Playground**: An interactive chat surface for testing the agent. - **Cost & Usage**: The agent’s budget status and its spend and request volume, broken down by LLM provider. - **Transcripts**: A record of every conversation against the agent. - **Permissions**: Who can act on the agent and what the agent can access. This tab appears when access policies, a preview capability, are enabled for your organization. 3. (Optional) Open the **Playground** tab of an existing agent and send it a prompt to see how it behaves. If an existing agent already covers your use case, you can stop here and use it. Otherwise, continue to build your own. ## [](#create-an-mcp-server-with-a-public-openapi-spec)Create an MCP server with a public OpenAPI spec The OpenAPI managed MCP server type takes any OpenAPI 3.x spec URL and generates one MCP tool per operation. You point it at the public Swagger PetStore, which exposes a small pet-inventory API with no authentication. > 📝 **NOTE** > > PetStore is a public demo that Swagger hosts for testing and is occasionally unavailable. Before you start, open [the spec URL](https://petstore3.swagger.io/api/v3/openapi.json) in your browser. If it doesn’t return JSON, wait a few minutes and retry; PetStore usually recovers quickly. Checking now saves you from debugging your agent later for an upstream outage that isn’t your fault. 1. In the sidebar, open **MCP Servers** and click **Add MCP server**. 2. Search for and select the **OpenAPI** card from the marketplace picker. ![The Create MCP Server marketplace picker filtered by a search for openAPI, showing the Remote (Proxied) card and the OpenAPI card](https://docs.redpanda.com/agentic-data-plane/get-started/_images/create-mcp-server-picker.png) 3. `Name`: Enter `petstore-quickstart`. 4. `Description`: Enter `Public PetStore API exposed as MCP tools for quickstart testing`. Optional, but helps you find the server later. 5. Leave the `Enable Code Mode` toggle off. Code mode adds search and execute tools that let agents run sandboxed code against the server’s tools. The PetStore catalog is small enough that you don’t need it for this quickstart. 6. Leave the Output format section at its defaults. 7. `Spec`: Enter `[https://petstore3.swagger.io/api/v3/openapi.json](https://petstore3.swagger.io/api/v3/openapi.json)`. 8. `Base URL`: Leave blank. The PetStore spec already declares the right `servers` block, so no override is needed. 9. Skip all other optional configuration fields. 10. For the authentication mode, select `No Auth`. PetStore is a public API. 11. Click **Create server**. 12. Redpanda fetches the spec, parses it, and generates one MCP tool per operation. When the server is ready, open its detail page. 13. On the **Connection** tab, copy the `Server URL`. MCP clients use this URL to connect to the server. 14. Open the **Inspector** tab. Redpanda runs a live `tools/list` against the server and lists every tool it discovered. Confirm the list includes `findpetsbystatus`, `getpetbyid`, `findpetsbytags`, and `addpet`. The OpenAPI MCP server lowercases tool names when it generates them from the spec’s `operationId` values. The exact tool count depends on the operations defined in the spec at fetch time. A populated list confirms the connection works. If the list is empty or the tab shows an error, see [OpenAPI MCP troubleshooting](https://docs.redpanda.com/agentic-data-plane/connect/managed/openapi/#troubleshooting). For the full OpenAPI MCP reference (authentication modes, operation filters, base URL overrides), see [OpenAPI managed MCP server](https://docs.redpanda.com/agentic-data-plane/connect/managed/openapi/). ## [](#build-your-agent)Build your agent Combine your organization’s LLM provider and the PetStore MCP server into an agent. Agentic Data Plane configures a managed agent on a single canvas titled **New managed agent**, with sections for **Identity**, **Model**, **Instructions**, and **Tools**, plus a collapsible **Advanced** area for subagents and tags. Required fields are marked with an asterisk (`*`). For the full reference, see [Create an agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/). 1. In the sidebar, open **Agents**. 2. Click **Create agent**. 3. Choose how the agent runs. Click `Redpanda manages it`, so Redpanda deploys, runs, and observes the agent for you. ![The runtime choice in the create-agent flow, with a Redpanda manages it card for the managed runtime and an I host it myself card for self-managed agents](https://docs.redpanda.com/agentic-data-plane/shared/_images/create-agent-runtime-choice.png) > 📝 **NOTE** > > The other option, `I host it myself`, registers an agent you run on your own infrastructure. This quickstart uses the managed runtime. 4. On the **New managed agent** page, in the **Identity** section, set the following: - `Name`: Enter `pet-store-assistant`. This name appears in the agent registry list and the detail header. Agentic Data Plane derives the agent’s resource ID from it when you create the agent; you don’t enter the ID directly, and it can’t be changed after creation. - `Description`: Enter `Quickstart agent that answers questions about the PetStore inventory`. 5. For **Model**, open the picker and select `claude-sonnet-4-6`. It handles multi-step tool use reliably, which is what an agent needs. The picker is a single searchable list of every model your organization’s configured providers expose, with the provider that serves each model shown next to it. Selecting the model also selects its provider; for this quickstart that’s the `anthropic-quickstart` provider your administrator configured. When more than one provider serves the same model, each pairing is its own entry, so pick the one that names `anthropic-quickstart`. If the picker is empty, no provider exists yet; ask your administrator to complete the [administrator quickstart](https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart/). > 📝 **NOTE** > > These names assume the Anthropic provider from the administrator quickstart. If your organization configured a different provider, pick any model it offers that supports tool use. 6. For **Instructions**, paste in the following system prompt: ```text You are a pet store inventory assistant. You have access to PetStore API tools through MCP. Use them to answer questions about pets. The most useful read tools are: - findpetsbystatus: List pets by status (available, pending, sold) - getpetbyid: Look up a specific pet by ID - findpetsbytags: Search by tag Rules: - Always call a tool before answering. Don't make up pet data. - After each tool call, summarize what the tool returned and cite the tool name in your response. - If a tool fails or returns no data, say so and stop. Don't retry with invented IDs. - Stick to read operations for this quickstart. Don't add, update, or delete pets. ``` 7. For **Tools**, select the `petstore-quickstart` checkbox in the server list. If your organization runs many MCP servers, a **Search MCP servers** box appears above the list; use it to find the server. (Don’t click **Add MCP server** on the left. That builds a new server, and you already have one.) The agent discovers the server’s tools at runtime; you don’t pre-select individual tools, and the system prompt’s "stick to read operations" rule constrains behavior. You can change this list at any time. 8. Leave the **Advanced** area collapsed. The quickstart uses a single agent with no subagents or tags. 9. Click **Create agent**. If the button is disabled, the helper text next to it tells you which required field is still missing (a name and a model). 10. Wait for the agent status to change from `Starting` to `Running`. ## [](#test-the-agent-in-the-playground)Test the agent in the Playground The **Playground** tab is a chat surface for the agent. Type a prompt into the input at the bottom of the page and click the paper-plane icon in the input’s bottom-right corner to send. Each prompt creates a Task card in the conversation that walks through the agent’s execution turn-by-turn: TaskStatusUpdate entries (`Submitted` → `Working` → `Completed`) with token counts, individual tool-call cards showing the tool name and latency, and an Artifact card holding the final response. **Clear context** (at the bottom of the input) resets the conversation between tests. 1. On the agent detail page, open the **Playground** tab. 2. Send the first prompt: ```text What pets are available right now? ``` The agent calls `findpetsbystatus` with `status=available` and returns a summary of the pets, naming the tool. This PetStore endpoint is occasionally flaky; if it returns an HTTP 500, retry the prompt. 3. (Optional) Send an open-ended prompt to confirm multi-step reasoning: ```text Find a pet tagged "friendly" and tell me whether it's available. ``` The agent may call `findpetsbytags` first, then `getpetbyid` for one of the matches, then summarize. If the agent hallucinates data instead of calling a tool, tighten the system prompt’s "Always call a tool before answering" rule. See [system prompt best practices](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) for patterns. If a tool card shows an HTTP 500 error and the Artifact apologizes about the pet store API being unavailable, that’s the public Swagger PetStore being intermittently down. The agent is behaving correctly (the "if a tool fails, say so and stop" rule in the system prompt is what produced the apology). Retry the prompt; PetStore usually recovers within a few minutes. ## [](#iterate-on-your-agent)Iterate on your agent Agent configuration isn’t fixed after you create it. Edit the system prompt or model, save, and re-test in the Playground to see how behavior changes. 1. On the agent detail page, open the **Overview** tab. The tab shows the agent’s full configuration, organized into sections that you edit and save one at a time. 2. In the System prompt section, select the edit icon and adjust the prompt to change how the agent responds. For example: - Add a constraint: `Never call more than one tool per response.` - Change the output format: `Always format pet details as a bulleted list.` - Add a confirmation step: `Before answering, restate which tool you plan to call and why.` 3. Click **Save**. 4. Open the **Playground** tab, click **Clear context**, and re-send one of the earlier prompts to compare the agent’s behavior against your change. For system prompt patterns and anti-patterns, see [Write effective system prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/). ## [](#see-what-your-agent-did)See what your agent did Every conversation with the agent is recorded as a [transcript](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript) you can replay turn-by-turn. Your Playground tests are already there. > 📝 **NOTE** > > The **Chat** view and the tool arguments and results depend on message recording, which the LLM provider controls with the `Record inputs` and `Record outputs` toggles, both off by default. The paired [administrator quickstart](https://docs.redpanda.com/agentic-data-plane/get-started/adp-quickstart/) turns them on. If your administrator left them off, transcripts still capture token counts, latency, and tool calls, but the message text is empty. For details, see [Configure transcript logging](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/#transcript-logging). 1. On the agent detail page, open the **Transcripts** tab. The tab lists the agent’s recent conversations, one row per conversation, with its start time, duration, turn count, status, and token total. 2. Select the conversation from your Playground test. 3. Toggle between the two views: - **Chat**: The user-visible exchange, as a conversation. - **Detailed**: Adds per-turn metadata: latency, input/output token splits, and each tool call with its arguments, result, latency, and status. The **Detailed** view is where you verify the agent did what you asked: which tool it called, with which arguments, and what came back. For the full transcript reference, see [See what your agent did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/). ## [](#make-it-real)Make it real PetStore proves the flow. To build something your team uses, swap the demo API for a real system: 1. In the sidebar, open **MCP Servers** and click **Add MCP server**. 2. Pick a type from the marketplace picker that matches a system your team uses, such as Slack, Jira, or a SQL database. See the [managed MCP catalog](https://docs.redpanda.com/agentic-data-plane/connect/managed/managed-catalog/) for every type Redpanda hosts, with links to setup guides. 3. Configure its authentication. Unlike PetStore, real systems need credentials: an API key stored as a secret, or per-user OAuth. Some authentication modes need one-time setup from your administrator; see [Set up authentication](https://docs.redpanda.com/agentic-data-plane/connect/authentication/). 4. Add the new server to your agent’s **Tools** section, and update the system prompt to describe the new tools and rules. Your own API works too: if it has an OpenAPI 3.x spec, the same OpenAPI server type you used for PetStore exposes it as tools. See [OpenAPI Managed MCP Server](https://docs.redpanda.com/agentic-data-plane/connect/managed/openapi/). ## [](#use-your-agent-beyond-the-ui)Use your agent beyond the UI Your agent isn’t limited to the Agentic Data Plane UI. After it’s running, you can reach it and its tools from other software: - Surface your MCP tools in a chat client like Claude Desktop or ChatGPT: [Connect Remote MCP Clients](https://docs.redpanda.com/agentic-data-plane/connect/remote-mcp-clients/) - Call the agent from your own application over its A2A endpoint: [Connect to the agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/#connect-to-the-agent) - Let the agent respond in Microsoft Teams: [Trigger Agents](https://docs.redpanda.com/agentic-data-plane/connect/triggers/overview/) ## [](#clean-up)Clean up If this was a one-off exercise, delete the resources you created so they don’t continue to count against your organization. 1. Delete the agent (`pet-store-assistant`) first. Open the agent detail page, scroll to the Delete this agent section, click **Delete**, and type `delete` to confirm. 2. Delete the MCP server (`petstore-quickstart`). Same pattern. Leave your organization’s LLM provider alone. It’s a shared resource that other builders use. ## [](#next-steps)Next steps - [Use the Agentic Data Plane CLI](https://docs.redpanda.com/agentic-data-plane/cli/) - [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) --- # Page 73: Monitor & Debug **URL**: https://docs.redpanda.com/agentic-data-plane/monitor.md --- # Monitor & Debug > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Monitor & Debug latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/index.adoc description: See what your agents did, investigate runs, and check speed, cost, and errors. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-06-18" --- See what your agents are doing and why. Monitor agent activity, read individual run transcripts, troubleshoot failures, and track speed, cost, and errors. - [How Observability Works](concepts/) Understand how Redpanda captures end-to-end execution transcripts on an immutable distributed log for agent governance and observability. - [Monitor Agent Activity](monitor-agents/) Monitor agent execution, inspect available MCP tools, analyze conversation history, track token usage, and debug issues using Playground and Transcripts. - [Visualize Your Agent Network](agent-network/) See a live map of your agents, the models they call, and the MCP servers they use, with usage, cost, and health for a chosen time window. - [Review the Audit Log](audit-log/) Review authorization decisions across Agentic Data Plane, filter events, and inspect who acted on which resource and whether the request was allowed. - [See What Your Agent Did](transcripts/) Open a transcript in Agentic Data Plane, read the conversation turn by turn, and investigate errors, costs, and performance. - [Troubleshoot Agents](troubleshoot-ai-agents/) Diagnose and fix common issues with AI agents including deployment failures, runtime behavior problems, and tool execution errors. --- # Page 74: Visualize Your Agent Network **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/agent-network.md --- # Visualize Your Agent Network > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Visualize Your Agent Network latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: agent-network page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: agent-network.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/agent-network.adoc description: See a live map of your agents, the models they call, and the MCP servers they use, with usage, cost, and health for a chosen time window. page-topic-type: how-to personas: platform_engineer, agent_builder, security_compliance_lead learning-objective-1: Trace which models and tools each agent uses in the network graph learning-objective-2: Scope the view to a time window and health status learning-objective-3: Inspect an agent, LLM provider, MCP server, or tool to check usage, cost, and errors page-git-created-date: "2026-07-31" page-git-modified-date: "2026-08-10" --- The **Agent Network** view is a live map of your Agentic Data Plane: every agent, the [LLM](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#large-language-model-llm) providers it calls, and the [MCP servers](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#mcp-server) and tools it uses, drawn as a graph with usage, cost, and health for the time window you choose. Use it to see at a glance what your agents are actually doing, which models and tools carry the most traffic, and where errors are clustering. After reading this page, you will be able to: - Trace which models and tools each agent uses in the network graph - Scope the view to a time window and health status - Inspect an agent, LLM provider, MCP server, or tool to check usage, cost, and errors The view draws on the same recorded telemetry that powers [transcripts](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/), enriched with cost data. It shows what actually ran during the selected window. A configured resource with no traffic in the window still appears, with its metrics marked as having no recorded data yet. > 📝 **NOTE** > > The Agent Network view is a preview capability, available when your organization has it enabled. If you don’t see **Agent Network** under **Agents** in the sidebar, it isn’t enabled for your organization. ## [](#prerequisites)Prerequisites The Agent Network view requires the `dataplane_adp_agentnetwork_get` permission. See [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/). ## [](#open-the-agent-network-view)Open the Agent Network view Open **Agents** in the sidebar, then click **Agent Network**. The header shows your agent and MCP server counts, plus conversations, tool calls, cost, and tokens for the selected time window. These totals always cover the whole network, even when the graph is filtered or showing only the busiest agents. ## [](#read-the-graph)Read the graph The graph arranges nodes in columns, with traffic flowing left to right: - **Agents**: Your agents, grouped into a cluster you can click to expand. - **LLM providers**: The providers agents send model requests to, with cost and token usage on each card. When you have many providers, the column paginates. - **MCP servers**: The servers agents call tools on. Click a server to expand its tools into their own column, showing per-tool call counts. Lines connect each agent to the providers and servers it used during the window, and line thickness shows traffic volume. Agent-to-provider lines can represent either token usage or cost: switch between **Tokens** and **Cost** in the on-canvas legend. Agent-to-server lines always represent tool calls. ## [](#scope-the-view)Scope the view Use the toolbar to control what the graph shows: - **Window**: Choose from last hour, last 24 hours (the default), last 7 days, or last 30 days. - **Status**: Show all nodes, only nodes with activity, or only nodes with errors. - **Sort LLMs**: Order providers by cost or token use, or list them alphabetically. - **Sort MCPs**: Order servers by tool calls, or list them alphabetically. - **Refresh**: Reload the graph with current data. - **Reset view**: Return to the default layout and selection. Click a node to inspect it, click it again to deselect, and press Esc to reset the selection. The view state persists in the URL, so you can bookmark a scoped view or share it with your team. ## [](#inspect-a-node)Inspect a node Selecting a node opens a detail panel: - **Agent**: The agent’s model and description, plus window metrics: conversations, spend, turns per conversation, and 95th-percentile request latency. - **LLM provider**: Cost and token usage, split by the models observed in the window. - **MCP server**: Tool count, connected agent count, tool calls, and connection status, plus the server’s tools and the agents and users that called it during the window. Click an agent to jump to its details. - **Tool**: The tool’s description and its usage in the window: how many agents called it, how many calls it received, the server it belongs to, and its status. ## [](#monitor-health)Monitor health Each node carries a health state derived from its error rate in the window: healthy, degraded, or error. Use the **Status** filter to isolate nodes with errors. The view flags every MCP server that needs attention, whether its credentials fail to resolve (for example, an expired or missing user connection) or its tool calls fail on their own. Each alert names the cause and shows how many agents are affected and how many calls failed, so you can act before more calls fail. For a credential failure, reconnect the user connection. See [Configure User-Delegated OAuth](https://docs.redpanda.com/agentic-data-plane/connect/user-delegated-oauth/) for managing user connections. ## [](#next-steps)Next steps - [Monitor Agent Activity](https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents/) - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [Troubleshoot Agents](https://docs.redpanda.com/agentic-data-plane/monitor/troubleshoot-ai-agents/) --- # Page 75: Review the Audit Log **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/audit-log.md --- # Review the Audit Log > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Review the Audit Log latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: audit-log page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: audit-log.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/audit-log.adoc description: Review authorization decisions across Agentic Data Plane, filter events, and inspect who acted on which resource and whether the request was allowed. page-topic-type: how-to personas: security_compliance_lead, platform_engineer learning-objective-1: Filter authorization events by time, outcome, subsystem, or search term learning-objective-2: Inspect the actor, action, resource, source, and policy behind an authorization decision learning-objective-3: Choose the audit log or a transcript for an investigation page-git-created-date: "2026-08-04" page-git-modified-date: "2026-08-10" --- The audit log records authorization decisions across Agentic Data Plane. Use it to answer who performed an action, which resource they acted on, where the request came from, and whether the request was allowed or denied. After reading this page, you will be able to: - Filter authorization events by time, outcome, subsystem, or search term - Inspect the actor, action, resource, source, and policy behind an authorization decision - Choose the audit log or a transcript for an investigation > 📝 **NOTE** > > The audit log is a preview capability. Access to the page is granted per organization, and audit recording is enabled per deployment; the two are independent. If you don’t see **Audit Log** in the sidebar, it isn’t enabled for your organization. > > A deployment can therefore show the page with no recorded events. If the table is empty for a period you know had activity, see [Troubleshooting](#troubleshoot-the-audit-log). ## [](#prerequisites)Prerequisites Your account needs the `dataplane_adp_auditlog_list` permission to open the audit log and view event details. Outside the Admin role, audit log reads come from an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) naming `Action::"AuditLog.list"`. See [Audit log permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#audit-log-permissions). > ❗ **IMPORTANT** > > Audit events can include user identities, resource names, network information, configuration changes, and captured request or response data. Grant audit log access only to people who need it, and review events before sharing them. ## [](#open-the-audit-log)Open the audit log 1. Open **Audit Log** in the sidebar. 2. Review the most recent events. The default time range is Last 24 hours. 3. Select a row to open its details. The table shows these fields: | Field | Description | | --- | --- | | Actor | The user accountable for the action. When an agent or another identity acted for the user, the table also identifies that caller when available. | | Subsystem | The part of Agentic Data Plane that handled the action. | | Action | The operation and service involved. | | Resource | The type and identifier of the affected resource. When one call decided on more than one resource, this shows a count, such as 3 resources, and the detail panel lists each resource with its own decision. | | Outcome | Whether the authorization decision allowed the action, denied it, or, for a call that decided on more than one resource, allowed some resources and denied others (Partial). Hover over the outcome to see the deciding policy when available. | | Time | When the decision occurred. | ## [](#filter-events)Filter events Use the controls above the table to narrow an investigation: - **Search**: Match a keyword against the event messages. - **Actor**: Enter an exact actor identity, such as an email address, to show only that actor’s events. - **Resource type**: Enter a full namespaced resource type, such as `redpanda.agent`, to show events for that type. Matching ignores case. The table’s Resource column shows the type without its namespace prefix, so `redpanda.agent` appears as `agent`. - **Resource ID**: Enter an exact resource identifier to show events for a single resource. Matching is case-sensitive. - **Outcome**: Show allowed actions, denied actions, or partial outcomes. Select more than one to combine them. - **Subsystem**: Select one or more of Management API, LLM proxy, MCP gateway, A2A proxy, and Spending. - **Time range**: Choose a duration of Last hour, Last 24 hours, Last 7 days, Last 30 days, or All time. To review a past window, set an end time, and the duration counts back from that time instead of from now. Filters work together. For example, select Denied, choose MCP gateway, and set Last hour to investigate recent MCP authorization failures. ## [](#inspect-an-event)Inspect an event Select an event to open its detail panel. The panel identifies the actor, outcome, subsystem, operation, resource, deciding policy, source, and time when that information is available. When a call decided on more than one resource, the panel opens with a Resources section that lists each resource and its own decision. Select a resource to see its individual outcome and deciding policy in the panel. Depending on the action, the panel may also show: - The agent or caller that acted for a user - The event class and activity - A configuration change with before and after values - Captured request data - Response status, error, or captured response data Captured values may be redacted, truncated, or absent. An empty field does not mean that no activity occurred. Use the event summary and outcome as the primary record of the authorization decision. ## [](#audit-log-compared-to-transcripts)Audit log compared to transcripts Use the audit log for access and accountability questions: - Who attempted an action? - Which resource and subsystem were involved? - Was the action allowed or denied? - Which policy made the decision? Use [transcripts](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) for execution questions about an agent conversation, including model turns, tool calls, errors, timing, and token usage. An investigation may need both views: start with the audit log to identify an authorization decision, then inspect the related agent activity when you need execution details. ## [](#troubleshoot-the-audit-log)Troubleshoot the audit log | Issue | What to do | | --- | --- | | Audit Log is missing from the sidebar | The preview may not be enabled for your organization. Contact your Redpanda representative. | | You cannot open the audit log | Ask an administrator to confirm that your account has dataplane_adp_auditlog_list, from the Admin role or from an access policy naming Action::"AuditLog.list". | | No events match | Clear the search and filters, then widen the time range. The default view includes only the last 24 hours. If the table stays empty, audit recording may not be enabled for this deployment yet; page access and recording are gated independently. | | An event has no request, response, or configuration details | Not every event includes these fields. Use the event’s actor, action, resource, outcome, and time to continue the investigation. | ## [](#next-steps)Next steps - [How Redpanda stores audit events](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#how-redpanda-stores-audit-events) - [Transcripts compared to audit logs](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#transcripts-compared-to-audit-logs) - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [Control Who Can Do What](https://docs.redpanda.com/agentic-data-plane/control/permissions-overview/) --- # Page 76: How Observability Works **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/concepts.md --- # How Observability Works > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: How Observability Works latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: concepts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: concepts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/concepts.adoc description: Understand how Redpanda captures end-to-end execution transcripts on an immutable distributed log for agent governance and observability. page-topic-type: concepts personas: domain_evaluator, security_compliance_lead, platform_engineer, agent_builder learning-objective-1: Explain how transcripts and spans capture execution flow learning-objective-2: Interpret transcript structure for debugging and monitoring learning-objective-3: Distinguish between transcripts and audit logs page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- Redpanda provides complete observability and governance for AI agents through automated [transcript](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#transcript) capture. Every agent execution, from simple tool calls to complex multi-agent, multi-turn workflows, generates a permanent, write-once record stored on Redpanda’s [log](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#log). The structure of every execution (its reasoning steps, tool invocations, model interactions, and data flows) is captured with 100% sampling and no gaps. Agentic Data Plane records the full message content of each step (the prompts sent to the model and the responses it returns) only when you turn on input and output capture for the LLM provider, because that content can contain sensitive data. See [Configure an LLM Provider](https://docs.redpanda.com/agentic-data-plane/gateway/configure-provider/). With transcripts, organizations gain the ability to debug agent behavior, identify performance bottlenecks, meet regulatory compliance requirements, and maintain accountability for AI-driven decisions. Transcripts use OpenTelemetry standards and [Raft](https://raft.github.io/)\-based consensus for correctness, establishing a trustworthy foundation for agent governance. After reading this page, you will be able to: - Explain how transcripts and spans capture execution flow - Interpret transcript structure for debugging and monitoring - Distinguish between transcripts and audit logs ## [](#what-are-transcripts)What are transcripts A transcript records the complete execution of an agentic behavior from start to finish. It captures every step (across multiple agents, tools, models, and services) in a single, traceable record. The AI Gateway and every [agent](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#ai-agent) and [MCP server](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#mcp-server) in your Agentic Data Plane automatically emit OpenTelemetry traces to a [topic](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#topic) called `redpanda.otel_traces`. Redpanda’s immutable distributed log stores these traces. Transcripts capture: - Tool invocations and results - Agent reasoning steps - Data processing operations - External API calls - Error conditions - Performance metrics With 100% sampling, every operation is captured with no gaps. Agentic Data Plane records message content (the prompts and model responses behind each step) only when you enable input and output capture on the LLM provider. Until then, transcripts hold the execution structure and metadata but not the raw prompt and response bodies. The underlying storage uses a distributed log built on Raft consensus (with TLA+ proven correctness), giving transcripts a trustworthy, immutable record for governance, debugging, and performance analysis. ## [](#traces-and-spans)Traces and spans [OpenTelemetry](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#opentelemetry) traces provide a complete picture of how a request flows through your system: - A _trace_ represents the entire lifecycle of a request (for example, a tool invocation from start to finish). - A _span_ represents a single unit of work within that trace (such as a data processing operation or an external API call). - A trace contains one or more spans organized hierarchically, showing how operations relate to each other. ## [](#agent-transcript-hierarchy)Agent transcript hierarchy Agent executions create a hierarchy of spans that reflect how agents process requests. Understanding this hierarchy helps you interpret agent behavior and identify where issues occur. ### [](#agent-span-types)Agent span types Agent transcripts contain these span types: | Span Type | Description | Use To | | --- | --- | --- | | ai-agent | Top-level span representing the entire agent invocation from start to finish. Includes all processing time, from receiving the request through executing the reasoning loop, calling tools, and returning the final response. | Measure total request duration and identify slow agent invocations. | | agent | Internal agent processing that represents reasoning and decision-making. Shows time spent in the LLM reasoning loop, including context processing, tool selection, and response generation. Multiple agent spans may appear when the agent iterates through its reasoning loop. | Track reasoning time and identify iteration patterns. | | invoke_agent | Agent and sub-agent invocation in multi-agent architectures, following the OpenTelemetry agent invocation semantic conventions. Represents one agent calling another through the A2A protocol. | Trace calls between root agents and sub-agents, measure cross-agent latency, and identify which sub-agent was invoked. | | openai, anthropic, or other LLM providers | LLM provider API call showing calls to the language model. The span name matches the provider, and attributes typically include the model name (like gpt-5.2 or claude-sonnet-4-5). | Identify which model was called, measure LLM response time, and debug LLM API errors. | | rpcn-mcp | MCP tool invocation representing calls to Remote MCP servers. Shows tool execution time, including network latency and tool processing. | Measure tool execution time and identify slow MCP tool calls. | ### [](#typical-agent-execution-flow)Typical agent execution flow A simple agent request creates this hierarchy: ai-agent (6.65 seconds) ├── agent (6.41 seconds) │ ├── invoke\_agent: customer-support-agent (6.39 seconds) │ │ └── openai: chat gpt-5.2 (6.2 seconds) This hierarchy shows that the LLM API call (6.2 seconds) accounts for most of the total agent invocation time (6.65 seconds), revealing the bottleneck in this execution flow. ## [](#mcp-server-transcript-hierarchy)MCP server transcript hierarchy MCP server tool invocations produce a different span hierarchy focused on tool execution and internal processing. This structure reveals performance bottlenecks and helps debug tool-specific issues. ### [](#mcp-server-span-types)MCP server span types MCP server transcripts contain these span types: | Span Type | Description | Use To | | --- | --- | --- | | mcp-{server-id} | Top-level span representing the entire MCP server invocation. The server ID uniquely identifies the MCP server instance. This span encompasses all tool execution from request receipt to response completion. | Measure total MCP server response time and identify slow tool invocations. | | service | Internal service processing span that appears at multiple levels in the hierarchy. Represents internal MCP server operations including routing, processing, and component execution. | Track internal processing overhead and identify where time is spent in the service layer. | | Tool name (for example, get_order_status, get_customer_history) | The specific MCP tool being invoked. This span name matches the tool name defined in the MCP server configuration. | Identify which tool was called and measure tool-specific execution time. | | processors | Processor pipeline execution span showing the collection of processors that process the tool’s data. Appears as a child of the tool invocation span. | Measure total processor pipeline execution time. | | Processor name (for example, mapping, http, branch) | Individual processor execution span representing a single internal processor in the MCP server pipeline. The span name matches the processor type. | Identify slow processors and debug processing logic. | ### [](#typical-mcp-server-execution-flow)Typical MCP server execution flow An MCP tool invocation creates this hierarchy: mcp-d5mnvn251oos73 (4.00 seconds) ├── service > get\_order\_status (4.07 seconds) │ └── service > processors (43 microseconds) │ └── service > mapping (18 microseconds) This shows: 1. Total MCP server invocation: 4.00 seconds 2. Tool execution (get\_order\_status): 4.07 seconds 3. Processor pipeline: 43 microseconds 4. Mapping processor: 18 microseconds (data transformation) The majority of time (4+ seconds) is spent in tool execution, while internal processing (mapping) takes only microseconds. This indicates the tool itself (likely making external API calls or database queries) is the bottleneck, not the MCP server’s internal processing. ## [](#transcript-layers-and-scope)Transcript layers and scope Transcripts contain multiple layers of instrumentation, from HTTP transport through application logic to external service calls. The `scope.name` field in each span identifies which instrumentation layer created that span. ### [](#instrumentation-layers)Instrumentation layers A complete agent transcript includes these layers: | Layer | Scope Name | Purpose | | --- | --- | --- | | HTTP Server | go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp | HTTP transport layer receiving requests. Shows request/response sizes, status codes, client addresses, and network details. | | AI SDK (Agent) | github.com/redpanda-data/ai-sdk-go/plugins/otel | Agent application logic. Shows agent invocations, LLM calls, tool executions, conversation IDs, token usage, and model details. Includes gen_ai.* semantic convention attributes. | | HTTP Client | go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp | Outbound HTTP calls from agent to MCP servers. Shows target URLs, request methods, and response codes. | | MCP Server | rpcn-mcp | MCP server tool execution. Shows tool name, input parameters, result size, and execution time. Appears as a separate service.name in resource attributes. | ### [](#how-layers-connect)How layers connect Layers connect through parent-child relationships in a single transcript: ai-agent-http-server (HTTP Server layer) └── invoke\_agent customer-support-agent (AI SDK layer) ├── chat gpt-5-nano (AI SDK layer, LLM call 1) ├── execute\_tool get\_order\_status (AI SDK layer) │ └── HTTP POST (HTTP Client layer) │ └── get\_order\_status (MCP Server layer, different service) │ └── processors (MCP Server layer) └── chat gpt-5-nano (AI SDK layer, LLM call 2) The request flow demonstrates: 1. HTTP request arrives at agent 2. Agent invokes sub-agent 3. Agent makes first LLM call to decide what to do 4. Agent executes tool, making HTTP call to MCP server 5. MCP server processes tool through its pipeline 6. Agent makes second LLM call with tool results 7. Response returns through HTTP layer ### [](#cross-service-transcripts)Cross-service transcripts When agents call MCP tools, the transcript spans multiple services. Each service has a different `service.name` in the resource attributes: - Agent spans: `"service.name": "ai-agent"` - MCP server spans: `"service.name": "mcp-{server-id}"` Both use the same `traceId`, allowing you to follow a request across service boundaries. ### [](#key-attributes-by-layer)Key attributes by layer Different layers expose different attributes: HTTP Server/Client layer (following [OpenTelemetry semantic conventions for HTTP](https://opentelemetry.io/docs/specs/semconv/http/http-spans/)): - `http.request.method`, `http.response.status_code` - `server.address`, `url.path`, `url.full` - `network.peer.address`, `network.peer.port` - `http.request.body.size`, `http.response.body.size` AI SDK layer (following [OpenTelemetry semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/)): - `gen_ai.operation.name`: Operation type (`invoke_agent`, `chat`, `execute_tool`) - `gen_ai.conversation.id`: Links spans to the same conversation session. A conversation may include multiple agent invocations (one per user request). Each invocation creates a separate trace that shares the same conversation ID. - `gen_ai.agent.name`: Sub-agent name for multi-agent systems - `gen_ai.provider.name`, `gen_ai.request.model`: LLM provider and model - `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`: Token consumption - `gen_ai.usage.input_tokens_cost_usd`, `gen_ai.usage.output_tokens_cost_usd`: USD cost per call, derived from token counts and a per-model pricing table - `gen_ai.tool.name`, `gen_ai.tool.call.arguments`: Tool execution details - `gen_ai.input.messages`, `gen_ai.output.messages`: Full LLM conversation context MCP Server layer: - Tool-specific attributes like `order_id`, `customer_id` - `result_prefix`, `result_length`: Tool result metadata - Component-specific attributes from your tool configuration The `scope.name` field identifies which instrumentation layer created each span. ## [](#understand-the-transcript-structure)Understand the transcript structure Each span captures a unit of work. Here’s what a typical MCP tool invocation looks like: ```json { "traceId": "71cad555b35602fbb35f035d6114db54", "spanId": "43ad6bc31a826afd", "name": "http_processor", "attributes": [ {"key": "city_name", "value": {"stringValue": "london"}}, {"key": "result_length", "value": {"intValue": "198"}} ], "startTimeUnixNano": "1765198415253280028", "endTimeUnixNano": "1765198424660663434", "instrumentationScope": {"name": "rpcn-mcp"}, "status": {"code": 0, "message": ""} } ``` - `traceId` links all spans in the same request across services - `spanId` uniquely identifies this span - `name` identifies the operation or tool - `instrumentationScope.name` identifies which layer created the span (for example, `rpcn-mcp` for MCP tools) - `attributes` contain operation-specific metadata - `status.code` indicates success (0) or error (2) ### [](#parent-child-relationships)Parent-child relationships Transcripts show how operations relate. A tool invocation (parent) may trigger internal operations (children): ```json { "traceId": "71cad555b35602fbb35f035d6114db54", "spanId": "ed45544a7d7b08d4", "parentSpanId": "43ad6bc31a826afd", "name": "http", "instrumentationScope": {"name": "redpanda-connect"}, "status": {"code": 0, "message": ""} } ``` The `parentSpanId` links this child span to the parent tool invocation. Both share the same `traceId` so you can reconstruct the complete operation. ## [](#error-events-in-transcripts)Error events in transcripts When something goes wrong, transcripts capture error details: ```json { "traceId": "71cad555b35602fbb35f035d6114db54", "spanId": "ba332199f3af6d7f", "parentSpanId": "43ad6bc31a826afd", "name": "http_request", "events": [ { "name": "event", "timeUnixNano": "1765198420254169629", "attributes": [{"key": "error", "value": {"stringValue": "type"}}] } ], "status": {"code": 0, "message": ""} } ``` The `events` array captures what happened and when. Use `timeUnixNano` to see exactly when the error occurred within the operation. ## [](#opentelemetry-traces-topic)How Redpanda stores trace data The `redpanda.otel_traces` topic stores OpenTelemetry spans using Redpanda’s [Schema Registry](https://docs.redpanda.com/agentic-data-plane/reference/glossary/#schema-registry) wire format, with a custom Protobuf schema named `redpanda.otel_traces-value` that follows the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/) specification. Spans include attributes following OpenTelemetry [semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/), such as `gen_ai.operation.name` and `gen_ai.conversation.id`. The schema is automatically registered in the Schema Registry with the topic, so Kafka clients can consume and deserialize trace data correctly. Redpanda manages both the `redpanda.otel_traces` topic and its schema automatically. If you delete either the topic or the schema, they are recreated automatically. However, deleting the topic permanently deletes all trace data, and the topic comes back empty. Do not produce your own data to this topic. It is reserved for OpenTelemetry traces. ### [](#topic-configuration-and-lifecycle)Topic configuration and lifecycle The `redpanda.otel_traces` topic has a predefined retention policy. Configuration changes to this topic are not supported. If you modify settings, Redpanda reverts them to the default values. The topic persists even after all agents and MCP servers are deleted, allowing you to retain historical trace data for analysis. Transcripts may contain sensitive information from your tool inputs and outputs. Review the data in transcripts before sharing or exporting to external systems. Read access to transcript data is governed by the transcript permissions rather than by topic ACLs on `redpanda.otel_traces`. Among the built-in roles, only Admin holds them. See [Transcript permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#transcript-permissions). ## [](#how-redpanda-stores-audit-events)How Redpanda stores audit events When the Audit Log preview is enabled, Agentic Data Plane structures audit events according to the [Open Cybersecurity Schema Framework (OCSF)](https://ocsf.io/), an open, vendor-neutral schema for security events. OCSF gives events from different subsystems a shared vocabulary. Agentic Data Plane uses the OCSF API Activity event class for authorization decisions on API operations and the Entity Management event class for configuration changes. This keeps fields such as actor, resource, activity, and outcome consistent across the audit log. OCSF standardizes what an event means; it does not define how events are stored or exported. In Agentic Data Plane, use the Audit Log view to review these records. Redpanda keeps recent events in streaming storage and archives them in an Apache Iceberg-backed historical store in object storage. The Audit Log view presents both as one time-ordered history, so you can investigate new and older activity from the same view. Captured request, response, or configuration data may contain sensitive information. Agentic Data Plane redacts supported secret and sensitive configuration fields. Identities and other audit attributes may remain visible, and captured data may be truncated or omitted. Review an event before sharing it outside your organization. The Audit Log view returns events for your Agentic Data Plane environment. Access requires an audit log permission. See [Audit log permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#audit-log-permissions). ## [](#transcripts-compared-to-audit-logs)Transcripts compared to audit logs Transcripts and audit logs serve different but complementary purposes. Transcripts provide: - A complete, immutable record of every execution step, stored on Redpanda’s distributed log with no gaps - Hierarchical view of request flow through your system (parent-child span relationships) - Detailed timing information for performance analysis - Ability to reconstruct execution paths and identify bottlenecks Transcripts are optimized for execution-level observability and governance. The audit log provides: - The user accountable for an action, plus the agent or caller that acted for the user when available - The affected resource, operation, subsystem, and time - The allowed, partial, or denied outcome and the deciding policy when available For access and accountability questions, use [the audit log](https://docs.redpanda.com/agentic-data-plane/monitor/audit-log/). For questions about an agent’s model turns, tool calls, timing, or errors, use transcripts. An investigation may use both views. ## [](#history-reconstruction)Reconstructed transcript history Trace data on `redpanda.otel_traces` is subject to a retention policy. When a transcript covers a long-running conversation whose earliest spans have already been evicted, Redpanda reconstructs the missing turns from the LLM message context carried on later spans (`gen_ai.input.messages`) and sets the boolean field `is_reconstructed` to `true` on each affected turn. The UI surfaces this as a **Reconstructed** badge on those turns: `is_reconstructed` is the backing data field; "Reconstructed" is the visible label. Reconstructed turns preserve the high-level intent and role ordering of the conversation, but do not preserve byte-level fidelity: token counts, per-turn latency, and tool-call arguments are unavailable for the reconstructed range. ## [](#next-steps)Next steps - [See What Your Agent Did](https://docs.redpanda.com/agentic-data-plane/monitor/transcripts/) - [Monitor Agent Activity](https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents/) - [Visualize Your Agent Network](https://docs.redpanda.com/agentic-data-plane/monitor/agent-network/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) --- # Page 77: Monitor Agent Activity **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents.md --- # Monitor Agent Activity > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Monitor Agent Activity latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: monitor-agents page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: monitor-agents.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/monitor-agents.adoc description: Monitor agent execution, inspect available MCP tools, analyze conversation history, track token usage, and debug issues using Playground and Transcripts. page-topic-type: how-to personas: agent_builder, platform_engineer, domain_evaluator learning-objective-1: Verify agent behavior and review its MCP capabilities using the Playground tab learning-objective-2: Track token usage and performance metrics learning-objective-3: Debug agent execution using transcripts page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-10" --- Use monitoring to track agent performance, analyze conversation patterns, debug execution issues, and optimize token costs. After reading this page, you will be able to: - Verify agent behavior and review its MCP capabilities using the **Playground** tab - Track token usage and performance metrics - Debug agent execution using transcripts For conceptual background on traces and observability, see [How Observability Works](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/). ## [](#prerequisites)Prerequisites You must have a running agent. If you do not have one, see [Agentic Data Plane Quickstart for Agent Builders](https://docs.redpanda.com/agentic-data-plane/get-started/agent-builder-quickstart/). ## [](#debug-agent-execution-with-transcripts)Debug agent execution with transcripts An agent’s **Transcripts** tab shows each conversation with timing, errors, and token usage. Use it to debug issues, verify agent behavior, and monitor performance. ### [](#navigate-an-agents-transcripts)Navigate an agent’s transcripts 1. Open **Agents** in the sidebar and select your agent. 2. Open the **Transcripts** tab. The tab lists the agent’s recent conversations, one row per conversation: - `Conversation`: The conversation ID, with the conversation title when one exists. - `Started`: When the conversation began. - `Duration`: End-to-end wall-clock time. - `Turns`: Number of turns in the conversation. - `Status`: `Completed`, `Error`, or `Running`. - `Tokens`: Total tokens across the conversation. Use the search box to match a conversation ID or title, and the status dropdown to narrow the list to `Completed`, `Error`, or `Running` conversations. The list loads in pages; click **Load more** to fetch older conversations. #### [](#conversation-detail)Conversation detail Click a row to open the conversation. The header shows the conversation ID with a status badge, the start time, duration, and turn count, plus a total-tokens chip. Toggle between two views: - **Chat**: The user-visible exchange, as a conversation. - **Detailed**: Adds per-turn metadata: latency, input/output token splits, and each tool call with its arguments, result, latency, and status. If any turns were rebuilt from LLM message context after their original spans were evicted, those turns carry a _reconstructed_ marker. For the mechanics, see [Reconstructed transcript history](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#history-reconstruction). ### [](#check-agent-health)Check agent health Use the **Transcripts** tab to verify your agent is healthy. Recent conversations should show `Completed` status, duration within your expected range, and stable token usage without unexpected growth. Several warning signs indicate problems. Conversations with `Error` status need investigation. When duration increases over time, your context window may be growing or tool calls could be slowing down. Many LLM calls for simple requests often signal that the agent is stuck in loops or making unnecessary iterations. If you see no new transcripts, the agent may be stopped or encountering deployment issues. Pay attention to patterns across multiple conversations. When all recent transcripts show errors, start by checking agent status, MCP server connectivity, and system prompt configuration. A list that alternates between success and error typically points to intermittent tool failures or external API issues. If duration increases steadily over a session, your context window is likely filling up. Clear the conversation history to reset it. High token usage combined with relatively few LLM calls usually means tool results are large or your system prompts are verbose. ### [](#debug-with-transcripts)Debug with transcripts Use the **Transcripts** tab to diagnose specific issues: If the agent is not responding: 1. Check the list for recent conversations. If none appear, the agent may be stopped. 2. Verify agent status in the main **Agents** view. 3. Look for error transcripts with deployment or initialization failures. If the agent fails during execution: 1. Set the status dropdown to `Error` and open the failed conversation. 2. Switch to the **Detailed** view and find the turn carrying the error. 3. Check the tool call’s arguments and result for error messages. 4. Cross-reference with MCP server status. If performance is slow: 1. Compare the `Duration` column across recent conversations. 2. Open a slow conversation in the **Detailed** view and scan per-turn latency to find the bottleneck. 3. Check if LLM calls are taking longer than expected. 4. Verify tool execution time on the nested tool calls. ### [](#track-token-usage-and-costs)Track token usage and costs View token consumption in the conversation detail view. The **Detailed** view breaks each turn into input tokens (everything sent to the LLM including system prompt, conversation history, and tool results) and output tokens (what the LLM generates in agent responses); the header chip shows the conversation total. Calculate cost per request: Cost = (input\_tokens x input\_price) + (output\_tokens x output\_price) Example: GPT-5.2 with 4,302 input tokens and 1,340 output tokens at $0.00000175 per input token and $0.000014 per output token costs $0.026 per request. For cost optimization strategies, see [Cost calculation](https://docs.redpanda.com/agentic-data-plane/connect/concepts/#cost-calculation). ## [](#test-agent-behavior-with-playground)Test agent behavior with Playground The **Playground** tab provides real-time conversation testing. Use it to test agent responses interactively and verify behavior before deploying changes. ![Playground test loop. A user sends a prompt, the agent streams responses, artifacts, and MCP tool calls, and Playground shows context and token usage. The user can retry or clear context, while completed runs remain available in transcripts and activity views.](https://docs.redpanda.com/agentic-data-plane/monitor/_images/agent-inspector-test-loop.svg) Figure 1. Playground closes the loop between a test prompt and production evidence ### [](#access-playground)Access Playground 1. Open **Agents** in the sidebar. 2. Click your agent name. 3. Open the **Playground** tab. 4. Enter test queries and review responses. 5. Check the conversation panel to see tool calls. 6. Start a new session to test fresh conversations or click **Clear context** to reset history. ### [](#review-mcp-tools-and-subagents)Review MCP tools and subagents For a managed agent, the Playground composer has a **Tools** control, and a **Subagents** control when the agent delegates to subagents. Use them to review the capabilities available to the agent before you send a test message. To review the agent’s direct MCP servers and their tools: ![Playground tool discovery. The Tools control displays direct MCP servers, and the Subagents control displays the agent’s subagents. Direct servers expose their tools to the agent. Each subagent exposes only the tools from the MCP servers configured for that subagent. Authentication and availability affect which tools are visible.](https://docs.redpanda.com/agentic-data-plane/monitor/_images/inspector-mcp-tool-discovery.svg) Figure 2. Playground shows tool access through direct MCP servers and subagents 1. Open the agent’s **Playground** tab. 2. Click **Tools** in the composer. 3. Under Direct MCP servers, expand a server to see its type, description, and exposed tools. Each tool entry shows its name and description when the server provides one. The list reflects the agent’s current configuration and the tools each direct MCP server exposes. It does not guarantee that a later tool call succeeds. Server availability, access policy, or an external connection can still affect a call at runtime. Some remote MCP servers require you to connect your own account before their tools are shown. Expand the server and click **Connect**, complete the authorization in the new browser tab, then return to Playground. The list refreshes when you return. If it does not, click **Refresh tools**. Use the status shown under a server to resolve common issues: | Status | What to do | | --- | --- | | Authentication required | Click Connect and complete the authorization flow. Protected tools remain hidden until the connection grants the required access. | | Unavailable | Ask an administrator to confirm that the MCP server exists and is enabled for the agent. | | Couldn’t load tools | Click Try again. If the error continues, check the MCP server’s availability and authentication settings. | | No tools exposed by this MCP server | Confirm that the server publishes tools and that the agent is connected to the intended server. | When the agent delegates to subagents, the composer also shows a **Subagents** control next to **Tools**. Click it to review the delegation targets this agent can orchestrate. Each entry lists the subagent’s name, its description, and the MCP servers configured on it. ### [](#context-window-usage)Context-window usage As you test, the composer tracks how much of the model’s context window the conversation consumes. When the agent uses a model from Redpanda’s catalog, a context-usage indicator appears in the composer after the first response and shows the share of the context window in use. Click it to open a breakdown of the window size, the input, output, reasoning, and cached tokens for the session, the **Session total**, and the estimated **Total cost**. The indicator counts input tokens plus cached tokens, so it reflects the full context occupancy under prompt caching, not fresh input alone. As a conversation grows and the indicator approaches the window size, start a new session or click **Clear context** to reset the history before responses slow down or the agent starts dropping earlier context. For models that aren’t in the catalog, such as OpenAI-compatible providers or hand-entered model IDs, the composer shows a plain input and output token count instead, because the context-window size isn’t known. ### [](#long-running-tasks)Long-running tasks The Playground live view streams a test run as it happens. If a task is still running after about five minutes, Playground stops the live view and shows a **Still running** notice: the task keeps running in the background, so you don’t need to hold the Playground open. Follow it to completion on the **Activity** tab of the **Cost & Usage** page under **Governance**. ### [](#testing-best-practices)Testing best practices Test your agents systematically by exploring edge cases and potential failure scenarios. Begin with boundary testing. Requests at the edge of agent capabilities verify that scope enforcement works correctly. Error handling becomes clear when you request unavailable data and observe whether the agent degrades gracefully. Even with proper system prompt constraints, testing confirms that your agent responds appropriately to edge cases. Monitor iteration counts during complex requests to ensure they complete within your configured limits. Ambiguous or vague queries reveal whether the agent asks clarifying questions or makes risky assumptions. Throughout testing, track token usage per request to estimate costs and identify which query patterns consume the most resources. ## [](#next-steps)Next steps - [How Observability Works](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/) - [Visualize Your Agent Network](https://docs.redpanda.com/agentic-data-plane/monitor/agent-network/) - [Troubleshoot Agents](https://docs.redpanda.com/agentic-data-plane/monitor/troubleshoot-ai-agents/) --- # Page 78: See What Your Agent Did **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/transcripts.md --- # See What Your Agent Did > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: See What Your Agent Did latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: transcripts page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: transcripts.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/transcripts.adoc description: Open a transcript in Agentic Data Plane, read the conversation turn by turn, and investigate errors, costs, and performance. page-topic-type: how-to personas: agent_builder, domain_evaluator, platform_engineer, security_compliance_lead learning-objective-1: Open a transcript from an agent's Transcripts tab learning-objective-2: Read a transcript as a turn-by-turn conversation, including tool calls and token usage learning-objective-3: Investigate errors, slow turns, and cost anomalies from the transcript detail view page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-03" --- Use an agent’s transcripts to read a complete record of each conversation, turn by turn. Each transcript captures the conversation between the user, the agent, any LLM calls, and any tools it invoked, along with token usage, latency, estimated cost, and any errors. For conceptual background on the underlying OpenTelemetry data model, see [How Observability Works](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/). After reading this page, you will be able to: - Open a transcript from an agent’s Transcripts tab - Read a transcript as a turn-by-turn conversation, including tool calls and token usage - Investigate errors, slow turns, and cost anomalies from the transcript detail view ## [](#prerequisites)Prerequisites - A [running agent](https://docs.redpanda.com/agentic-data-plane/connect/create-agent/) with at least one execution. - Access to read transcripts, from an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) granting the `Transcript` actions, or from the Admin role. No other built-in role grants transcript reads. ## [](#open-an-agents-transcripts)Open an agent’s transcripts 1. Open **Agents** in the sidebar and select the agent. 2. Open the **Transcripts** tab. The tab lists the agent’s recent conversations, one row per conversation. ## [](#read-the-transcripts-list)Read the transcripts list Each row represents one conversation. Columns include: - `Conversation`: The conversation ID, with the conversation title when one exists. - `Started`: When the conversation began. - `Duration`: End-to-end wall-clock time. - `Turns`: Number of turns in the conversation. - `Status`: `Completed`, `Error`, or `Running`. - `Tokens`: Total tokens across the conversation. A transcript marked _reconstructed_ is one in which some turns were rebuilt from LLM message context after the original spans were evicted from `redpanda.otel_traces`. See [Reconstructed transcript history](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#history-reconstruction) for what that means. ### [](#filter-the-list)Filter the list - Use the search box to match against a conversation ID or title. - Use the status dropdown to narrow the list to `Completed`, `Error`, or `Running` conversations. The list loads in pages. Click **Load more** to fetch older conversations, and use the refresh control to pick up new ones. ## [](#open-a-transcript)Open a transcript Click any row to open the conversation detail view. The view has two parts: a summary header and the conversation. ### [](#summary-header)Summary header The summary header reports: - The conversation ID and a status badge (`Completed`, `Error`, or `Running`). - The start time, end-to-end duration, and turn count. - A total-tokens chip. - A **Chat** / **Detailed** view toggle. Chat shows the user-visible exchange; Detailed adds per-turn metadata such as latency, token splits, and tool calls. If the conversation ended with a top-level error, an error banner appears below the header with the code and message. ### [](#read-the-conversation)Read the conversation Turns are listed in order by role: - **SYSTEM**: The system prompt and any priming instructions. - **USER**: A user message that started or continued the conversation. - **ASSISTANT**: A response from the LLM. In the Detailed view, shows input/output token counts and latency. If the assistant turn called a tool, its tool calls are nested underneath. - **TOOL**: A tool invocation. Shows the tool name, the arguments passed, the result, and the latency of the call. Any turn may carry the `is_reconstructed` marker. Reconstructed turns preserve role order and the high-level content of the conversation but do not carry per-turn token counts, latency, or tool-call arguments. See [Reconstructed transcript history](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#history-reconstruction) for the mechanics. ### [](#errors)Errors An errored transcript shows `TranscriptStatus.ERROR` in the summary header. The specific failure appears on the turn that raised it, with: - **Code**: `TranscriptError.code` (for example, a provider error code or `INVALID_ARGUMENT`) - **Message**: A short description from the LLM provider, the tool, or the agent runtime If the failure happened during a tool call, the error is attached to the TOOL turn; if it was an LLM call, it’s on the ASSISTANT turn; if neither, it’s on the trace root. ## [](#common-investigation-tasks)Common investigation tasks ### [](#debug-errors)Debug errors 1. Set the status dropdown to `Error`. 2. Open the failing transcript and scroll to the turn carrying the error code. 3. Read the immediately preceding turns (tool arguments, assistant output) for root cause. ### [](#investigate-performance-issues)Investigate performance issues 1. Sort the list visually by the `Duration` column to spot slow conversations. 2. Open a slow transcript, switch to the **Detailed** view, and scan the per-turn latency to find the bottleneck turn. 3. For tool-bound bottlenecks, expand the tool call to see arguments and result size: a large result often correlates with slow tool execution. ### [](#analyze-tool-usage)Analyze tool usage Open a transcript in the **Detailed** view to read each tool call: arguments in, results out, latency, and a status pill. Tool calls are nested under the assistant turn that made them. ### [](#monitor-llm-interactions-and-cost)Monitor LLM interactions and cost The transcript shows token usage per turn. For spend analysis across agents, models, and users, open **Cost & Usage** under **Governance** in the sidebar (see [Analyze Cost and Usage](https://docs.redpanda.com/agentic-data-plane/control/cost-usage/)), or use breakdown queries through `SpendingService`; per-transcript `estimated_cost_usd` is also available through the Transcripts API. ## [](#limitations)Limitations - The list loads in pages from most recent backward. If the conversation you need is old, keep loading more pages or search by its conversation ID. - Reconstructed turns do not carry token counts, latency, or tool-call arguments for the reconstructed range. For byte-level fidelity, lower the ingestion lag or extend `redpanda.otel_traces` retention (see [How Redpanda stores trace data](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#opentelemetry-traces-topic)). - Estimated USD cost is only populated for models covered by the pricing table. ## [](#troubleshooting)Troubleshooting ### [](#transcript-stuck-in-running)Transcript stuck in RUNNING A transcript stays in `RUNNING` until the root span closes. Common causes: - The agent or MCP server is still executing (this is normal: Wait, or open a newer transcript). - The root span never flushed because the process was killed mid-execution. Expect this to resolve after the OTLP ingestion lag clears; if it doesn’t after several minutes, the trace is likely orphaned. ### [](#usd-cost-shows-0)USD cost shows 0 `TranscriptUsage.estimated_cost_usd` is populated by the cost-reporting pipeline from the `gen_ai.usage.*` attributes on each LLM-call span combined with a per-model pricing table. For the full list of cost-bearing attributes (including the explicit USD-cost fields), see [Key attributes by layer](https://docs.redpanda.com/agentic-data-plane/monitor/concepts/#key-attributes-by-layer). If cost is `0` for a transcript that clearly used tokens, check: - The model is in the pricing table. To use a custom rate (negotiated contract, internal chargeback), see [Override per-model pricing](https://docs.redpanda.com/agentic-data-plane/control/budgets/#override-per-model-pricing). - The cost-reporting pipeline is enabled on your environment. - The LLM-call spans carry the `gen_ai.usage.*` attributes the pipeline reads: Either the token-count inputs (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) or the explicit USD-cost fields listed on the concepts page. > 📝 **NOTE** > > Cache-write tokens (Anthropic 4.x, OpenAI 4.x prompt caches) are attributed to the **CACHED** bucket on streaming and non-streaming responses alike. If a transcript shows zero cache cost on a request that clearly populated a prompt cache, check that the upstream actually wrote to the cache (the response includes a non-zero `cache_creation_input_tokens` or equivalent token-count field) before treating it as a discrepancy. ### [](#all-turns-marked-reconstructed)All turns marked reconstructed Reconstruction happens when the original spans have been evicted from `redpanda.otel_traces`. Causes: - Retention on `redpanda.otel_traces` is aggressive relative to how long the conversation has been running. - OTLP ingestion fell behind and the span was dropped before it reached the topic. For long-running conversations, accept some reconstruction; for short conversations whose turns are all reconstructed, investigate ingestion and retention. ### [](#transcript-missing-entirely)Transcript missing entirely - Confirm the agent or MCP server actually ran: Check its logs and the corresponding session or task topic. - Confirm your user can read transcripts. Outside the Admin role, transcript reads come from an [access policy](https://docs.redpanda.com/agentic-data-plane/control/access-policies/) naming the `Transcript` actions. Check the Access page for a policy that covers your user or one of its groups, and for a `forbid` that overrides it. See [Transcript permissions](https://docs.redpanda.com/agentic-data-plane/control/permissions-reference/#transcript-permissions). - Confirm the feature flag enabling transcripts is on for your environment. ## [](#next-steps)Next steps - [Monitor Agent Activity](https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents/) - [Test an MCP Server’s Tools with the Inspector](https://docs.redpanda.com/agentic-data-plane/connect/test-tools/) - [Troubleshoot Agents](https://docs.redpanda.com/agentic-data-plane/monitor/troubleshoot-ai-agents/) --- # Page 79: Troubleshoot Agents **URL**: https://docs.redpanda.com/agentic-data-plane/monitor/troubleshoot-ai-agents.md --- # Troubleshoot Agents > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Troubleshoot Agents latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: troubleshoot-ai-agents page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: troubleshoot-ai-agents.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/monitor/pages/troubleshoot-ai-agents.adoc description: Diagnose and fix common issues with AI agents including deployment failures, runtime behavior problems, and tool execution errors. page-topic-type: troubleshooting personas: agent_builder, platform_engineer learning-objective-1: Diagnose deployment failures and resource allocation errors learning-objective-2: Resolve runtime behavior issues including tool selection and iteration limits learning-objective-3: Fix tool execution problems and authentication failures page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-07" --- Use this page to diagnose and fix common issues with AI agents, including deployment failures, runtime behavior problems, tool execution errors, and integration issues. ## [](#deployment-issues)Deployment issues Fix issues that prevent agents from connecting to required resources. ### [](#mcp-server-connection-failures)MCP server connection failures **Symptoms:** Agent starts but the tools don’t respond or return connection errors. **Causes:** - MCP server stopped or crashed after agent creation - Network connectivity issues between agent and MCP server - MCP server authentication or permission issues **Solution:** 1. Verify MCP server status in **Remote MCP**. 2. Check MCP server logs for errors. 3. Restart the MCP server if needed. 4. Verify agent has permission to access the MCP server. **Prevention:** - Monitor MCP server health - Use appropriate retry logic in tools ## [](#runtime-behavior-issues)Runtime behavior issues Resolve problems with agent decision-making, tool selection, and response generation. ### [](#agent-not-calling-tools)Agent not calling tools **Symptoms:** Agent responds without calling any tools, or fabricates information instead of using tools. **Causes:** - System prompt doesn’t clearly specify when to use tools - Tool descriptions are vague or missing - LLM model lacks sufficient reasoning capability - Max iterations is too low **Solution:** 1. Strengthen tool usage guidance in your system prompt: ```text ALWAYS use get_order_status when customer mentions an order ID. NEVER respond about order status without calling the tool first. ``` 2. Review tool descriptions in your MCP server configuration. 3. Use a more capable model from the supported list for your gateway. 4. Increase max iterations if the agent is stopping before reaching tools. **Prevention:** - Write explicit tool selection criteria in system prompts - Test agents with the [systematic testing approach](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/#evaluation-and-testing) - Use models appropriate for your task complexity ### [](#calling-wrong-tools)Calling wrong tools **Symptoms:** Agent selects incorrect tools for the task, or calls tools with invalid parameters. **Causes:** - Tool descriptions are ambiguous or overlap - Too many similar tools confuse the LLM - System prompt doesn’t provide clear tool selection guidance **Solution:** 1. Make tool descriptions more specific and distinct. 2. Add "when to use" guidance to your system prompt: ```text Use get_order_status when: - Customer provides an order ID (ORD-XXXXX) - You need to check current order state Use get_shipping_info when: - Order status is "shipped" - Customer asks about delivery or tracking ``` 3. Reduce the number of tools you expose to the agent. 4. Use subagents to partition tools by domain. **Prevention:** - Follow tool design patterns in [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - Limit each agent to 10-15 tools maximum - Test boundary cases where multiple tools might apply ### [](#stuck-in-loops-or-exceeding-max-iterations)Stuck in loops or exceeding max iterations **Symptoms:** Agent reaches max iterations without completing the task, or repeatedly calls the same tool with the same parameters. **Causes:** - Tool returns errors that the agent doesn’t know how to handle - Agent doesn’t recognize when the task is complete - Tool returns incomplete data that prompts another call - System prompt encourages exhaustive exploration **Solution:** 1. Add completion criteria to your system prompt: ```text When you have retrieved all requested information: 1. Present the results to the user 2. Stop calling additional tools 3. Do not explore related data unless asked ``` 2. Add error handling guidance: ```text If a tool fails after 2 attempts: - Explain what went wrong - Do not retry the same tool again - Move on or ask for user guidance ``` 3. Review tool output to ensure it signals completion clearly. 4. Increase max iterations if the task legitimately requires many steps. **Prevention:** - Design tools to return complete information in one call - Set max iterations appropriate for task complexity (see [Why iterations matter](https://docs.redpanda.com/agentic-data-plane/connect/concepts/#why-iterations-matter)) - Test with ambiguous requests that might cause loops ### [](#making-up-information)Making up information **Symptoms:** Agent provides plausible-sounding answers without calling tools, or invents data when tools fail. **Causes:** - System prompt doesn’t explicitly forbid fabrication - Agent treats tool failures as suggestions rather than requirements - Model is hallucinating due to lack of constraints **Solution:** 1. Add explicit constraints to your system prompt: ```text Critical rules: - NEVER make up order numbers, tracking numbers, or customer data - If a tool fails, explain the failure - do not guess - If you don't have information, say so explicitly ``` 2. Test error scenarios by temporarily disabling tools. 3. Use a more capable model that follows instructions better. **Prevention:** - Include "never fabricate" rules in all system prompts - Test with requests that require unavailable data - Monitor transcripts and session topic for fabricated responses ### [](#analyzing-conversation-patterns)Analyzing conversation patterns **Symptoms:** Agent behavior is inconsistent or produces unexpected results. **Solution:** Review conversation history in transcripts to identify problematic patterns: - Agents calling the same tool repeatedly: Indicates loop detection is needed - Large gaps between messages: Suggests tool timeout or slow execution - Agent responses without tool calls: Indicates a tool selection issue - Fabricated information: Suggests a missing "never make up data" constraint - Truncated early messages: Indicates the context window was exceeded **Analysis workflow:** 1. Use **Playground** to reproduce the issue. 2. Review full conversation including tool invocations. 3. Identify where agent behavior diverged from expected. 4. Check system prompt for missing guidance. 5. Verify tool responses are formatted correctly. ## [](#performance-issues)Performance issues Diagnose and fix issues related to agent speed and resource consumption. ### [](#slow-response-times)Slow response times **Symptoms:** Agent takes 10+ seconds to respond to simple queries. **Causes:** - LLM model is slow (large context processing) - Too many tool calls in sequence - Tools themselves are slow (database queries, API calls) - Large context window from long conversation history **Solution:** 1. Use a faster, lower-latency model tier for simple queries and reserve larger models for complex reasoning. 2. Review conversation history in the **Playground** tab to identify unnecessary tool calls. 3. Optimize tool implementations: 1. Add caching where appropriate 2. Reduce query complexity 3. Return only needed data (use pagination, filters) 4. Clear the conversation history if the context is very large. **Prevention:** - Right-size model selection based on task complexity - Design tools to execute quickly (< 2 seconds ideal) - Set appropriate max iterations to prevent excessive exploration - Monitor token usage and conversation length ### [](#high-token-costs)High token costs **Symptoms:** Token usage is higher than expected, costs are increasing rapidly. **Causes:** - Max iterations configured too high - Agent making unnecessary tool calls - Large tool results filling context window - Long conversation history not being managed - Using expensive models for simple tasks **Solution:** 1. Review token usage in transcripts. 2. Lower max iterations for this agent. 3. Optimize tool responses to return less data: ```text Bad: Return all 10,000 customer records Good: Return paginated results, 20 records at a time ``` 4. Add cost control guidance to system prompt: ```text Efficiency guidelines: - Request only the data you need - Stop when you have enough information - Do not call tools speculatively ``` 5. Switch to a more cost-effective model for simple queries. 6. Clear conversation history periodically in the **Playground** tab. **Prevention:** - Set appropriate max iterations (10-20 for simple, 30-40 for complex) - Design tools to return minimal necessary data - Monitor token usage trends - See cost calculation guidance in [Cost calculation](https://docs.redpanda.com/agentic-data-plane/connect/concepts/#cost-calculation) ## [](#tool-execution-issues)Tool execution issues Fix problems with timeouts, invalid parameters, and error responses. ### [](#tool-timeouts)Tool timeouts **Symptoms:** Tools fail with timeout errors, agent receives incomplete results. **Causes:** - External API is slow or unresponsive - Database query is too complex - Network latency between tool and external system - Tool processing large datasets in memory **Solution:** 1. Add timeout handling to tool implementation: ```yaml http: url: https://api.example.com/data timeout: "5s" # Set explicit timeout ``` 2. Optimize external queries: 1. Add database indexes 2. Reduce query scope 3. Cache frequent queries 3. Increase tool timeout if operation legitimately takes longer. 4. Add retry logic for transient failures. **Prevention:** - Set explicit timeouts in all tool configurations - Test tools under load - Monitor external API performance - Design tools to fail fast on unavailable services ### [](#invalid-parameters)Invalid parameters **Symptoms:** Tools return validation errors about missing or incorrectly formatted parameters. **Causes:** - Tool schema doesn’t match implementation - Agent passes wrong data types - Required parameters not marked as required in schema - Agent misunderstands parameter purpose **Solution:** 1. Verify tool schema matches implementation: ```yaml input_schema: properties: order_id: type: string # Must match what tool expects description: "Order ID in format ORD-12345" ``` 2. Add parameter validation to tools. 3. Improve parameter descriptions in tool schema. 4. Add examples to tool descriptions: ```yaml description: | Get order status by order ID. Example: get_order_status(order_id="ORD-12345") ``` **Prevention:** - Write detailed parameter descriptions - Include format requirements and examples - Test tools with invalid inputs to verify error messages - Use JSON Schema validation in tool implementations ### [](#tool-returns-errors)Tool returns errors **Symptoms:** Tools execute but return error responses or unexpected data formats. **Causes:** - External API returned error - Tool implementation has bugs - Data format changed in external system - Tool lacks error handling **Solution:** 1. Check tool logs in MCP server. 2. Test tool directly (outside agent context). 3. Verify external system is operational. 4. Add error handling to tool implementation: ```yaml processors: - try: - http: url: ${API_URL} catch: - mapping: | root.error = "API unavailable: " + error() ``` 5. Update agent system prompt to handle this error type. **Prevention:** - Implement comprehensive error handling in tools - Monitor external system health - Add retries for transient failures - Log all tool errors for analysis ## [](#integration-issues)Integration issues Fix problems with external applications calling agents and pipeline-to-agent integration. ### [](#agent-card-does-not-contain-a-url)Agent card does not contain a URL **Symptoms:** Pipeline fails with error: `agent card does not contain a URL` or `failed to init processor path root.pipeline.processors.0` **Causes:** - The `agent_card_url` points to the base agent endpoint instead of the agent card JSON file **Solution:** The `agent_card_url` must point to the agent card JSON file, not the base agent endpoint. **Incorrect configuration:** ```yaml processors: - a2a_message: agent_card_url: "https://your-agent-id.ai-agents.your-cluster-id.clusters.rdpa.co" prompt: "Analyze this transaction: ${!content()}" ``` **Correct configuration:** ```yaml processors: - a2a_message: agent_card_url: "https://your-agent-id.ai-agents.your-cluster-id.clusters.rdpa.co/.well-known/agent-card.json" prompt: "Analyze this transaction: ${!content()}" ``` The agent card is always available at `/.well-known/agent-card.json` according to the A2A protocol standard. **Prevention:** - Always append `/.well-known/agent-card.json` to the agent endpoint URL - Test the agent card URL in a browser before using it in pipeline configuration - See [Agent card location](https://docs.redpanda.com/agentic-data-plane/connect/a2a-concepts/#agent-card-location) for details ### [](#pipeline-integration-failures)Pipeline integration failures **Symptoms:** Pipelines using `a2a_message` processor fail or timeout. **Causes:** - Agent is not running or restarting - Agent timeout is too low for pipeline workload - Authentication issues between pipeline and agent - High event volume overwhelming agent **Solution:** 1. Check agent status and resource allocation. 2. Increase agent resource tier for high-volume pipelines. 3. Add error handling in pipeline: ```yaml processors: - try: - a2a_message: agent_card_url: "https://your-agent-url/.well-known/agent-card.json" catch: - log: message: "Agent invocation failed: ${! error() }" ``` **Prevention:** - Test pipeline-agent integration with low volume first - Size agent resources appropriately for event rate - See integration patterns in [Integrate with Redpanda Pipelines](https://docs.redpanda.com/agentic-data-plane/connect/pipeline-integration-patterns/) ## [](#monitor-and-debug-agents)Monitor and debug agents For comprehensive guidance on monitoring agent activity, analyzing conversation history, tracking token usage, and debugging issues, see [Monitor Agent Activity](https://docs.redpanda.com/agentic-data-plane/monitor/monitor-agents/). ## [](#next-steps)Next steps - [Write Effective System Prompts](https://docs.redpanda.com/agentic-data-plane/connect/system-prompts/) - [How MCP Servers Work](https://docs.redpanda.com/agentic-data-plane/connect/mcp-overview/) - [Choose an Agent Architecture](https://docs.redpanda.com/agentic-data-plane/connect/architecture-patterns/) --- # Page 80: Reference **URL**: https://docs.redpanda.com/agentic-data-plane/reference.md --- # Reference > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Reference latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/index.adoc description: Field and command reference for Agentic Data Plane. page-git-created-date: "2026-04-29" page-git-modified-date: "2026-06-01" --- Look up the details. This section covers the `rpk` command-line reference and the Agentic Data Plane glossary. - [Install or Update rpk](rpk-install/) Install or update rpk to interact with Redpanda from the command line. - [rpk Command Reference](rpk/) Reference for the rpk commands used to manage the Redpanda Agentic Data Plane from the command line. - [Glossary](glossary/) Definitions of key terms used in Redpanda Agentic Data Plane documentation. - [Release Notes](release-notes/) What's new in the Agentic Data Plane: new features, improvements, and bug fixes, by release. --- # Page 81: Glossary **URL**: https://docs.redpanda.com/agentic-data-plane/reference/glossary.md --- # Glossary > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Glossary latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: glossary page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: glossary.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/glossary.adoc description: Definitions of key terms used in Redpanda Agentic Data Plane documentation. page-git-created-date: "2026-04-23" page-git-modified-date: "2026-06-29" --- ## [](#agentic-data-plane)Agentic Data Plane ### [](#agent2agent-a2a-protocol)Agent2Agent (A2A) protocol Communication protocol that enables AI agents to discover, coordinate with, and delegate tasks to other agents in a distributed system. The A2A protocol allows agents to work together by sharing capabilities, coordinating workflows, and distributing complex tasks across multiple specialized agents. It provides standardized messaging, capability discovery, and task delegation mechanisms for multi-agent systems. ### [](#agentic-data-plane-2)Agentic Data Plane Infrastructure layer that enables AI agents to discover, connect to, and interact with data sources and tools through standardized protocols. The Agentic Data Plane provides the underlying infrastructure for AI agents to access streaming data, invoke tools, and coordinate operations across distributed systems using protocols like MCP and A2A. ### [](#ai-agent)AI agent A system built around a large language model (LLM) that interprets user intent, selects the right tools, and chains multiple steps into a workflow. AI agents can understand natural language instructions, reason about tasks, invoke tools through MCP servers, and coordinate multiple operations to accomplish complex workflows. ### [](#ai-gateway)AI Gateway A unified entry point for AI traffic that provides LLM routing, MCP tool aggregation, cost controls, and observability across multiple LLM providers. ### [](#ai-token)AI token A credential used specifically for authenticating AI agents and authorizing their access to resources in agentic systems. AI tokens are specialized authentication credentials for AI agents, distinct from bearer tokens used in traditional API authentication. They enable agents to authenticate with MCP servers and access data plane resources while maintaining audit trails of agent operations. ### [](#context-window)context window The maximum amount of text (measured in tokens) that an LLM can process in a single request. The context window determines how much information an agent can consider at once, including the system prompt, conversation history, tool outputs, and retrieved documents. Larger context windows enable more sophisticated reasoning but may increase latency and cost. Common sizes range from 8K to 200K+ tokens. ### [](#declarative-agent)declarative agent An AI agent that runs in Redpanda’s managed runtime, where you configure its behavior (LLM, system prompt, and attached tools) instead of writing agent code. Contrast with a self-managed agent, which you build and run in your own runtime. ### [](#frontier-model)frontier model The most advanced and capable AI models available, representing the current state-of-the-art in language understanding and reasoning. Frontier models are cutting-edge large language models with exceptional reasoning, planning, and problem-solving capabilities. Examples include GPT-4, Claude 3, and Gemini Ultra. These models are commonly used to power sophisticated AI agents that require advanced decision-making and tool orchestration. ### [](#large-language-model-llm)large language model (LLM) An AI model trained on vast amounts of text data that can understand and generate human-like text, reason about tasks, and follow instructions. Large language models power AI agents by providing natural language understanding, reasoning capabilities, and the ability to plan and execute complex tasks. LLMs interpret user requests, decide which tools to invoke, and synthesize responses based on retrieved data. ### [](#mcp-client)MCP client An AI application or agent that connects to MCP servers to discover and invoke tools. MCP clients use the Model Context Protocol to communicate with MCP servers, discovering available tools, understanding their capabilities, and invoking them with appropriate parameters. The client handles authentication, request formatting, and response processing. ### [](#mcp-server)MCP server A service that exposes tools and resources using the Model Context Protocol, allowing AI agents to discover and invoke them. MCP servers act as bridges between AI agents and external systems, providing standardized interfaces for tool discovery, invocation, and resource access. ### [](#model-context-protocol-mcp)Model Context Protocol (MCP) A standardized protocol that enables AI agents to connect with external data sources and tools in Redpanda. MCP provides a consistent interface for AI applications to discover and interact with data sources, services, and computational tools through Redpanda infrastructure. ### [](#oauth-client)OAuth client An Agentic Data Plane resource that governs inbound authentication: an external app (Claude Desktop, ChatGPT, Copilot Studio) authenticating to AI Gateway so the app’s users can invoke MCP tools. Separate from an OAuth provider, which governs outbound authentication. ### [](#oauth-connection)OAuth connection A stored authorization that links a user to a third-party provider, created when the user completes the OAuth consent flow. OAuth connections hold access and refresh tokens in the token vault for agents to use on the user’s behalf. ### [](#oauth-provider)OAuth provider An Agentic Data Plane resource that governs outbound authentication: AI Gateway authenticating to an upstream system (GitHub, Slack, Salesforce, and so on) on a user’s behalf so MCP servers can call it. Separate from an OAuth client, which governs inbound authentication. ### [](#observability-o11y)observability (o11y) The ability to understand a system’s internal state by examining its external outputs, such as traces, metrics, and logs. In Redpanda’s agentic systems, observability enables debugging agent behavior, monitoring performance, analyzing execution flow, and identifying bottlenecks through transcripts captured in the `redpanda.otel_traces` topic. ### [](#opentelemetry)OpenTelemetry Open-source observability framework that provides standardized APIs, libraries, and tools for capturing and exporting telemetry data. OpenTelemetry provides standardized APIs for capturing traces, metrics, and logs from applications. Redpanda agents and MCP servers automatically emit OpenTelemetry traces to the `redpanda.otel_traces` topic to provide complete observability into agentic system operations. ### [](#otlp-opentelemetry-protocol)OTLP (OpenTelemetry Protocol) Standard protocol for encoding and transmitting telemetry data defined by the OpenTelemetry project. OTLP is the OpenTelemetry Protocol specification for encoding and transmitting telemetry data. Redpanda stores spans in the `redpanda.otel_traces` topic using a Protobuf schema that closely follows the OTLP specification. ### [](#prompt)prompt Natural language instructions or context provided to an LLM to guide its behavior and responses. Prompts are the primary way to communicate with LLMs and AI agents. They can include instructions, examples, context, and questions that guide the model’s reasoning and output. Effective prompt design is critical for agent performance and reliability. ### [](#resource)resource Read-only data that an MCP server exposes and agents can fetch by URI. Resources provide file contents, database rows, or API responses without performing side effects. ### [](#self-managed-agent)self-managed agent An AI agent you build and run yourself, in your own runtime and framework, registered with Redpanda Agentic Data Plane as an identity. The AI Gateway becomes the agent’s LLM and MCP endpoint, so the Agentic Data Plane attributes spend, tokens, latency, and traces back to the agent. The Agentic Data Plane does not host or run the agent. Contrast with a declarative agent, which runs in Redpanda’s managed runtime. ### [](#self-managed-mcp-server)self-managed MCP server An MCP server you host and operate yourself, registered with Redpanda Agentic Data Plane so the AI Gateway can proxy tool calls to it. Contrast with a managed MCP server, which Redpanda hosts in-process for you. ### [](#span)span A single unit of work within a trace representing one operation, such as a data processing operation or an external API call. Spans are organized in the Redpanda UI as parent-child relationships that show how operations flow through the system. Each span captures details about a specific operation, including timing, status, and metadata. ### [](#spending-event)spending event A record produced by the AI Gateway for every LLM call. Each event captures input, output, and cached token counts, the total cost in microcents, request count, and the provider, model, user, and organization context that the call ran under. ### [](#subagent)subagent A specialized AI agent that handles specific tasks or domains as part of a larger multi-agent system. Subagents are autonomous components within a multi-agent architecture that have focused expertise in particular domains or operations. They communicate with a parent agent or other subagents to accomplish complex workflows that require coordination across multiple specializations. ### [](#system-prompt)system prompt Initial instructions that define an agent’s role, capabilities, and behavioral guidelines. The system prompt is provided at the start of an agent session and establishes the agent’s identity, available tools, operating constraints, and response style. It remains active throughout the conversation and shapes all subsequent agent behavior and decision-making. ### [](#token-vault)token vault Encrypted per-user store for the OAuth access and refresh tokens used to access third-party providers. Redpanda injects a user’s token at call time and refreshes it automatically; connections persist until the user revokes them. ### [](#tool-invocation)tool invocation The process of an AI agent executing an MCP tool to perform a specific operation. Tool invocation occurs when an agent determines that it needs to use a tool, formats the request with appropriate parameters, sends it to the MCP server, and processes the response. Each invocation is captured in transcripts as spans for observability and debugging. ### [](#tool)tool A capability exposed by an MCP server that an AI agent can discover at runtime and invoke through structured JSON-RPC calls. Tools encapsulate operations such as querying a database, calling an API, or sending a message. ### [](#trace)trace The complete lifecycle of a request captured as a collection of spans, showing how operations relate to each other. A trace represents the complete lifecycle of a request (for example, a tool invocation from start to finish). A trace contains one or more spans organized hierarchically, showing how operations relate to each other. ### [](#transcript)transcript Complete observability record of agent or MCP server operations captured as OpenTelemetry traces and stored in the redpanda.otel\_traces topic. Transcripts capture tool invocations, agent reasoning steps, data processing operations, external API calls, error conditions, and performance metrics. They provide a complete record of how agentic systems operate, enabling debugging, auditing, and performance analysis. ## [](#redpanda-cloud)Redpanda Cloud ### [](#beta)beta Features in beta are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. ### [](#byoc)BYOC Bring Your Own Cloud (BYOC) is a fully-managed Redpanda Cloud deployment where clusters run in your private cloud, so all data is contained in your own environment. Redpanda handles provisioning, operations, and maintenance. ### [](#byovnet)BYOVNet A Bring Your Own Virtual Network (BYOVNet) cluster allows you to deploy the Redpanda data plane into your existing Azure VNet to fully manage the networking lifecycle. Compared to standard BYOC, BYOVNet provides more security, but the configuration is more complex. ### [](#byovpc)BYOVPC A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda data plane into your existing VPC on AWS or GCP to fully manage the networking lifecycle. Compared to standard BYOC, BYOVPC provides more security, but the configuration is more complex. ### [](#connector)connector Enables Redpanda to integrate with external systems, such as databases. ### [](#control-plane)control plane This part of Redpanda Cloud enforces rules in the data plane, including cluster management, operations, and maintenance. ### [](#data-plane)data plane This part of Redpanda Cloud contains Redpanda clusters and other components, such as Redpanda Console, Redpanda Operator, and `rpk`. It is managed by an agent that receives cluster specifications from the control plane. Sometimes used interchangeably with clusters. ### [](#data-sovereignty)data sovereignty Containing all your data in your environment. With BYOC, Redpanda handles provisioning, monitoring, and upgrades, but you manage your streaming data without Redpanda’s control plane ever seeing it. Additionally, with BYOVPC, the Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. ### [](#dedicated-cloud)Dedicated Cloud A fully-managed Redpanda Cloud deployment option where you host your data in Redpanda’s VPC, and Redpanda handles provisioning, operations, and maintenance. Dedicated clusters are single-tenant deployments that support private networking (for example, VPC peering to talk over private IPs) for better data isolation. ### [](#limited-availability)limited availability Features in limited availability (LA) are production-ready and are covered by Redpanda Support for early adopters. ### [](#pipeline)pipeline A single configuration file running in Redpanda Connect with an input connector, an output connector, and optional processors in between. A pipeline typically streams data into Redpanda from an operational source (like PostgreSQL) or streams data out of Redpanda into an analytical system (like Snowflake). ### [](#redpanda-cloud-2)Redpanda Cloud A fully-managed data streaming service deployed with Redpanda Console. It includes automated upgrades and patching, backup and recovery, data and partition balancing, and built-in connectors. Redpanda Cloud is available in Serverless, Dedicated, and Bring Your Own Cloud (BYOC) deployment options to suit different data sovereignty and infrastructure requirements. ### [](#redpanda-console)Redpanda Console The web-based UI for managing and monitoring Redpanda clusters and streaming workloads. You can also set up and manage connectors in Redpanda Console. Redpanda Console is an integral part of Redpanda Cloud, but it also can be used as a standalone program as part of a Redpanda Self-Managed deployment. ### [](#remote-mcp)Remote MCP An MCP server hosted in your Redpanda Cloud cluster. It exposes custom tools that AI assistants can call to access your data and workflows. ### [](#resource-group)resource group A container for Redpanda Cloud resources, including clusters and networks. You can rename your default resource group, and you can create more resource groups. For example, you may want different resource groups for production and testing. ### [](#serverless)Serverless Serverless is the fastest and easiest way to start data streaming. You host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. ### [](#sink-connector)sink connector Exports data from a Redpanda cluster into a target system. ### [](#source-connector)source connector Imports data from a source system into a Redpanda cluster. ## [](#redpanda-connect)Redpanda Connect ### [](#mcp-tool)MCP tool A function that an AI assistant can call to perform a specific task, such as fetching data from an API, querying a database, or processing streaming data. Each tool is defined using Redpanda Connect components and annotated with MCP metadata. ### [](#processor)processor A Redpanda Connect component that transforms data, validates inputs, or calls external APIs within a processing pipeline. Processors are stateless components in Redpanda Connect that operate on individual messages or batches. When used as MCP tools, processors handle data transformations, validate parameters, and invoke external services. Each processor executes independently per request with no state maintained between invocations. ### [](#redpanda-connect-mcp-server)Redpanda Connect MCP server A process that exposes Redpanda Connect components to MCP clients. You write each tool’s logic using Redpanda Connect configurations and annotate them with MCP metadata so clients can discover and invoke them. ### [](#redpanda-connect-2)Redpanda Connect A framework for building data streaming applications using declarative YAML configurations. Redpanda Connect provides components such as inputs, processors, outputs, and caches to define data flows and transformations. ## [](#redpanda-sql)Redpanda SQL ### [](#oxla)Oxla The SQL engine technology that powers Redpanda SQL. "Oxla" may appear in version strings, error messages, and some API field names. ### [](#redpanda-catalog)Redpanda catalog A catalog in Redpanda SQL that maps Redpanda topics to SQL tables. The default Redpanda catalog (`default_redpanda_catalog`) is provisioned automatically when Redpanda SQL is enabled. ### [](#redpanda-sql-2)Redpanda SQL A PostgreSQL-compatible SQL engine built into Redpanda Data Platform for querying streaming data in Redpanda topics. Redpanda SQL provides analytical SQL queries over Redpanda topic data without requiring ETL pipelines or a separate analytics system. It can also query the Iceberg-translated history of Iceberg-enabled topics alongside live records. ## [](#redpanda-core)Redpanda core ### [](#availability-zone-az)availability zone (AZ) One or more data centers served by high-bandwidth links with low latency, typically within a close distance of one another. ### [](#broker)broker An instance of Redpanda that stores and manages event streams. Multiple brokers join together to form a Redpanda cluster. Sometimes used interchangeably with node, but a node is typically a physical or virtual server. See also: node ### [](#client)client A producer application that writes events to Redpanda, or a consumer application that reads events from Redpanda. This could also be a client library, like librdkafka or franz-go. ### [](#cluster)cluster One or more brokers that work together to manage real-time data streaming, processing, and storage. ### [](#consumer-group)consumer group A set of consumers that cooperate to read data for better scalability. As group members arrive and leave, partitions are re-assigned so each member receives a proportional share. ### [](#consumer-offset)consumer offset The position of a consumer in a specific topic partition, to track which records they have read. A consumer offset of 3 means it has read messages 0-2 and will next read message 3. ### [](#consumer)consumer A client application that subscribes to Redpanda topics to asynchronously read events. ### [](#controller-broker)controller broker A broker that manages operational metadata for a Redpanda cluster and ensures replicas are distributed among brokers. At any given time, one active controller exists in a cluster. If the controller fails, another broker is automatically elected as the controller. ### [](#data-stream)data stream A continuous flow of events in real time that are produced and consumed by client applications. Redpanda is a data streaming platform. Also known as event stream. ### [](#event)event A record of something changing state at a specific time. Events can be generated by various sources, including sensors, applications, and devices. Producers write events to Redpanda, and consumers read events from Redpanda. ### [](#iceberg-catalog)Iceberg catalog A metadata service that tracks Apache Iceberg tables and their schemas. Redpanda supports REST catalogs (such as AWS Glue, GCP BigLake, Snowflake Open Catalog, or Databricks Unity Catalog) and filesystem-based catalogs. Redpanda SQL connects to a REST catalog with `CREATE ICEBERG CATALOG` to query the Iceberg-committed history of a topic. ### [](#iceberg-mode)Iceberg mode The Redpanda topic property (`redpanda.iceberg.mode`) that controls whether and how the topic’s records are translated into an Apache Iceberg table. ### [](#iceberg-topic)Iceberg topic A Redpanda topic with Iceberg integration enabled, so its data is also stored as an Apache Iceberg table that downstream analytical systems can read directly. Also called an Iceberg-enabled topic. ### [](#iceberg-translation)Iceberg translation The background process that converts records on an Iceberg topic into Apache Iceberg format and commits them to the configured Iceberg catalog. ### [](#kafka-api)Kafka API Producers and consumers interact with Redpanda using the Kafka API. It uses the default port 9092. ### [](#learner)learner A broker that is a follower in a Raft group but is not part of quorum. In a Raft group, a broker can be in learner status. Learners are followers that cannot vote and so do not count towards quorum (the majority). They cannot be elected to leader nor can they trigger leader elections. Brokers can be promoted or demoted between learner and voter. New Raft group members start as learners. ### [](#listener)listener Configuration on a broker that defines how it should accept client or inter-broker connections. Each listener is associated with a specific protocol, hostname, and port combination. The listener defines where the broker should listen for incoming connections. ### [](#log)log An ordered, append-only, immutable sequence of records. The log is Redpanda’s core storage abstraction for event streams. At the conceptual level, topics represent replayable logs. Physically, each partition is implemented as a log file on disk, divided into segments. Redpanda uses the Raft consensus algorithm to coordinate writing data to log files and replicate them across brokers for fault tolerance. See also: topic, partition, segment ### [](#message)message One or more records representing individual events being transmitted. Redpanda transfers messages between producers and consumers. Sometimes used interchangeably with record. ### [](#node)node A machine, which could be a server, a virtual machine (instance), or a Docker container. Every node has its own disk. Partitions are stored locally on nodes. In Kubernetes, a Node is the machine that Redpanda runs on. Outside the context of Kubernetes, this term may be used interchangeably with broker, such as `node_id`. See also: broker ### [](#offset-commit)offset commit An acknowledgement that the event has been read. ### [](#offset)offset A unique integer assigned to each record to show its location in the partition. ### [](#pandaproxy)pandaproxy Original name for the subsystem of Redpanda that allows access to your data through a REST API. This name still appears in the HTTP Proxy API and the Schema Registry API. ### [](#partition-leader)partition leader Every Redpanda partition forms a Raft group with a single elected leader. This leader handles all writes, and it replicates data to followers to ensure that a majority of brokers store the data. ### [](#partition)partition A subset of events in a topic, like a log file. It is an ordered, immutable sequence of records. Partitions allow you to distribute a stream, which lets producers write messages in parallel and consumers read messages in parallel. Partitions are made up of segment files on disk. ### [](#producer)producer A client application that writes events to Redpanda. Redpanda stores these events in sequence and organizes them into topics. ### [](#rack)rack A failure zone that has one or more Redpanda brokers assigned to it. ### [](#raft)Raft The consensus algorithm Redpanda uses to coordinate writing data to log files and replicating that data across brokers. For more details, see [https://raft.github.io/](https://raft.github.io/) ### [](#record)record A self-contained data entity with a defined structure, representing a single event. Sometimes used interchangeably with message. ### [](#replicas)replicas Copies of partitions that are distributed across different brokers, so if one broker goes down, there is a copy of the data. ### [](#retention)retention The mechanism for determining how long Redpanda stores data on local disk or in object storage before purging it. ### [](#replication-factor)replication factor The number of partition copies in a cluster. This is set to 3 in Redpanda Cloud deployments and 1 (no replication) in Self-Managed deployments. A replication factor of at least 3 ensures that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers. ### [](#schema)schema An external mechanism to describe the structure of data and its encoding. Schemas validate the structure and ensure that producers and consumers can connect with data in the same format. ### [](#seastar)Seastar An open-source thread-per-core C++ framework, which binds all work to physical cores. Redpanda is built on Seastar. For more details, see [https://seastar.io/](https://seastar.io/) ### [](#seed-server)seed server The initial set of brokers that a Redpanda broker contacts to join the cluster. Seed servers play a crucial role in cluster formation and recovery, acting as a point of reference for new or restarting brokers to understand the current topology of the cluster. ### [](#segment)segment Discrete part of a partition, used to break down a continuous stream into manageable chunks. You can set the maximum duration (`segment.ms`) or size (`segment.bytes`) for a segment to be open for writes. ### [](#serialization)serialization The process of converting a record into a format that can be stored. Deserialization is the process of converting a record back to the original state. Redpanda Schema Registry supports Avro and Protobuf serialization formats. ### [](#shard)shard A CPU core. ### [](#subject)subject A logical grouping or category for schemas. When data formats are updated, a new version of the schema can be registered under the same subject, allowing for backward and forward compatibility. ### [](#thread-per-core)thread-per-core Programming model that allows Redpanda to pin each of its application threads to a CPU core to avoid context switching and blocking. ### [](#topic-partition)topic partition A topic may be partitioned through multiple brokers. A "topic partition" represents this logical separation in Redpanda, which is managed natively by Raft. ### [](#topic)topic A logical stream of related events that are written to the same log. It can be divided into multiple partitions. A topic can have various clients writing events to it and reading events from it. ### [](#wire-format)wire format A serialization format used by Schema Registry that prefixes each record with a magic byte and 4-byte schema ID, so consumers can identify and apply the correct schema. ## [](#redpanda-features)Redpanda features ### [](#admin-api)Admin API A REST API used to manage and monitor Redpanda Self-Managed clusters. It uses the default port 9644. For more information about using this API with Self-Managed Redpanda, see [/api/doc/admin](https://docs.redpanda.com/api/doc/admin). Note: The Redpanda Admin API is different from the [Kafka Admin API](https://kafka.apache.org/documentation/#adminapi). ### [](#cloud-topic)Cloud Topic A Redpanda topic type, Cloud Topics use object storage (S3, GCS, or MinIO) as the primary data store (rather than replicating data across brokers). Unlike standard Redpanda topics, Cloud Topics allow users with flexible latency requirements to lower or eliminate costs associated with cross-AZ networking. ### [](#compaction)compaction Feature that retains the latest value for each key within a partition while discarding older values. ### [](#controller-snapshot)controller snapshot Snapshot of the current cluster metadata state saved to disk, so broker startup is fast. ### [](#data-transforms)data transforms Framework to manipulate or enrich data written to Redpanda topics. You can develop custom data functions, which run asynchronously using a WebAssembly (Wasm) engine inside a Redpanda broker. ### [](#http-proxy)HTTP Proxy Redpanda HTTP Proxy (pandaproxy) allows access to your data through a REST API. It is built into the Redpanda binary and uses the default port 8082. ### [](#leader-pinning)Leader Pinning Feature that places a topic’s partition leaders in a preferred location, such as a cloud availability zone, to reduce networking costs and latency for nearby clients. ### [](#maintenance-mode)maintenance mode A state where a Redpanda broker temporarily doesn’t take any partition leaderships. It continues to store data as a follower. This is usually done for system maintenance or a rolling upgrade. ### [](#rack-awareness)rack awareness Feature that lets you distribute replicas of the same partition across different racks to minimize data loss and improve fault tolerance in the event of a rack failure. ### [](#rebalancing)rebalancing Process of moving partition replicas and transferring partition leadership for improved performance. Redpanda provides various topic-aware tools to balance clusters for best performance. - Leadership balancing changes where data is written to first, but it does not involve any data transfer. The partition leader regularly sends heartbeats to its followers. If a follower does not receive a heartbeat within a timeout, it triggers a new leader election. Redpanda also provides leadership balancing when brokers are added or decommissioned. - Partition replica balancing moves partition replicas to alleviate disk pressure and to honor the configured replication factor across brokers and the additional redundancy across failure domains (such as racks). Redpanda provides partition replica rebalancing when brokers are added or decommissioned. - With an Enterprise license, you can additionally enable Continuous Data Balancing to continuously monitor broker and rack availability and disk usage. ### [](#rolling-upgrade)rolling upgrade The process of upgrading each broker in a Redpanda cluster, one at a time, to minimize disruption and ensure continuous availability. ### [](#rpk)rpk Redpanda’s command-line interface tool for managing Redpanda clusters. ### [](#remote-read-replica)Remote Read Replica A read-only topic that mirrors a topic on a different cluster, using data from Tiered Storage. ### [](#schema-registry-context)Schema Registry context An independent namespace for subjects within a Schema Registry. Contexts let multiple groups of schemas coexist in one registry without naming conflicts. The default context is named ".". ### [](#schema-registry)Schema Registry Redpanda Schema Registry (pandaproxy) is the interface for storing and managing event schemas. Producers and consumers register and retrieve schemas they use from the registry. It is built into the Redpanda binary and uses the default port 8081. ### [](#shadow-cluster)shadow cluster A read-only Redpanda cluster that continuously receives replicated data from a source cluster through a shadow link. During a disaster, you can fail over to the shadow cluster so it handles production traffic. ### [](#shadow-link)shadow link A persistent connection between two Redpanda clusters that continuously replicates topic data, metadata, consumer offsets, ACLs, and Schema Registry content from a source cluster to a shadow cluster for disaster recovery. ### [](#tiered-storage)Tiered Storage Feature that lets you offload log segments to object storage in near real-time, providing long-term data retention and topic recovery. ## [](#redpanda-in-kubernetes)Redpanda in Kubernetes ### [](#cert-manager)cert-manager A Kubernetes controller that simplifies the process of obtaining, renewing, and using certificates. For more details, see [https://cert-manager.io/docs/](https://cert-manager.io/docs/) ### [](#redpanda-helm-chart)Redpanda Helm chart Generates and applies all the manifest files you need for deploying Redpanda in Kubernetes. ### [](#redpanda-operator)Redpanda Operator Extends Kubernetes with custom resource definitions (CRDs), which allow Redpanda clusters to be treated as native Kubernetes resources. ## [](#redpanda-licenses)Redpanda licenses ### [](#redpanda-community-edition)Redpanda Community Edition Redpanda software that is available under the Redpanda Business Source License (BSL). These core features are free and source-available. ### [](#redpanda-enterprise-edition)Redpanda Enterprise Edition Redpanda software that is available under the Redpanda Community License (RCL). It includes the free features licensed with the Redpanda Community Edition, as well enterprise features, such as Tiered Storage, Remote Read Replicas, and Continuous Data Balancing. ### [](#self-managed)Self-Managed Redpanda Self-Managed refers to the product offering that includes both the Enterprise Edition and the Community Edition of Redpanda. Sometimes used interchangeably with self-hosted. ## [](#redpanda-security)Redpanda security ### [](#access-control-list-acl)access control list (ACL) A security feature used to define and enforce granular permissions to resources, ensuring only authorized users or applications can perform specific operations. ACLs act on principals. ### [](#advertised-listener)advertised listener The address a Redpanda broker broadcasts to producers, consumers, and other brokers. It specifies the hostname and port for connections to different listeners. Clients and other brokers use advertised listeners to connect to services such as the Admin API, Kafka API, and HTTP Proxy API. The advertised address might differ from the listener address in scenarios where brokers are behind a NAT, in a Docker container, or in Kubernetes. Advertised addresses ensure clients can reach the Redpanda brokers even in complex network setups. ### [](#authentication)authentication The process of verifying the identity of a principal, user, or service account. Also known as AuthN. ### [](#authorization)authorization The process of specifying access rights to resources. Access rights are enforced through roles or access control lists (ACLs). Also known as AuthZ. ### [](#bearer-token)bearer token An access token used for authentication and authorization in web applications and APIs. It holds user credentials, usually in the form of random strings of characters. ### [](#gbac)GBAC Group-based access control lets you manage Redpanda permissions at scale by assigning them to OIDC groups instead of individual users. GBAC lets you manage Redpanda permissions at scale using the groups that already exist in your identity provider (IdP). You define access once for a group and your IdP controls who belongs to it. You can grant permissions to groups in two ways: create ACLs with `Group:` principals, or assign groups as members of RBAC roles. Both approaches can be used independently or together. ### [](#identity-provider-idp)identity provider (IdP) A service that creates, maintains, and manages identity information while providing authentication services to applications. Identity providers authenticate users and issue tokens that applications can use to verify identity and access permissions. Common IdPs include Okta, Auth0, Azure AD, and Google Identity Platform. ### [](#openid-connect-oidc)OpenID Connect (OIDC) Authentication layer built on OAuth 2.0 that allows clients to verify user identity and obtain basic profile information. OpenID Connect provides a standardized way for applications to authenticate users through identity providers. In Redpanda’s agentic systems, OIDC enables secure authentication for AI agents and MCP servers accessing cloud resources. ### [](#principal)principal An authenticated identity (user, service account, or group) that Redpanda evaluates when enforcing ACLs and role assignments. Redpanda supports `User:` and `Group:` principal types. Permissions are granted to principals through ACLs or RBAC role assignments. ### [](#rbac)RBAC Role-based access control lets you assign users access to specific resources. ### [](#service-account)service account An identity independent of the user who created it that can be used to authenticate and perform operations. This is especially useful for authentication of machines. --- # Page 82: Release Notes **URL**: https://docs.redpanda.com/agentic-data-plane/reference/release-notes.md --- # Release Notes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Release Notes latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: release-notes page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: release-notes.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/release-notes.adoc description: "What's new in the Agentic Data Plane: new features, improvements, and bug fixes, by release." page-topic-type: reference page-git-created-date: "2026-07-01" page-git-modified-date: "2026-08-12" --- The latest updates to the Agentic Data Plane. For the in-product summary, open **What’s new** from the top bar. ## [](#v0-2-40-2026-08-12)v0.2.40 (2026-08-12) ### [](#improvements)Improvements Platform This release includes general stability improvements. ## [](#v0-2-39-2026-08-12)v0.2.39 (2026-08-12) ### [](#improvements-2)Improvements Security This release includes general security and stability improvements. ## [](#v0-2-38-2026-08-12)v0.2.38 (2026-08-12) ### [](#features)Features LLM Providers `[UI]` The provider **Overview** tab shows which agents and users call which models, with requests, errors, and spend per caller, over a preset period or an exact window. LLM Providers `[UI]` All provider settings are on one **Settings** tab. Old edit-page links still work. MCP Servers The SQL MCP server can return rows as named fields, which a data policy can target column by column, so a policy can mask a salary column and leave the rest readable. Set `row_format` to `ROW_FORMAT_OBJECT`. Positional rows stay the default. ### [](#improvements-3)Improvements Cost & Usage `[UI]` Report on an exact UTC window, to the minute, not only a relative period such as the last 7 days. ### [](#bug-fixes)Bug fixes Cost & Usage Spending keys on each agent’s permanent identifier, so reusing a deleted agent’s name no longer mixes the two agents' costs. Past periods are corrected too. Home `[UI]` The top spenders card counts agents that shared a name as one, so its top five are the real top five. My Connections `[UI]` Managed MCP servers are listed on the connections they use, with any scopes those servers need that the connection is missing. Interface `[UI]` Code blocks are readable in dark mode, and opening a menu no longer shifts the page. Agents `[UI]` The self-managed agent **Setup** tab is corrected: the `X-Redpanda-Genai-Conversation` step is gone, conversations group on `gen_ai.conversation.id`, the Java sample sends its token request, and the Vercel AI SDK, CrewAI, and Mastra samples match current SDKs. Take the current version of any sample you copied. ## [](#v0-2-37-2026-08-10)v0.2.37 (2026-08-10) ### [](#improvements-4)Improvements MCP Servers `[UI]` The **Data Policies** tab now counts a server’s tools by protection state: **protected** for a tool a data policy covers, **findings** for a tool with no policy that returns fields whose names look sensitive, and **no policy** for the rest. Every tool falls in exactly one group. The tool lists on **Data Policies** and the MCP server **Inspector** tab are also searchable, and long tool names wrap in full instead of being cut off, so tools whose names differ only at the end are easier to tell apart. Agents `[UI]` The agent **Inspector** tab is now called **Playground**. ### [](#bug-fixes-2)Bug fixes LLM Providers `[UI]` Editing a provider’s connection settings in the ADP UI, for example pointing it at a different API key secret, reported success and kept the old value. Those edits now save. The display name and model selection were unaffected, as were changes made with the `rpk ai` plugin or the API. Re-open any provider whose connection settings you edited in the UI and confirm it uses the secret you intended. Agents `[UI]` The agent **Playground** now tells you when an agent has no subagents attached instead of opening an empty panel. OAuth Providers `[UI]` All granted scopes are readable on provider and client detail pages. A long scope ran outside its popup, and a long list of scopes ran off the bottom of the screen with no way to scroll, so not every scope was visible. Interface `[UI]` Side panels close when you click outside them. ## [](#v0-2-36-2026-08-06)v0.2.36 (2026-08-06) ### [](#improvements-5)Improvements Security This release includes general security and stability improvements. ## [](#v0-2-35-2026-08-06)v0.2.35 (2026-08-06) ### [](#features-2)Features Access `[UI]` The new **Access** page, available as a preview, controls what people can do with the resources in your ADP environment. Write your own permit and forbid rules on **Policies**, or start from a **Templates** preset: **Read only**, **Sandboxed** to use agents and MCP servers without changing them, **Standard** to also manage resources, or **Full access**. **Roles** shows what each of your organization’s roles can do, and **System policies** shows who already has access through those roles. Agents `[UI]` The agent prompt editor now formats your prompt as you write it. **Live** mode renders headings, lists, and emphasis, and shows the raw Markdown only on the line your cursor is on. Switch between **Live**, **Markdown**, and **Preview** without losing your place, or start from a template for a support agent, data analyst, coding assistant, incident responder, or research coordinator. ### [](#improvements-6)Improvements Agents `[UI]` The agent **Setup** tab splits MCP servers into **Managed**, run by Redpanda, and **Self-managed**, run by you, and shows each managed server’s logo, so a long list is easier to scan. Interface `[UI]` The app opens faster on a first visit. ### [](#bug-fixes-3)Bug fixes OAuth Clients Claude can now connect to an MCP server on ADP with a Client ID Metadata Document (CIMD). Retry the connection from Claude, with nothing to change on your side. Connecting failed with **unable to retrieve client by id** because ADP refused an entire document over a single OAuth grant type that the document lists and ADP does not issue. ADP now ignores those grant types and honors the rest of the document, for any client, not only Claude. Agents `[UI]` The create-agent wizard previewed a self-managed agent’s client ID in a format that `rpk ai agent credential create` no longer returns. Only the preview was wrong, so credentials you already issued are unaffected. Interface `[UI]` Notifications are readable in dark mode instead of washing out against a dark background, and they now appear at the top right of the screen. ## [](#v0-2-34-2026-08-04)v0.2.34 (2026-08-04) ### [](#features-3)Features OAuth Clients An MCP client can now reach a protected server with a Client ID Metadata Document (CIMD), the MCP spec’s successor to Dynamic Client Registration. The app hosts its own metadata at an `https` URL and presents that URL as its client ID, so nobody creates a client by hand. Turn on **Accept client metadata documents (CIMD)** on the **Integrations setup** page, choose whether to accept documents from **Any domain** or **Only these domains**, and list the MCP servers those clients can reach. Models GPT-5.6 Sol, Terra, and Luna are now in the Amazon Bedrock catalog. Enable `openai.gpt-5.6-sol`, `openai.gpt-5.6-terra`, or `openai.gpt-5.6-luna` on a Bedrock provider. ### [](#improvements-7)Improvements Security This release includes general security and stability improvements. ### [](#bug-fixes-4)Bug fixes LLM Providers Creating an LLM provider now works on Google Cloud environments. Saving the provider’s API key failed because the form attached an internal label that Google Secret Manager rejects. Cost & Usage Cost reporting overstated GPT-5.6 Luna and Terra spend. OpenAI reduced the rates for both models on July 30, so reported cost now matches the current rates. Figures recorded before this release keep their earlier, higher numbers. ## [](#v0-2-33-2026-08-03)v0.2.33 (2026-08-03) ### [](#improvements-8)Improvements MCP Servers `[UI]` The server details page groups setup, tools, and connection details more clearly, and tool schemas stay readable when you expand a nested object. Environments `[UI]` Switching to another environment now takes you to the home page of that environment, instead of leaving you on a page that belongs to the previous one. Security This release includes general security and stability improvements. ### [](#bug-fixes-5)Bug fixes OAuth Clients An MCP client that registers itself with Dynamic Client Registration (DCR, RFC 7591) can now connect to a server on ADP. Claude.ai, Claude Code, and Cursor were rejected during registration, and the error named the MCP server’s vendor rather than the real cause. Interface `[UI]` Confirmation dialogs now place their buttons in the same order as the rest of the product. ## [](#v0-2-32-2026-07-30)v0.2.32 (2026-07-30) ### [](#features-4)Features MCP Servers Connecting a remote MCP server that uses **User OAuth** now takes a URL and one sign-in per person, with no app created by hand in a vendor console. Select **Set up automatically** and ADP reads the server’s OAuth configuration and registers itself as a client using Dynamic Client Registration (DCR, RFC 7591). For a server that does not support DCR, you can still enter a client ID and secret yourself. Models Claude Opus 5, Gemini 3.6 Flash, and Gemini 3.5 Flash Lite are now in the model catalog. Enable `claude-opus-5` on an Anthropic provider, or `gemini-3.6-flash` and `gemini-3.5-flash-lite` on a Google provider. Claude Opus 5 is also in the Amazon Bedrock catalog. CLI The `rpk ai connection` commands bring your own OAuth connections to the terminal, matching what **My Connections** shows in the UI. Run `rpk ai connection list` to see the services you are signed in to, and `rpk ai connection revoke ` to disconnect your own account from one. ### [](#improvements-9)Improvements LLM Providers `[UI]` Creating an LLM provider is now a single form: paste model IDs one per line, and create the provider’s API key secret without leaving the page. For Amazon Bedrock you can browse the catalog by model family and enable or disable models in bulk. MCP Servers OpenAI MCP servers now transcribe audio files with `gpt-transcribe` instead of `whisper-1`. Servers that set a transcription model explicitly keep it. MCP Servers `[UI]` When the inspector cannot reach a server, it now explains the failure in readable text and offers **Try again** and **Edit server**. The raw response stays available in a collapsed panel for network debugging. ### [](#bug-fixes-6)Bug fixes MCP Servers Remote MCP servers set to **OAuth (Service Account)** now attach the service account’s access token to their upstream requests. They previously sent no credentials at all, so a server requiring authentication rejected every request. Cost & Usage Cost reporting understated large Anthropic requests: above 200,000 input tokens, Anthropic charges a higher long-context rate that ADP was not applying. Those requests now report their real cost, so expect reported spend to step up with no change in usage. Figures recorded before this release keep their earlier, lower numbers. Interface `[UI]` Selecting **Contact support** on an error now files a support request with Redpanda. The error was previously logged for diagnostics without a request being created. ## [](#v0-2-31-2026-07-28)v0.2.31 (2026-07-28) ### [](#improvements-10)Improvements Environments `[UI]` When your organization has no Agentic Data Plane environment yet, ADP now shows which of your clusters can host one, with each cluster’s cloud region and city. ### [](#bug-fixes-7)Bug fixes Activity Conversation transcripts are recorded again after new conversations stopped appearing in the Activity views on **LLM Providers** and **Cost & Usage** in the previous release. Interface `[UI]` Filtering and paging through lists now work correctly across **Connections**, **Integrations**, and **Secrets**. Filter checkboxes show and clear what you selected, and lists no longer jump to the wrong page while data reloads. ## [](#v0-2-30-2026-07-27)v0.2.30 (2026-07-27) ### [](#improvements-11)Improvements Security This release includes general security and stability improvements. ## [](#v0-2-29-2026-07-27)v0.2.29 (2026-07-27) ### [](#features-5)Features Agents Agents now support scheduled triggers. Give an agent a cron schedule and it runs automatically at the times you set. MCP Servers You can now pick a token-optimized output format for each MCP server. Set a server’s **Output format** to JSON, TOON, or JTON to control how tool results are encoded before an agent reads them. ### [](#bug-fixes-8)Bug fixes MCP Servers Connecting a remote MCP server now streams its responses reliably, so its tools load instead of appearing empty. ## [](#v0-2-28-2026-07-24)v0.2.28 (2026-07-24) ### [](#features-6)Features MCP Servers As a preview, MCP servers now have a **Data Policies** tab where you can define rules that mask, redact, or filter fields out of a tool’s results before the model sees them. Author and preview policies per server. MCP Servers OpenAPI MCP servers can now authenticate to a downstream API with an OAuth client-credentials (service-account) flow. All requests share one service-account identity instead of each user’s own credentials. ### [](#improvements-12)Improvements MCP Servers OpenAPI MCP servers now return tool results as structured output with the response’s status code, headers, and parsed body, so agents can read them reliably. LLM Providers `[UI]` You can now find a provider in the list by its display name, not just its ID. ### [](#bug-fixes-9)Bug fixes CLI Deriving an environment with `rpk ai env add --from ` now inherits the base environment’s ADP API URL and authentication mode instead of resetting them to defaults. CLI The `rpk ai env list` output no longer shows duplicate or stale local-only environments. ## [](#v0-2-27-2026-07-22)v0.2.27 (2026-07-22) ### [](#improvements-13)Improvements Security This release includes general security and stability improvements. ## [](#v0-2-26-2026-07-22)v0.2.26 (2026-07-22) ### [](#features-7)Features OAuth Providers Connecting a remote MCP server that uses OAuth no longer requires you to paste in authorization and token endpoints or a client ID and secret. Redpanda reads the server’s OAuth configuration from its URL and registers a client automatically through Dynamic Client Registration (DCR). MCP Servers The ServiceNow MCP server can now search CMDB configuration items, so agents can look up servers, laptops, applications, and services. ### [](#improvements-14)Improvements MCP Servers `[UI]` The setup guides for connecting an AI client now cover Claude, ChatGPT, Zed, and the Codex CLI, with corrected steps. MCP Servers `[UI]` The copy-paste code examples for connecting to MCP servers and LLM providers now use the latest SDK versions across Node, Python, Go, Java, and curl. Interface `[UI]` List rows are now fully clickable to open the detail view, instead of only the name link. Interface `[UI]` When something fails, the interface now offers clear next steps such as retry, reload, and copying the error details. ### [](#bug-fixes-10)Bug fixes My Connections Connecting a remote MCP server that signs you in with your own account (user-delegated OAuth) now works. Sign-in no longer fails before completing. ## [](#v0-2-25-2026-07-17)v0.2.25 (2026-07-17) ### [](#features-8)Features Models `[UI]` Every LLM provider now has a **Models** tab where you can browse the models it serves, turn individual models on or off, and filter them by capability or status. LLM Providers `[UI]` Each provider now has an **Activity** tab that shows the agents, requests, and conversations routed through it over a time range you choose. ### [](#improvements-15)Improvements LLM Providers `[UI]` The providers list shows each provider’s models with small trend charts for recent requests and spend, and you can filter the list to providers that serve a specific model. LLM Providers `[UI]` Each provider’s **Settings** tab shows which Secret Store entries authenticate it, with a shortcut to manage those keys. LLM Providers `[UI]` The **Connect** setup guide lets you pick your coding agent and operating system, and gives copy-paste `rpk ai` steps tailored to your environment. MCP Servers `[UI]` Running a tool in the server inspector now shows clear, actionable validation errors for invalid inputs instead of a generic failure. ### [](#bug-fixes-11)Bug fixes OAuth Clients Connecting an AI client to a **Code Mode** MCP server over OAuth now succeeds; Code Mode adds tools that let agents run sandboxed code against the server. Clients such as Claude Code and the MCP Inspector now register automatically through Dynamic Client Registration (DCR) instead of failing with a missing-scope error. My Connections `[UI]` Using the browser Back button after starting a connection no longer leaves the **Connect** button stuck; the connection state resets cleanly. Home `[UI]` A brand-new environment no longer shows a misleading all-clear health status before you have set anything up. ## [](#v0-2-24-2026-07-13)v0.2.24 (2026-07-13) ### [](#features-9)Features Models OpenAI providers now offer the GPT-5.6 model family: GPT-5.6 Luna, GPT-5.6 Terra, and GPT-5.6 Sol. ### [](#improvements-16)Improvements Models `[UI]` Each model now has a detail page showing its overview, pricing, and capabilities. ### [](#bug-fixes-12)Bug fixes Models `[UI]` Model usage now shows a clear error when a usage query fails, instead of misleading zero values. Cost & Usage Cost and usage reports now count cached tokens accurately even when a provider returns incomplete usage data. MCP Servers `[UI]` The inspector shows upstream HTML and XML errors as readable text instead of raw markup. My Connections `[UI]` The Connect and Reconnect buttons now show a loading state while the provider’s consent screen opens. ## [](#v0-2-23-2026-07-13)v0.2.23 (2026-07-13) ### [](#features-10)Features Models You can now use the Google Gemma 4 model family with your Amazon Bedrock providers, in three sizes from largest to smallest: Gemma 4 31B, Gemma 4 26B-A4B, and Gemma 4 E2B. ### [](#improvements-17)Improvements Integrations setup `[UI]` Manage OAuth providers and MCP client access in one tabbed **Integrations setup** area, replacing the separate pages. My Connections `[UI]` The **My Connections** page groups your account connections by status and lets you connect or reconnect without leaving the page. Security This release includes general security and stability improvements. ### [](#bug-fixes-13)Bug fixes My Connections Connections that renew automatically, for example GitHub, no longer show as expired shortly after connecting. You are prompted to reconnect only when a connection can no longer renew itself. ## [](#v0-2-22-2026-07-10)v0.2.22 (2026-07-10) ### [](#features-11)Features Models Amazon Bedrock providers now offer two new models: Mistral Large 3 and Amazon Nova 2 Lite. ### [](#improvements-18)Improvements MCP Servers The Amazon Bedrock MCP server can now generate embeddings with Cohere Embed v4. Agents Creating or updating an agent now rejects an invalid Bedrock model name immediately, instead of accepting it and failing later. Agents `[UI]` When an agent task is still running after five minutes, the inspector stops the live view and points you to the **Activity** page to follow it; the task keeps running. Security This release includes general security and stability improvements. ### [](#bug-fixes-14)Bug fixes Agents A reply that was cut off by the model’s token limit no longer causes the next message in the conversation to fail. ## [](#v0-2-20-2026-07-09)v0.2.20 (2026-07-09) ### [](#features-12)Features Cost & Usage Tag your agents and break down cost and usage by tag: group the report by a tag key, filter by tag values, and see untagged traffic in its own bucket. ### [](#improvements-19)Improvements Security This release includes general security and stability improvements. ## [](#v0-2-19-2026-07-08)v0.2.19 (2026-07-08) ### [](#features-13)Features Cost & Usage Download your full cost and usage report as a CSV, broken down by provider, model, user, or agent and by hour, day, or month. ### [](#improvements-20)Improvements Home The home dashboard is rebuilt around an overall health status, an hourly chart of token use and spend, and your top spending agents and users at a glance. LLM Providers `[UI]` The Connect tab gives you ready-to-run `rpk ai` setup steps and client code snippets, including running Claude Code against a Bedrock provider without local AWS credentials. Interface When a request fails, a clearer error page lets you retry, open full error details, and copy the failing request as a `curl` command to reproduce or report it. Security This release includes general security and stability improvements. ### [](#bug-fixes-15)Bug fixes Agents `[UI]` In the agent editor, the model picker now stays anchored to its field instead of jumping to the side when you open it. Interface After a new release is deployed, the app now reloads cleanly instead of getting stuck on a stale page. ## [](#v0-2-18-2026-07-03)v0.2.18 (2026-07-03) ### [](#improvements-21)Improvements Interface Text across the interface is now larger and easier to read. Security This release includes general security and stability improvements. ### [](#bug-fixes-16)Bug fixes Interface Collapsible and accordion panels now animate smoothly. ## [](#v0-2-17-2026-07-02)v0.2.17 (2026-07-02) ### [](#features-14)Features CLI Launch Claude Code against one of your Amazon Bedrock LLM providers with `rpk ai run claude -L `; the gateway authenticates to AWS for you, so your machine needs no AWS credentials. Agents Give each subagent its own model and LLM provider in the agent editor. ### [](#improvements-22)Improvements Agents Updating an agent no longer interrupts it: the new version starts and takes over before the old one stops. Models Each model now reports its `max_input_tokens` and `max_output_tokens` limits through the API. ### [](#bug-fixes-17)Bug fixes LLM Providers `[UI]` Filters in the metric detail panel on provider and model pages now apply to the chart, and the panel no longer hides its filter menus or covers the sidebar. LLM Providers `[UI]` The spend total in the metric detail panel was inflated when grouping by cost type; only the display was wrong, recorded usage was unaffected. Cost & Usage `[UI]` In period comparisons grouped by model or user, each line now compares against its own numbers from the previous period, not another line’s. ## [](#v0-2-16-2026-07-01)v0.2.16 (2026-07-01) ### [](#features-15)Features OAuth Clients MCP clients such as Claude Code, Cursor, and ChatGPT can now register themselves through OAuth Dynamic Client Registration (DCR), instead of needing an OAuth client created by hand. ### [](#improvements-23)Improvements Agents `[UI]` Creating policies is now faster, since resource options load as you need them instead of all at once. MCP Servers `[UI]` Claude Code users can now set up managed MCP servers with a single command, or use manual configuration from a separate tab. ## [](#v0-2-12-2026-07-01)v0.2.12 (2026-07-01) ### [](#features-16)Features Models Claude Sonnet 5 is now available to select on the Anthropic and Bedrock providers. ### [](#bug-fixes-18)Bug fixes Agents `[UI]` A completed tool call in the agent inspector now switches to **Completed** the moment it finishes, instead of showing **Working** until the next step. ## [](#v0-2-11-2026-06-30)v0.2.11 (2026-06-30) ### [](#bug-fixes-19)Bug fixes CLI Signing in with `rpk ai auth login` now clears the expired-token errors that kept `rpk ai` commands failing even after a successful login. ## [](#v0-2-10-2026-06-30)v0.2.10 (2026-06-30) ### [](#features-17)Features Agents `[UI]` When an agent produces a chart, the agent inspector now displays it: switch between the chart, its data, and its definition, and zoom in or download it. Error handling `[UI]` Capture a failed request for support and get a reference ID to quote. ### [](#improvements-24)Improvements Error handling `[UI]` Failed requests now show the full error message and type, with expandable technical details, instead of a generic internal error. List pages `[UI]` The MCP servers, LLM providers, OAuth providers, OAuth clients, and Secrets Store lists now have consistent search, filtering, and multi-row selection. ### [](#bug-fixes-20)Bug fixes LLM Providers The Bedrock provider rejected valid model IDs whose version contains a dot, in all regions; these now route correctly. ## [](#v0-2-9-2026-06-29)v0.2.9 (2026-06-29) ### [](#features-18)Features MCP Servers Connect to Sentry through a new managed MCP server. LLM Providers Set your own per-model pricing on a provider, so usage and cost reflect the rates you actually pay. rpk ai `[CLI]` Apply or diff your LLM providers, MCP servers, and OAuth providers from config files, now against an Agentic Data Plane environment. ### [](#improvements-25)Improvements MCP Servers Filter the server list by type and status. OAuth Providers Filter providers and clients by status and how they were created. Cost & Usage Export charts to CSV and download a full usage report. Environments The environment switcher now shows each environment’s region, so you don’t make changes in the wrong one. ### [](#bug-fixes-21)Bug fixes Reliability Fixed hangs when switching environments, and the app now recovers on its own after a new version ships instead of showing errors. Lists Fixed filters showing wrong results, and lists now refresh immediately after you add or edit an item. --- # Page 83: Install or Update rpk **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk-install.md --- # Install or Update rpk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: Install or Update rpk latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk-install page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk-install.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk-install.adoc description: Install or update rpk to interact with Redpanda from the command line. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-05-28" --- The `rpk` tool is a single binary application that provides a way to interact with your Redpanda clusters from the command line. For example, you can use `rpk` to do the following: - Monitor your cluster’s health - Create, produce, and consume from topics - Set up access control lists (ACLs) and other security features Redpanda Cloud deployments should always use the latest version of `rpk`. ## [](#check-rpk-version)Check rpk version To check your current version of the rpk binary, run `rpk --version`. The following example lists the latest version of `rpk`. If your installed version is lower than this latest version, then update `rpk`. For a list of versions, see [Redpanda releases](https://github.com/redpanda-data/redpanda/releases/). ```bash rpk --version ``` ```bash rpk version 26.2.1 (rev 5218325) ``` ## [](#install-or-update-rpk-on-linux)Install or update rpk on Linux To install, or update to, the latest version of `rpk` for Linux, run: ### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. ## [](#install-or-update-rpk-on-macos)Install or update rpk on macOS ### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` ### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. #### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` #### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ## [](#upgrade-rpk-plugins)Upgrade rpk plugins Plugins, such as `rpk connect` and `rpk ai`, are separate binaries that `rpk` downloads and runs on your behalf. Upgrading `rpk` does not upgrade its plugins. An installed plugin stays at its version until you upgrade it, even across `rpk` upgrades. To see which plugins are installed: ```bash rpk plugin list --local ``` To upgrade a plugin, run its `upgrade` command. For example: ```bash rpk connect upgrade ``` > 📝 **NOTE** > > Some plugins install themselves the first time you run one of their commands. For example, `rpk connect run` downloads the latest Redpanda Connect if it is not already installed. This happens only on first use. After that, the plugin never updates itself. ## [](#next-steps)Next steps For the complete list of `rpk` commands and their syntax, see the [rpk reference](https://docs.redpanda.com/agentic-data-plane/reference/rpk/). --- # Page 84: rpk Command Reference **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk.md --- # rpk Command Reference > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk Command Reference latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/index page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/index.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/index.adoc description: Reference for the rpk commands used to manage the Redpanda Agentic Data Plane from the command line. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-07-29" --- This section documents the `rpk` commands you use to manage the Redpanda Agentic Data Plane from the command line. Use `rpk ai` to manage AI Gateway resources (LLM providers, MCP servers, OAuth providers) and call MCP tools directly from your terminal. `rpk ai` is self-contained: it has its own login (`rpk ai auth login`) and its own Agentic Data Plane environment selection (`rpk ai env`), independent of any `rpk cloud` session. The active AI Gateway URL comes from the selected environment unless you override it with `--rpai-endpoint`. For subcommands available after install (`rpk ai llm-provider`, `rpk ai mcp-server`, `rpk ai oauth-provider`, and more), run `rpk ai --help` from your terminal. - [rpk ai](rpk-ai/rpk-ai/) Manage the Redpanda AI Gateway. - [rpk profile](rpk-profile/rpk-profile/) Manage `rpk` profiles. An rpk profile talks to a single Redpanda cluster. - [rpk -X](rpk-x-options/) Override any rpk configuration option for a single command with the -X flag, or set it persistently with an RPK\_ environment variable. --- # Page 85: rpk ai agent a2a card **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-card.md --- # rpk ai agent a2a card > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a card latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-card page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-card.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-card.adoc description: "Fetch the A2A agent card: the JSON discovery document describing the agent's identity, skills, supported transports and capabilities. Use -o json to feed the raw card to other tools." page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Fetch the A2A agent card: the JSON discovery document describing the agent’s identity, skills, supported transports and capabilities. Use `-o json` to feed the raw card to other tools. ## [](#usage)Usage ```bash rpk ai agent a2a card [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --timeout | duration | abort the call after this long (0 to wait forever). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 86: rpk ai agent a2a send **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-send.md --- # rpk ai agent a2a send > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a send latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-send page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-send.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-send.adoc description: Send a text message to an agent over A2A and print the reply. The message comes from the positional argument, or from stdin when the argument is omitted or "-" (so you can pipe a prompt in). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Send a text message to an agent over A2A and print the reply. The message comes from the positional argument, or from stdin when the argument is omitted or "-" (so you can pipe a prompt in). By default the call blocks until the agent replies. Replies that spawn a long-running task print the task id so you can follow up with `rpk ai agent a2a task get|watch|cancel`. For work that may outlive `--timeout` (default 5m), prefer `--stream` or `--no-block` so the task id is in hand from the start. Conversation state: every reply prints a context-id (stderr in the default format, part of the JSON in `-o json`). Pass it back via `--context-id` to continue the same conversation. When a task ends in state input-required, answer it by sending again with both `--task-id` and `--context-id` from the reply. Output: the agent’s reply text goes to stdout; ids and state go to stderr as `key: value` lines so pipes stay clean. Use `-o json` for the full A2A response (message or task object). With `--stream`, `json` and `yaml` both emit one JSON event per line (JSONL). Exit codes: 0 success or input-required, 4 task failed/canceled/ rejected, 1 anything else. ## [](#usage)Usage ```bash rpk ai agent a2a send [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai agent a2a send`. Ask and wait for the answer ```bash rpk ai agent a2a send financial-advisor "What moved the S&P 500 today?" ``` Continue the conversation from a previous reply’s context-id ```bash rpk ai agent a2a send financial-advisor --context-id CTX "Why?" ``` Answer a task that ended in input-required ```bash rpk ai agent a2a send financial-advisor --task-id TASK --context-id CTX "Account A-17" ``` Pipe the prompt from a file, get the full JSON reply ```bash cat prompt.txt | rpk ai agent a2a send financial-advisor -o json ``` Stream events as they happen ```bash rpk ai agent a2a send financial-advisor --stream "Give me a market summary" ``` Fire-and-forget: submit, then poll with `task get` ```bash rpk ai agent a2a send financial-advisor --no-block "Deep analysis please" ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --context-id | string | continue an existing conversation (printed by a previous send). | | --no-block | bool | return immediately with the submitted task instead of waiting for completion. | | --stream | bool | stream the reply as A2A events (message/stream) instead of waiting for the final result. | | --task-id | string | continue an existing task (for example, answer an input-required task). | | --timeout | duration | abort the call after this long (0 to wait forever). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 87: rpk ai agent a2a task cancel **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-task-cancel.md --- # rpk ai agent a2a task cancel > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a task cancel latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-task-cancel page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-task-cancel.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-task-cancel.adoc description: Ask the agent to cancel a running task and print the task's resulting state. Cancellation is cooperative; the agent may already have finished, in which case the terminal state is returned unchanged. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Ask the agent to cancel a running task and print the task’s resulting state. Cancellation is cooperative; the agent may already have finished, in which case the terminal state is returned unchanged. ## [](#usage)Usage ```bash rpk ai agent a2a task cancel [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai agent a2a task cancel`. ```bash rpk ai agent a2a task cancel financial-advisor TASK_ID ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --timeout | duration | abort the call after this long (0 to wait forever). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 88: rpk ai agent a2a task get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-task-get.md --- # rpk ai agent a2a task get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a task get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-task-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-task-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-task-get.adoc description: Fetch a task's current state, status message, artifacts and (optionally truncated) message history. Use -o json for the full task object. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Fetch a task’s current state, status message, artifacts and (optionally truncated) message history. Use `-o json` for the full task object. ## [](#usage)Usage ```bash rpk ai agent a2a task get [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai agent a2a task get`. ```bash rpk ai agent a2a task get financial-advisor TASK_ID rpk ai agent a2a task get financial-advisor TASK_ID --history 10 -o json ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --history | int | number of most recent history messages to include. | | --timeout | duration | abort the call after this long (0 to wait forever). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 89: rpk ai agent a2a task watch **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-task-watch.md --- # rpk ai agent a2a task watch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a task watch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-task-watch page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-task-watch.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-task-watch.adoc description: Reattach to a running task's event stream (A2A tasks/resubscribe) and print events until the task reaches a terminal state. Use after a disconnected --stream send or a --no-block send. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Reattach to a running task’s event stream (A2A tasks/resubscribe) and print events until the task reaches a terminal state. Use after a disconnected `--stream` send or a `--no-block` send. Watch waits as long as the task runs (no timeout by default; bound it with `--timeout`). Exit codes match send: 0 success or input-required, 4 task failed/canceled/rejected, 1 anything else. ## [](#usage)Usage ```bash rpk ai agent a2a task watch [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai agent a2a task watch`. ```bash rpk ai agent a2a task watch financial-advisor TASK_ID rpk ai agent a2a task watch financial-advisor TASK_ID -o json # one JSON event per line ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --timeout | duration | abort after this long (default: wait forever). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 90: rpk ai agent a2a task **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a-task.md --- # rpk ai agent a2a task > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a task latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a-task page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a-task.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a-task.adoc description: Manage tasks created by agent-to-agent (A2A) conversations. Task IDs come from rpk ai agent a2a send replies. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Manage tasks created by agent-to-agent (A2A) conversations. Task IDs come from `rpk ai agent a2a send` replies. Use subcommands to get, watch, or cancel a task. ## [](#usage)Usage ```bash rpk ai agent a2a task [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 91: rpk ai agent a2a **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-a2a.md --- # rpk ai agent a2a > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent a2a latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-a2a page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-a2a.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-a2a.adoc description: Interact with an agent over the A2A (Agent-to-Agent) protocol. The AGENT argument is either a registry agent name (resolved to the agent's A2A endpoint via its runtime status) or a full A2A endpoint URL (anything starting with http:// or https://). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Interact with an agent over the A2A (Agent-to-Agent) protocol. The AGENT argument is either a registry agent name (resolved to the agent’s A2A endpoint via its runtime status) or a full A2A endpoint URL (anything starting with http:// or https://). Authentication: your environment’s bearer token is attached when the target is a registry agent or an explicit URL on the environment’s dataplane host. Explicit URLs on other hosts are called without credentials so your token never leaves the platform (a note on stderr says so when this happens). Output formats: table (`default`, human-readable) and `-o json` / `-o yaml`. Streams (`--stream`, task watch) emit one JSON event per line under both `json` and `yaml`. Exit codes: 0 success (including tasks waiting for more input), 4 task ended failed/canceled/rejected, 1 anything else. ## [](#usage)Usage ```bash rpk ai agent a2a [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai agent a2a`. Discover what an agent can do ```bash rpk ai agent a2a card financial-advisor ``` Ask a question (waits for the reply) ```bash rpk ai agent a2a send financial-advisor "How did tech stocks do today?" ``` Continue the same conversation ```bash rpk ai agent a2a send financial-advisor --context-id CTX "And yesterday?" ``` Stream the reply as it is produced ```bash rpk ai agent a2a send financial-advisor --stream "Summarize the market" ``` Inspect, watch, or cancel a long-running task ```bash rpk ai agent a2a task get financial-advisor TASK_ID rpk ai agent a2a task watch financial-advisor TASK_ID rpk ai agent a2a task cancel financial-advisor TASK_ID ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 92: rpk ai agent apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-apply.md --- # rpk ai agent apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-apply.adoc description: "Reconcile agents from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Reconcile agents from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. ## [](#usage)Usage ```bash rpk ai agent apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 93: rpk ai agent create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-create.md --- # rpk ai agent create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-create.adoc description: Create an agent. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Create an agent. ## [](#usage)Usage ```bash rpk ai agent create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --description | string | longer description of the agent’s purpose. | | --display-name | string | display name. | | --llm-provider | string | name of the LLM provider resource the agent uses. | | --max-iterations | int32 | maximum agent loop iterations per invocation. | | --mcp-server | strings | MCP server resource name the agent may call (repeatable). | | --model | string | model the agent runs (for example, claude-sonnet-4-6). | | --self-managed | bool | register a metadata-only self-managed agent (the platform won’t run it). | | --spec-file | string | path to a JSON/YAML ManagedAgentSpec; enables subagents/agent_card that scalar flags can’t express (mutually exclusive with --self-managed and the spec flags). | | --system-prompt | string | system prompt for the agent. | | --tag | stringToString | metadata tag key=value (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 94: rpk ai agent credential create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-credential-create.md --- # rpk ai agent credential create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent credential create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-credential-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-credential-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-credential-create.adoc description: Create a client ID and secret pair for an agent. The client secret is shown once and cannot be retrieved again. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Create a client ID and secret pair for an agent. The client secret is shown once and cannot be retrieved again. ## [](#usage)Usage ```bash rpk ai agent credential create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --description | string | human-readable description of the credential. | | --ttl | duration | lifetime before the credential expires (for example, 720h); 0 means no expiry. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 95: rpk ai agent credential delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-credential-delete.md --- # rpk ai agent credential delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent credential delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-credential-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-credential-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-credential-delete.adoc description: Delete a credential. Specify the full resource name as shown by rpk ai agent credential list, for example agents/my-agent/credentials/abc123. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Delete a credential. Specify the full resource name as shown by `rpk ai agent credential list`, for example `agents/my-agent/credentials/abc123`. ## [](#usage)Usage ```bash rpk ai agent credential delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 96: rpk ai agent credential list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-credential-list.md --- # rpk ai agent credential list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent credential list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-credential-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-credential-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-credential-list.adoc description: List an agent's credentials. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- List an agent’s credentials. ## [](#usage)Usage ```bash rpk ai agent credential list [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 97: rpk ai agent credential **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-credential.md --- # rpk ai agent credential > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent credential latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-credential page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-credential.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-credential.adoc description: Manage an agent's credentials (create, list, delete). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Manage an agent’s credentials (`create`, list, delete). ## [](#usage)Usage ```bash rpk ai agent credential [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 98: rpk ai agent delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-delete.md --- # rpk ai agent delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-delete.adoc description: Delete an agent. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Delete an agent. ## [](#usage)Usage ```bash rpk ai agent delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 99: rpk ai agent diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-diff.md --- # rpk ai agent diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-diff.adoc description: Dry-run of apply for agents. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Dry-run of apply for agents. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. ## [](#usage)Usage ```bash rpk ai agent diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 100: rpk ai agent get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-get.md --- # rpk ai agent get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-get.adoc description: Get an agent. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Get an agent. ## [](#usage)Usage ```bash rpk ai agent get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 101: rpk ai agent list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-list.md --- # rpk ai agent list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-list.adoc description: List agents. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- List agents. ## [](#usage)Usage ```bash rpk ai agent list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --filter | string | AIP-160 filter expression, for example, name:`web` or display_name=prod. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 102: rpk ai agent start **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-start.md --- # rpk ai agent start > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent start latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-start page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-start.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-start.adoc description: Start a managed agent, setting its desired state to running. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Start a managed agent, setting its desired state to running. ## [](#usage)Usage ```bash rpk ai agent start [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 103: rpk ai agent stop **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-stop.md --- # rpk ai agent stop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent stop latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-stop page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-stop.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-stop.adoc description: Stop a managed agent, setting its desired state to stopped. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Stop a managed agent, setting its desired state to stopped. ## [](#usage)Usage ```bash rpk ai agent stop [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 104: rpk ai agent transcript get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-transcript-get.md --- # rpk ai agent transcript get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent transcript get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-transcript-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-transcript-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-transcript-get.adoc description: Get a single conversation transcript with its turns. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Get a single conversation transcript with its turns. ## [](#usage)Usage ```bash rpk ai agent transcript get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 105: rpk ai agent transcript list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-transcript-list.md --- # rpk ai agent transcript list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent transcript list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-transcript-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-transcript-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-transcript-list.adoc description: List an agent's conversation transcripts. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- List an agent’s conversation transcripts. ## [](#usage)Usage ```bash rpk ai agent transcript list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --errors-only | bool | only conversations that contain errors. | | --query | string | free-text search across conversation titles and content. | | --since | string | only conversations started at/after this time (RFC3339, or a duration ago like 24h). | | --status | string | filter by status: running, completed, error. | | --until | string | only conversations started before this time (RFC3339, or a duration ago like 1h). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 106: rpk ai agent transcript **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-transcript.md --- # rpk ai agent transcript > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent transcript latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-transcript page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-transcript.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-transcript.adoc description: Inspect an agent's conversation transcripts (list, get). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Inspect an agent’s conversation transcripts (`list`, get). ## [](#usage)Usage ```bash rpk ai agent transcript [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 107: rpk ai agent update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent-update.md --- # rpk ai agent update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent-update.adoc description: Update an agent. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Update an agent. ## [](#usage)Usage ```bash rpk ai agent update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --description | string | new description. | | --display-name | string | new display name. | | --llm-provider | string | name of the LLM provider resource the agent uses. | | --max-iterations | int32 | maximum agent loop iterations per invocation. | | --mcp-server | strings | MCP server resource name the agent may call (repeatable). | | --model | string | model the agent runs (for example, claude-sonnet-4-6). | | --system-prompt | string | system prompt for the agent. | | --tag | stringToString | replacement metadata tags key=value (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 108: rpk ai agent **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-agent.md --- # rpk ai agent > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai agent latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-agent page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-agent.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-agent.adoc description: Manage agents registered with the Redpanda AI platform. Agents are either managed (adp runs them; configured via a model, LLM provider, system prompt and MCP servers) or self-managed (a metadata-only record tracking a user-hosted agent). page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Manage agents registered with the Redpanda AI platform. Agents are either managed (adp runs them; configured via a model, LLM provider, system prompt and MCP servers) or self-managed (a metadata-only record tracking a user-hosted agent). `create` makes a managed agent by default; pass `--self-managed` for the metadata-only variant. The `credential` subcommand provisions client-id/secret pairs an agent uses to authenticate against the gateway. The `a2a` subcommand talks to a running agent over the A2A protocol (fetch its card, send messages, manage tasks). ## [](#usage)Usage ```bash rpk ai agent [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 109: rpk ai auth login **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-auth-login.md --- # rpk ai auth login > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai auth login latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-auth-login page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-auth-login.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-auth-login.adoc description: Run the OAuth 2.0 device authorization grant against Redpanda Cloud, persist the resulting credentials, and prompt to select an environment whose AI Gateway URL becomes the active profile's dataplane URL. Credentials are stored in the rpk ai credentials file with 0600 permissions, keyed by organization. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Run the OAuth 2.0 device authorization grant against Redpanda Cloud, persist the resulting credentials, and prompt to select an environment whose AI Gateway URL becomes the active profile’s dataplane URL. Credentials are stored in the `rpk ai` credentials file with `0600` permissions, keyed by organization. A successful login rewrites the config in the current schema, so running `rpk ai auth login` is also how you migrate a config left over from an older version of the plugin. If the current environment’s organization still holds valid credentials, login is a no-op and prints "Already logged in". Run `rpk ai auth logout` first to re-authenticate, or `rpk ai auth logout --all` for a full clean slate. ## [](#usage)Usage ```bash rpk ai auth login [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --no-browser | bool | do not attempt to open the browser; print the URL only. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 110: rpk ai auth logout **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-auth-logout.md --- # rpk ai auth logout > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai auth logout latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-auth-logout page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-auth-logout.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-auth-logout.adoc description: Delete stored credentials for the current organization (or --all for a full clean slate). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Delete stored credentials for the current organization (or `--all` for a full clean slate). ## [](#usage)Usage ```bash rpk ai auth logout [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --all | bool | full clean slate: remove ALL credentials AND the config file (including manual environments). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 111: rpk ai auth status **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-auth-status.md --- # rpk ai auth status > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai auth status latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-auth-status page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-auth-status.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-auth-status.adoc description: Show the authentication state for the current environment. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Show the authentication state for the current environment. ## [](#usage)Usage ```bash rpk ai auth status [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -o, --format | string | output format: table|`json`|`yaml`|markdown (overrides root --format). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 112: rpk ai auth token **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-auth-token.md --- # rpk ai auth token > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai auth token latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-auth-token page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-auth-token.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-auth-token.adoc description: Print the current bearer access token to stdout (refreshes if expired). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Print the current bearer access token to stdout (refreshes if expired). ## [](#usage)Usage ```bash rpk ai auth token [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --refresh-within | duration | proactively refresh if the token expires within this window (for polling callers like a Codex/Claude auth helper); 0 uses the default 60s leeway. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 113: rpk ai auth **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-auth.md --- # rpk ai auth > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai auth latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-auth page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-auth.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-auth.adoc description: Manage rpk ai authentication (login, logout, token, status). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Manage `rpk` ai authentication (`login`, logout, token, status). ## [](#usage)Usage ```bash rpk ai auth [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 114: rpk ai connection list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-connection-list.md --- # rpk ai connection list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai connection list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-connection-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-connection-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-connection-list.adoc description: List your OAuth connections. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- List your OAuth connections. ## [](#usage)Usage ```bash rpk ai connection list [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 115: rpk ai connection revoke **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-connection-revoke.md --- # rpk ai connection revoke > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai connection revoke latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-connection-revoke page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-connection-revoke.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-connection-revoke.adoc description: "Revoke your connection to the named OAuth provider: the stored tokens are invalidated and the provider's revocation endpoint is called best-effort. This affects only your own connection, not other users'." page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Revoke your connection to the named OAuth provider: the stored tokens are invalidated and the provider’s revocation endpoint is called best-effort. This affects only your own connection, not other users'. ## [](#usage)Usage ```bash rpk ai connection revoke [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 116: rpk ai connection **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-connection.md --- # rpk ai connection > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai connection latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-connection page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-connection.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-connection.adoc description: List and revoke the OAuth connections you hold to third-party providers. A connection is your personal grant to a provider (created by signing in through the consent flow) that lets user_oauth MCP servers act on your behalf. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- List and revoke the OAuth connections you hold to third-party providers. A connection is your personal grant to a provider (created by signing in through the consent flow) that lets user\_oauth MCP servers act on your behalf. Revoke is per user, per provider. ## [](#usage)Usage ```bash rpk ai connection [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 117: rpk ai env add **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-add.md --- # rpk ai env add > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env add latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-add page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-add.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-add.adoc description: Add a manual rpk ai environment with explicit URLs. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Add a manual `rpk` ai environment with explicit URLs. ## [](#usage)Usage ```bash rpk ai env add [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --adp-api-url | string | adp-api base URL (optional; derived from the AI Gateway URL when omitted). | | --ai-gateway-url | string | AI Gateway base URL (required). | | --auth-mode | string | auth mode: device|rpk|token|none (default device). device borrows your logged-in org’s token; use none/token for a local aigw. | | --description | string | optional human-readable description. | | --from | string | copy an existing manual environment as the base. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 118: rpk ai env delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-delete.md --- # rpk ai env delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-delete.adoc description: Delete a manual environment. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Delete a manual environment. ## [](#usage)Usage ```bash rpk ai env delete [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --force | bool | delete even if NAME is the current environment. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 119: rpk ai env list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-list.md --- # rpk ai env list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-list.adoc description: List environments (current org's live Agentic Data Plane envs + local manual envs). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-07" --- List environments (current org’s live Agentic Data Plane envs + local manual envs). ## [](#usage)Usage ```bash rpk ai env list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --local | bool | skip the live Agentic Data Plane environments lookup; show only local manual environments. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 120: rpk ai env rename **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-rename.md --- # rpk ai env rename > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env rename latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-rename page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-rename.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-rename.adoc description: Rename a manual environment. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Rename a manual environment. ## [](#usage)Usage ```bash rpk ai env rename [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 121: rpk ai env show **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-show.md --- # rpk ai env show > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env show latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-show page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-show.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-show.adoc description: Show the effective resolved environment as YAML (tokens redacted). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Show the effective resolved environment as YAML (tokens redacted). ## [](#usage)Usage ```bash rpk ai env show [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 122: rpk ai env use **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env-use.md --- # rpk ai env use > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env use latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env-use page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env-use.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env-use.adoc description: Switch to a manual environment, or select an Agentic Data Plane environment by name or id. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-07" --- Switch to a manual environment, or select an Agentic Data Plane environment by name or id. ## [](#usage)Usage ```bash rpk ai env use [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 123: rpk ai env **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-env.md --- # rpk ai env > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai env latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-env page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-env.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-env.adoc description: Manage rpk ai environments (list, use, add, show, rename, delete). page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Manage `rpk` ai environments (`list`, use, add, show, rename, delete). ## [](#usage)Usage ```bash rpk ai env [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 124: rpk ai install **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-install.md --- # rpk ai install > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai install latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-install page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-install.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-install.adoc description: Install the Redpanda AI CLI. This command installs the latest version by default. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- Install the Redpanda AI CLI. This command installs the latest version by default. Alternatively, you may specify an `rpk ai version` using the `--ai-version` flag. You may force the installation using the `--force` flag. ## [](#usage)Usage ```bash rpk ai install [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --ai-version | string | Redpanda AI CLI version to install (for example, 0.1.2). | | --force | bool | Force install of the Redpanda AI CLI. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 125: rpk ai llm-provider apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-apply.md --- # rpk ai llm-provider apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-apply.adoc description: "Reconcile LLM providers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Reconcile LLM providers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 126: rpk ai llm-provider check **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-check.md --- # rpk ai llm-provider check > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider check latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-check page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-check.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-check.adoc description: Runs a lightweight probe against the upstream to verify credentials and reachability. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Runs a lightweight probe against the upstream to verify credentials and reachability. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider check [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 127: rpk ai llm-provider create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-create.md --- # rpk ai llm-provider create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-create.adoc description: "Create an LLM provider. Choose the provider-config by setting flags from one group; the group you use selects it: openai-config, anthropic-config, google-config, bedrock-config, openai-compatible-config." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Create an LLM provider. Choose the provider-config by setting flags from one group; the group you use selects it: openai-config, anthropic-config, google-config, bedrock-config, openai-compatible-config. Setting flags from two groups is an error. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider create [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai llm-provider create`. ```bash rpk ai llm-provider create my-llm-provider \ --openai-config.api-key-ref OPENAI_API_KEY ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Human-readable name for display in the UI. | | --provider-models | stringArray | Model identifiers available through this provider. Leave empty to allow all models. (repeatable; name (bare values comma-split) or protojson object; replaces the full list on update). | | --enabled | bool | Whether this provider is active and accepting requests. (true when set; pass --enabled=false to disable). | | --openai-config.base-url | string | Override the default API endpoint. Leave empty for the standard OpenAI API. | | --openai-config.api-key-ref | string | Secret store reference for the API key. Leave empty for no-auth endpoints (Ollama, vLLM, local models). | | --anthropic-config.base-url | string | Override the default API endpoint. Leave empty for the standard Anthropic API. | | --anthropic-config.api-key-ref | string | Secret store reference for the API key. Leave empty when using authorization passthrough. | | --anthropic-config.authorization-passthrough | bool | Forward the client’s Authorization header to Anthropic instead of using a server-side API key. Used for enterprise/Max plan OAuth passthrough. (true when set; pass --anthropic-config.authorization-passthrough=false to disable). | | --google-config.base-url | string | Override the default API endpoint. Leave empty for the standard Google AI API. | | --google-config.api-key-ref | string | Secret store reference for the Google AI API key. | | --bedrock-config.region | string | AWS region where the Bedrock endpoint is deployed. (required in its group) (alias: --region). | | --bedrock-config.base-url | string | Override the default Bedrock endpoint URL. Leave empty for the standard regional endpoint. | | --bedrock-config.static-credentials.access-key-id-ref | string | Secret store reference for the AWS access key ID. (required in its group) (alias: --access-key-id-ref). | | --bedrock-config.static-credentials.secret-access-key-ref | string | Secret store reference for the AWS secret access key. (required in its group) (alias: --secret-access-key-ref). | | --bedrock-config.assume-role.role-arn | string | ARN of the IAM role to assume for Bedrock access. (required in its group) (alias: --role-arn). | | --bedrock-config.assume-role.external-id | string | External ID for cross-account role assumption. Required when the role’s trust policy mandates it. | | --bedrock-config.assume-role.session-name | string | Session name for CloudTrail audit logging. | | --openai-compatible-config.base-url | string | Override the default API endpoint. Leave empty for the standard OpenAI API. | | --openai-compatible-config.api-key-ref | string | Secret store reference for the API key. Leave empty for no-auth endpoints (Ollama, vLLM, local models). | | --transcripts.record-input-messages | bool | Capture full request bodies (prompts) on observability traces. (true when set; pass --transcripts.record-input-messages=false to disable). | | --transcripts.record-output-messages | bool | Capture full response bodies (completions) on observability traces. (true when set; pass --transcripts.record-output-messages=false to disable). | | --guardrail | string | Name of a Guardrail to attach to this provider. Leave empty for no guardrail. For Bedrock provider the guardrail uses AWS Bedrock Guardrail directly. For all other providers the proxy evaluates user input against the Bedrock Guardrail’s standalone backend. In both cases a guardrail intervention short-circuits the request. | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to create from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 128: rpk ai llm-provider delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-delete.md --- # rpk ai llm-provider delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-delete.adoc description: Delete an LLM provider. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Delete an LLM provider. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 129: rpk ai llm-provider diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-diff.md --- # rpk ai llm-provider diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-diff.adoc description: Dry-run of apply for LLM providers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Dry-run of apply for LLM providers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 130: rpk ai llm-provider get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-get.md --- # rpk ai llm-provider get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-get.adoc description: Get an LLM provider. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Get an LLM provider. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 131: rpk ai llm-provider list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-list.md --- # rpk ai llm-provider list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-list.adoc description: List LLM providers. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List LLM providers. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --name-contains | string | Name contains. | | --filter | string | AIP-160 filter expression (https://google.aip.dev/160). | | --page-size | int32 | Page size. | | --page-token | string | Page token. | | --order-by | string | AIP-132 order: a comma-separated list of fields, each optionally suffixed with " desc" (default ascending). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 132: rpk ai llm-provider update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider-update.md --- # rpk ai llm-provider update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider-update.adoc description: Update an LLM provider. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Update an LLM provider. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Human-readable name for display in the UI. | | --provider-models | stringArray | Model identifiers available through this provider. Leave empty to allow all models. (repeatable; name (bare values comma-split) or protojson object; replaces the full list on update). | | --enabled | bool | Whether this provider is active and accepting requests. (true when set; pass --enabled=false to disable). | | --openai-config.base-url | string | Override the default API endpoint. Leave empty for the standard OpenAI API. | | --openai-config.api-key-ref | string | Secret store reference for the API key. Leave empty for no-auth endpoints (Ollama, vLLM, local models). | | --anthropic-config.base-url | string | Override the default API endpoint. Leave empty for the standard Anthropic API. | | --anthropic-config.api-key-ref | string | Secret store reference for the API key. Leave empty when using authorization passthrough. | | --anthropic-config.authorization-passthrough | bool | Forward the client’s Authorization header to Anthropic instead of using a server-side API key. Used for enterprise/Max plan OAuth passthrough. (true when set; pass --anthropic-config.authorization-passthrough=false to disable). | | --google-config.base-url | string | Override the default API endpoint. Leave empty for the standard Google AI API. | | --google-config.api-key-ref | string | Secret store reference for the Google AI API key. | | --bedrock-config.region | string | AWS region where the Bedrock endpoint is deployed. (required in its group) (alias: --region). | | --bedrock-config.base-url | string | Override the default Bedrock endpoint URL. Leave empty for the standard regional endpoint. | | --bedrock-config.static-credentials.access-key-id-ref | string | Secret store reference for the AWS access key ID. (required in its group) (alias: --access-key-id-ref). | | --bedrock-config.static-credentials.secret-access-key-ref | string | Secret store reference for the AWS secret access key. (required in its group) (alias: --secret-access-key-ref). | | --bedrock-config.assume-role.role-arn | string | ARN of the IAM role to assume for Bedrock access. (required in its group) (alias: --role-arn). | | --bedrock-config.assume-role.external-id | string | External ID for cross-account role assumption. Required when the role’s trust policy mandates it. | | --bedrock-config.assume-role.session-name | string | Session name for CloudTrail audit logging. | | --openai-compatible-config.base-url | string | Override the default API endpoint. Leave empty for the standard OpenAI API. | | --openai-compatible-config.api-key-ref | string | Secret store reference for the API key. Leave empty for no-auth endpoints (Ollama, vLLM, local models). | | --transcripts.record-input-messages | bool | Capture full request bodies (prompts) on observability traces. (true when set; pass --transcripts.record-input-messages=false to disable). | | --transcripts.record-output-messages | bool | Capture full response bodies (completions) on observability traces. (true when set; pass --transcripts.record-output-messages=false to disable). | | --guardrail | string | Name of a Guardrail to attach to this provider. Leave empty for no guardrail. For Bedrock providers the guardrail rides the upstream call via AWS guardrail headers and is enforced by Bedrock itself. For all other providers the proxy evaluates user input against the guardrail’s standalone ApplyGuardrail backend before forwarding the request upstream. In both cases a guardrail intervention short-circuits the request. | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to update from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | | --clear | strings | field paths to clear (added to the update mask with the field unset), for example, --clear guardrail. | | --update-mask | strings | override the inferred update mask with exactly these paths. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 133: rpk ai llm-provider **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-llm-provider.md --- # rpk ai llm-provider > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai llm-provider latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-llm-provider page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-llm-provider.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-llm-provider.adoc description: Manage LLM providers (create, get, list, update, delete). page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Manage LLM providers (`create`, get, list, update, delete). > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai llm-provider [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 134: rpk ai mcp-server apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-apply.md --- # rpk ai mcp-server apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-apply.adoc description: "Reconcile MCP servers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Reconcile MCP servers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 135: rpk ai mcp-server create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-create.md --- # rpk ai mcp-server create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-create.adoc description: "Create an MCP server. Choose the backend by setting flags from one group; the group you use selects it: remote, managed." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Create an MCP server. Choose the backend by setting flags from one group; the group you use selects it: remote, managed. Setting flags from two groups is an error. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server create [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk ai mcp-server create`. ```bash rpk ai mcp-server create my-mcp-server \ --remote.url https://mcp.example.com/sse \ --remote.transport sse ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --enabled | bool | Whether this server is active and discoverable. (true when set; pass --enabled=false to disable). | | --description | string | Human-readable description shown in help text and UIs. | | --code-mode | bool | Enable code mode: adds {name}_search and {name}_execute tools alongside existing tools. (true when set; pass --code-mode=false to disable). | | --data-policies | stringArray | Data policies shaping this server’s tool calls (see MCPServer.data_policies). (repeatable; protojson object; replaces the full list on update). | | --response-format | string | Output format for tool results (token optimization). (one of: jton, toon). | | --tags | stringArray | Arbitrary key/value metadata tags (see MCPServer.tags). (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | --remote.url | string | Endpoint URL of the remote MCP server. (required in its group) (alias: --url). | | --remote.transport | string | MCP transport protocol. SSE for server-sent events, Streamable HTTP for newer bidirectional protocol. (one of: sse, streamable-http; required in its group) (alias: --transport). | | --remote.none | bool | None. (selects this auth mode; takes no value). | | --remote.token-passthrough | bool | Token passthrough. (selects this auth mode; takes no value). | | --remote.static-key.key-secret-ref | string | Secret store reference for the API key. (required in its group). | | --remote.static-key.header-name | string | HTTP header name for the API key. Defaults to 'Authorization'. | | --remote.service-account-oauth.client-id | string | OAuth client ID from the provider’s developer console. (required in its group). | | --remote.service-account-oauth.client-secret-ref | string | Secret store reference for the OAuth client secret. (required in its group). | | --remote.service-account-oauth.token-url | string | OAuth token endpoint URL for exchanging credentials. (required in its group). | | --remote.service-account-oauth.scopes | strings | OAuth scopes to request when obtaining tokens. (repeatable; replaces the full list on update). | | --remote.user-oauth.provider-name | string | OAuth provider that manages tokens for this server’s users. Leave empty to set OAuth up automatically from the server URL. | | --remote.user-oauth.required-scopes | strings | Minimum OAuth scopes a user’s connection must have. Insufficient scopes return a scope_upgrade_required error. (repeatable; replaces the full list on update). | | --remote.user-oauth.injection.header-name | string | HTTP header name for the token. Defaults to 'Authorization'. | | --remote.user-oauth.injection.header-prefix | string | Value prefix before the token. Defaults to 'Bearer'. Set empty for no prefix. | | --remote.user-oauth.client-id | string | OAuth client ID from the vendor console. Only for automatic setup of a server without dynamic client registration; leave empty otherwise. | | --remote.user-oauth.client-secret-ref | string | Secret-store reference for the OAuth client secret (confidential apps only). Leave empty for public PKCE clients. | | --remote.user-oauth.automatic-setup | bool | Ask the gateway to set OAuth up for this server instead of attaching an existing provider. (true when set; pass --remote.user-oauth.automatic-setup=false to disable). | | --managed.config | types | Type-specific configuration for the managed MCP server. (protojson with an "@type": a short type name (run the resource’s types command) or a full type URL). | | -f, --filename | string | manifest to create from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 136: rpk ai mcp-server delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-delete.md --- # rpk ai mcp-server delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-delete.adoc description: Delete an MCP server. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Delete an MCP server. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 137: rpk ai mcp-server diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-diff.md --- # rpk ai mcp-server diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-diff.adoc description: Dry-run of apply for MCP servers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Dry-run of apply for MCP servers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 138: rpk ai mcp-server get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-get.md --- # rpk ai mcp-server get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-get.adoc description: Get an MCP server. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Get an MCP server. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 139: rpk ai mcp-server list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-list.md --- # rpk ai mcp-server list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-list.adoc description: List MCP servers. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List MCP servers. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --name-contains | string | Name contains. | | --filter | string | AIP-160 filter expression (https://google.aip.dev/160). | | --page-size | int32 | Page size. | | --page-token | string | Page token. | | --order-by | string | AIP-132 order: a comma-separated list of fields, each optionally suffixed with " desc" (default ascending). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 140: rpk ai mcp-server tools call **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-tools-call.md --- # rpk ai mcp-server tools call > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server tools call latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-tools-call page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-tools-call.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-tools-call.adoc description: Invoke a tool on an MCP server through the aigw MCP proxy. The server's /mcp/v1/ endpoint is reached with the same bearer token used by the rest of rpk ai; aigw resolves user-delegated OAuth tokens from the vault when the MCP server is configured with --user-oauth-provider. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Invoke a tool on an MCP server through the aigw MCP proxy. The server’s /mcp/v1/ endpoint is reached with the same bearer token used by the rest of `rpk ai`; aigw resolves user-delegated OAuth tokens from the vault when the MCP server is configured with `--user-oauth-provider`. Arguments passed to the tool are a JSON object supplied via `--args`. Example: ```text rpk ai mcp tools call gf-servicenow-sand2 listtablerecords \ --args '{"tableName":"incident","sysparm_limit":3}' ``` With `--code-mode` the call targets the virtual code-mode sibling endpoint (/mcp/v1/-code). That endpoint exposes search (tool catalog for the primary) and execute (runs JavaScript in a sandbox with call\_tool bound to the primary’s tools). Example: ```text rpk ai mcp tools call pg-garrett execute --code-mode \ --args '{"code":"var r = call_tool({name:\"query\", arguments:{query:\"SELECT 1\"}}); JSON.stringify(r);"}' ``` > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server tools call [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --args | string | JSON object of tool arguments, for example, '{foo:`bar`}'. Empty means no arguments. | | --code-mode | bool | Target the virtual code-mode sibling endpoint (-code) instead of the primary. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 141: rpk ai mcp-server tools list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-tools-list.md --- # rpk ai mcp-server tools list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server tools list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-tools-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-tools-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-tools-list.adoc description: List tools on an MCP server by calling tools/list through the aigw MCP proxy. Hits the server's /mcp/v1/<name> endpoint, so the same auth and token-vault path used by tools/call is exercised here. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List tools on an MCP server by calling tools/list through the aigw MCP proxy. Hits the server’s /mcp/v1/ endpoint, so the same auth and token-vault path used by tools/call is exercised here. Useful for checking that a managed MCP’s tool schema loaded correctly and that a user-delegated server is reachable with the caller’s vault token. With `--code-mode` the session targets the virtual code-mode sibling endpoint (/mcp/v1/-code), which exposes the sandbox meta-tools (`search`, execute). To see the parent’s tool catalog from the sandbox, invoke the search tool explicitly: ```text rpk ai mcp tools call search --code-mode ``` > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server tools list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --code-mode | bool | Target the virtual code-mode sibling endpoint (-code) instead of the primary. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 142: rpk ai mcp-server tools **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-tools.md --- # rpk ai mcp-server tools > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server tools latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-tools page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-tools.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-tools.adoc description: Interact with tools on an MCP server. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Interact with tools on an MCP server. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server tools [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 143: rpk ai mcp-server types **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-types.md --- # rpk ai mcp-server types > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server types latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-types page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-types.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-types.adoc description: List available managed MCP types. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List available managed MCP types. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server types [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 144: rpk ai mcp-server update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server-update.md --- # rpk ai mcp-server update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server-update.adoc description: Update an MCP server. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Update an MCP server. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --enabled | bool | Whether this server is active and discoverable. (true when set; pass --enabled=false to disable). | | --description | string | Human-readable description shown in help text and UIs. | | --code-mode | bool | Enable code mode: adds {name}_search and {name}_execute tools alongside existing tools. (true when set; pass --code-mode=false to disable). | | --data-policies | stringArray | Data policies shaping this server’s tool calls (see MCPServer.data_policies). (repeatable; protojson object; replaces the full list on update). | | --response-format | string | Output format for tool results (token optimization). (one of: jton, toon). | | --tags | stringArray | Arbitrary key/value metadata tags (see MCPServer.tags). (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | --remote.url | string | Endpoint URL of the remote MCP server. (required in its group) (alias: --url). | | --remote.transport | string | MCP transport protocol. SSE for server-sent events, Streamable HTTP for newer bidirectional protocol. (one of: sse, streamable-http; required in its group) (alias: --transport). | | --remote.none | bool | None. (selects this auth mode; takes no value). | | --remote.token-passthrough | bool | Token passthrough. (selects this auth mode; takes no value). | | --remote.static-key.key-secret-ref | string | Secret store reference for the API key. (required in its group). | | --remote.static-key.header-name | string | HTTP header name for the API key. Defaults to 'Authorization'. | | --remote.service-account-oauth.client-id | string | OAuth client ID from the provider’s developer console. (required in its group). | | --remote.service-account-oauth.client-secret-ref | string | Secret store reference for the OAuth client secret. (required in its group). | | --remote.service-account-oauth.token-url | string | OAuth token endpoint URL for exchanging credentials. (required in its group). | | --remote.service-account-oauth.scopes | strings | OAuth scopes to request when obtaining tokens. (repeatable; replaces the full list on update). | | --remote.user-oauth.provider-name | string | OAuth provider that manages tokens for this server’s users. Leave empty to set OAuth up automatically from the server URL. | | --remote.user-oauth.required-scopes | strings | Minimum OAuth scopes a user’s connection must have. Insufficient scopes return a scope_upgrade_required error. (repeatable; replaces the full list on update). | | --remote.user-oauth.injection.header-name | string | HTTP header name for the token. Defaults to 'Authorization'. | | --remote.user-oauth.injection.header-prefix | string | Value prefix before the token. Defaults to 'Bearer'. Set empty for no prefix. | | --remote.user-oauth.client-id | string | OAuth client ID from the vendor console. Only for automatic setup of a server without dynamic client registration; leave empty otherwise. | | --remote.user-oauth.client-secret-ref | string | Secret-store reference for the OAuth client secret (confidential apps only). Leave empty for public PKCE clients. | | --remote.user-oauth.automatic-setup | bool | Ask the gateway to set OAuth up for this server instead of attaching an existing provider. (true when set; pass --remote.user-oauth.automatic-setup=false to disable). | | --managed.config | types | Type-specific configuration for the managed MCP server. (protojson with an "@type": a short type name (run the resource’s types command) or a full type URL). | | -f, --filename | string | manifest to update from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | | --clear | strings | field paths to clear (added to the update mask with the field unset), for example, --clear guardrail. | | --update-mask | strings | override the inferred update mask with exactly these paths. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 145: rpk ai mcp-server **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-mcp-server.md --- # rpk ai mcp-server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai mcp-server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-mcp-server page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-mcp-server.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-mcp-server.adoc description: Manage MCP servers (create, get, list, update, delete). page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Manage MCP servers (`create`, get, list, update, delete). > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai mcp-server [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 146: rpk ai model get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-model-get.md --- # rpk ai model get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai model get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-model-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-model-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-model-get.adoc description: Get model details from the catalog. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Get model details from the catalog. ## [](#usage)Usage ```bash rpk ai model get [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --provider-type | string | provider type (for disambiguation when a model is exposed by multiple providers). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 147: rpk ai model list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-model-list.md --- # rpk ai model list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai model list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-model-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-model-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-model-list.adoc description: List available models in the catalog. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- List available models in the catalog. ## [](#usage)Usage ```bash rpk ai model list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --provider-type | string | filter by provider type (openai, openai-compatible, anthropic, google, bedrock). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 148: rpk ai model **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-model.md --- # rpk ai model > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai model latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-model page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-model.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-model.adoc description: Discover the models the Redpanda AI gateway exposes. The catalog is read-only; model list and model get are the only verbs. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Discover the models the Redpanda AI gateway exposes. The catalog is read-only; `model list` and `model get` are the only verbs. The catalog is populated from each LLM provider’s metadata plus any extras the operator has pinned to a tenant. Aliases: `models`, `m`. The catalog is read-only. The catalog is populated from each LLM provider’s metadata plus any extras the operator has pinned to a tenant. ## [](#usage)Usage ```bash rpk ai model [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 149: rpk ai oauth-client apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-apply.md --- # rpk ai oauth-client apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-apply.adoc description: "Reconcile OAuth clients from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Reconcile OAuth clients from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. ## [](#usage)Usage ```bash rpk ai oauth-client apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 150: rpk ai oauth-client create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-create.md --- # rpk ai oauth-client create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-create.adoc description: Register an OAuth client with the AI gateway. The generated client secret is printed once and cannot be retrieved afterward. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Register an OAuth client with the AI gateway. The generated client secret is printed once and cannot be retrieved afterward. Save it immediately in a secret manager. ## [](#usage)Usage ```bash rpk ai oauth-client create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Shown on the user consent page. Use the external tool’s brand name. | | --logo-uri | string | Logo URL displayed alongside the client name on the consent page. HTTPS only. | | --redirect-uris | strings | Allowed redirect URIs (exact match, RFC 3986 normalized). No query or fragment components. (repeatable; replaces the full list on update). | | --allowed-resources | strings | MCP URLs this client may request tokens for. Use "*" to allow any MCP on this gateway. (repeatable; replaces the full list on update). | | --grant-types | strings | OAuth grant types this client may use. Authorization Code + Refresh Token is the standard combination. (one of: oauth-client-grant-type-authorization-code, oauth-client-grant-type-refresh-token; repeatable; replaces the full list on update). | | --token-endpoint-auth-method | string | How the client authenticates to the token endpoint. Phase 1 supports client_secret_basic and client_secret_post. (one of: oauth-client-token-endpoint-auth-method-client-secret-basic, oauth-client-token-endpoint-auth-method-client-secret-post, oauth-client-token-endpoint-auth-method-none). | | --pkce-required | bool | Require PKCE (RFC 7636) on authorization requests. Strongly recommended and enforced by default. (true when set; pass --pkce-required=false to disable). | | --enabled | bool | Whether this client may request new tokens. Disabled clients are rejected at the authorize endpoint. (true when set; pass --enabled=false to disable). | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to create from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 151: rpk ai oauth-client dcr get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-get.md --- # rpk ai oauth-client dcr get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-get.adoc description: Show the Dynamic Client Registration (DCR) settings for the current tenant, including whether DCR is enabled and the configured admission mode. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Show the Dynamic Client Registration (DCR) settings for the current tenant, including whether DCR is enabled and the configured admission mode. ## [](#usage)Usage ```bash rpk ai oauth-client dcr get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 152: rpk ai oauth-client dcr iat list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-list.md --- # rpk ai oauth-client dcr iat list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr iat list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-list.adoc description: List Initial Access Tokens (plaintext is never shown). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- List Initial Access Tokens (plaintext is never shown). ## [](#usage)Usage ```bash rpk ai oauth-client dcr iat list [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 153: rpk ai oauth-client dcr iat mint **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-mint.md --- # rpk ai oauth-client dcr iat mint > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr iat mint latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-mint page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-mint.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-mint.adoc description: Mint a one-shot Initial Access Token for use at the OAuth client registration endpoint. The token plaintext is printed once and only a hash is stored. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Mint a one-shot Initial Access Token for use at the OAuth client registration endpoint. The token plaintext is printed once and only a hash is stored. ## [](#usage)Usage ```bash rpk ai oauth-client dcr iat mint [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --label | string | human-readable label shown in the IAT list (recommended). | | --ttl | duration | validity window, for example, 1h, 24h, 168h (0 = server default of 1h; ceiling 720h). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 154: rpk ai oauth-client dcr iat revoke **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-revoke.md --- # rpk ai oauth-client dcr iat revoke > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr iat revoke latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-revoke page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-revoke.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat-revoke.adoc description: "Revoke an unconsumed Initial Access Token so it can no longer be exchanged at the registration endpoint. Idempotent: revoking an already-revoked or consumed token returns 0." page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Revoke an unconsumed Initial Access Token so it can no longer be exchanged at the registration endpoint. Idempotent: revoking an already-revoked or consumed token returns 0. ## [](#usage)Usage ```bash rpk ai oauth-client dcr iat revoke [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 155: rpk ai oauth-client dcr iat **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat.md --- # rpk ai oauth-client dcr iat > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr iat latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-iat.adoc description: "Manage Initial Access Tokens (IATs): one-shot bearer credentials a caller presents to the public registration endpoint when the tenant's admission mode is initial-access-token. The plaintext is printed exactly once on mint; only a hash is stored." page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Manage Initial Access Tokens (IATs): one-shot bearer credentials a caller presents to the public registration endpoint when the tenant’s admission mode is initial-access-token. The plaintext is printed exactly once on mint; only a hash is stored. ## [](#usage)Usage ```bash rpk ai oauth-client dcr iat [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 156: rpk ai oauth-client dcr update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr-update.md --- # rpk ai oauth-client dcr update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr-update.adoc description: Update the tenant's DCR settings. Only the flags you pass change; everything else keeps its current value (the CLI reads the current settings and writes back the merged result). page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Update the tenant’s DCR settings. Only the flags you pass change; everything else keeps its current value (the CLI reads the current settings and writes back the merged result). Enable open self-registration: ```text rpk ai oauth-client dcr update --enabled --admission-mode open ``` Require admin-minted Initial Access Tokens instead: ```text rpk ai oauth-client dcr update --admission-mode initial-access-token ``` Turn the endpoint off again: ```text rpk ai oauth-client dcr update --enabled=false ``` ## [](#usage)Usage ```bash rpk ai oauth-client dcr update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --admission-mode | string | how callers are admitted: open, initial-access-token. | | --allowed-resource | strings | MCP URL every DCR-issued client may request tokens for; "*" = any. repeatable. | | --client-cap | int32 | max concurrent DCR-issued clients (0 = runtime default). | | --enabled | bool | whether the public registration endpoint accepts requests. | | --inactive-ttl-days | int32 | days of inactivity before a DCR client is removed (0 = never). | | --rate-per-hour | int32 | max registrations per hour (0 = runtime default). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 157: rpk ai oauth-client dcr **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-dcr.md --- # rpk ai oauth-client dcr > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client dcr latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-dcr page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-dcr.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-dcr.adoc description: Manage Dynamic Client Registration (DCR) settings for the AI gateway. DCR allows OAuth clients to register themselves programmatically at a public endpoint. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Manage Dynamic Client Registration (DCR) settings for the AI gateway. DCR allows OAuth clients to register themselves programmatically at a public endpoint. When DCR is enabled, spec-conformant MCP clients (such as Claude and Cursor) self-register at the public `/oauth/idp/register` endpoint with no admin pre-provisioning. Admission is governed by the mode: - `open`: anyone may register (rate limit and client cap still apply) - `initial-access-token`: callers must present an admin-minted one-shot bearer token - `software-statement`: reserved, not yet supported DCR is disabled per tenant by default. The gateway operator must also enable the global `ingress.idp.dcr.global_enabled` flag. ## [](#usage)Usage ```bash rpk ai oauth-client dcr [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 158: rpk ai oauth-client delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-delete.md --- # rpk ai oauth-client delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-delete.adoc description: Delete an OAuth client. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Delete an OAuth client. ## [](#usage)Usage ```bash rpk ai oauth-client delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 159: rpk ai oauth-client diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-diff.md --- # rpk ai oauth-client diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-diff.adoc description: Dry-run of apply for OAuth clients. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Dry-run of apply for OAuth clients. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. ## [](#usage)Usage ```bash rpk ai oauth-client diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 160: rpk ai oauth-client get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-get.md --- # rpk ai oauth-client get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-get.adoc description: Get an OAuth client. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Get an OAuth client. ## [](#usage)Usage ```bash rpk ai oauth-client get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 161: rpk ai oauth-client list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-list.md --- # rpk ai oauth-client list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-list.adoc description: List OAuth clients. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- List OAuth clients. ## [](#usage)Usage ```bash rpk ai oauth-client list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --name-contains | string | Name contains. | | --filter | string | AIP-160 filter expression (https://google.aip.dev/160). | | --page-size | int32 | Page size. | | --page-token | string | Page token. | | --order-by | string | AIP-132 order: a comma-separated list of fields, each optionally suffixed with " desc" (default ascending). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 162: rpk ai oauth-client revoke-tokens **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-revoke-tokens.md --- # rpk ai oauth-client revoke-tokens > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client revoke-tokens latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-revoke-tokens page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-revoke-tokens.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-revoke-tokens.adoc description: Revoke every refresh token the AI gateway has issued for the named OAuth client. Forces all users who connected this client to sign in again. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Revoke every refresh token the AI gateway has issued for the named OAuth client. Forces all users who connected this client to sign in again. Already-issued short-lived access tokens may continue working until natural expiry (typically minutes). This command is idempotent: running it again returns 0. ## [](#usage)Usage ```bash rpk ai oauth-client revoke-tokens [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 163: rpk ai oauth-client update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client-update.md --- # rpk ai oauth-client update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client-update.adoc description: Update an OAuth client. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Update an OAuth client. > 📝 **NOTE** > > This command was introduced in ai version 0.2.28. ## [](#usage)Usage ```bash rpk ai oauth-client update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Shown on the user consent page. Use the external tool’s brand name. | | --logo-uri | string | Logo URL displayed alongside the client name on the consent page. HTTPS only. | | --redirect-uris | strings | Allowed redirect URIs (exact match, RFC 3986 normalized). No query or fragment components. (repeatable; replaces the full list on update). | | --allowed-resources | strings | MCP URLs this client may request tokens for. Use "*" to allow any MCP on this gateway. (repeatable; replaces the full list on update). | | --grant-types | strings | OAuth grant types this client may use. Authorization Code + Refresh Token is the standard combination. (one of: oauth-client-grant-type-authorization-code, oauth-client-grant-type-refresh-token; repeatable; replaces the full list on update). | | --token-endpoint-auth-method | string | How the client authenticates to the token endpoint. Phase 1 supports client_secret_basic and client_secret_post. (one of: oauth-client-token-endpoint-auth-method-client-secret-basic, oauth-client-token-endpoint-auth-method-client-secret-post, oauth-client-token-endpoint-auth-method-none). | | --pkce-required | bool | Require PKCE (RFC 7636) on authorization requests. Strongly recommended and enforced by default. (true when set; pass --pkce-required=false to disable). | | --enabled | bool | Whether this client may request new tokens. Disabled clients are rejected at the authorize endpoint. (true when set; pass --enabled=false to disable). | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to update from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | | --clear | strings | field paths to clear (added to the update mask with the field unset), for example, --clear guardrail. | | --update-mask | strings | override the inferred update mask with exactly these paths. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 164: rpk ai oauth-client **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-client.md --- # rpk ai oauth-client > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-client latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-client page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-client.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-client.adoc description: Manage OAuth clients registered with the AI gateway's OAuth Authorization Server. An OAuth client is an external tool (such as Claude AI, ChatGPT, or Cursor) that requests access tokens for an MCP server. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Manage OAuth clients registered with the AI gateway’s OAuth Authorization Server. An OAuth client is an external tool (such as Claude AI, ChatGPT, or Cursor) that requests access tokens for an MCP server. ## [](#usage)Usage ```bash rpk ai oauth-client [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 165: rpk ai oauth-provider apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-apply.md --- # rpk ai oauth-provider apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-apply.adoc description: "Reconcile OAuth providers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Reconcile OAuth providers from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. ## [](#usage)Usage ```bash rpk ai oauth-provider apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 166: rpk ai oauth-provider create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-create.md --- # rpk ai oauth-provider create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-create.adoc description: Create an OAuth provider. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Create an OAuth provider. ## [](#usage)Usage ```bash rpk ai oauth-provider create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Shown to users when selecting this provider. Use the provider’s brand name. | | --authorization-endpoint | string | The URL where users are redirected to grant consent. Found in your provider’s OAuth documentation. | | --token-endpoint | string | The URL where the gateway exchanges authorization codes for access tokens. | | --revocation-endpoint | string | Optional. Called when a user disconnects. Not all providers support token revocation. | | --client-id | string | OAuth client ID from the provider’s developer console. Omit when using register_from_url (it is discovered and registered automatically). | | --client-secret-ref | string | Reference to a secret stored in the secret store. Not required for public clients using PKCE. | | --scopes | strings | OAuth scopes determine what data and actions the token grants access to. Set to the union of all scopes needed across MCP servers to avoid re-consent. (repeatable; replaces the full list on update). | | --grant-types | strings | Select at least one grant type. Browser Consent is the standard OAuth 2.0 flow. Token Exchange enables server-to-server delegation. (one of: oauth-grant-type-browser-consent, oauth-grant-type-token-exchange; repeatable; replaces the full list on update). | | --pkce-required | bool | Proof Key for Code Exchange adds a challenge to the authorization flow, preventing code interception attacks. Recommended for all providers. (true when set; pass --pkce-required=false to disable). | | --token-endpoint-auth-method | string | How the gateway authenticates when exchanging codes for tokens. Check your provider’s documentation for the supported method. (one of: oauth-token-endpoint-auth-method-client-secret-basic, oauth-token-endpoint-auth-method-client-secret-post, oauth-token-endpoint-auth-method-none). | | --extra-auth-params | stringArray | Additional query parameters appended to the authorization URL. (repeatable key=value; replaces the full map on update). | | --extra-token-params | stringArray | Additional parameters sent in the token exchange request body. (repeatable key=value; replaces the full map on update). | | --enabled | bool | Whether this provider accepts new connections. (true when set; pass --enabled=false to disable). | | --register-from-url | string | Remote MCP server URL to auto-detect OAuth from. The gateway discovers endpoints and registers a client automatically. | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to create from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 167: rpk ai oauth-provider delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-delete.md --- # rpk ai oauth-provider delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-delete.adoc description: Delete an OAuth provider. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Delete an OAuth provider. ## [](#usage)Usage ```bash rpk ai oauth-provider delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 168: rpk ai oauth-provider diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-diff.md --- # rpk ai oauth-provider diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-diff.adoc description: Dry-run of apply for OAuth providers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Dry-run of apply for OAuth providers. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. ## [](#usage)Usage ```bash rpk ai oauth-provider diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 169: rpk ai oauth-provider get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-get.md --- # rpk ai oauth-provider get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-get.adoc description: Get an OAuth provider. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Get an OAuth provider. ## [](#usage)Usage ```bash rpk ai oauth-provider get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 170: rpk ai oauth-provider list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-list.md --- # rpk ai oauth-provider list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-list.adoc description: List OAuth providers. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- List OAuth providers. ## [](#usage)Usage ```bash rpk ai oauth-provider list [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --name-contains | string | Name contains. | | --filter | string | AIP-160 filter expression (https://google.aip.dev/160). | | --page-size | int32 | Page size. | | --page-token | string | Page token. | | --order-by | string | AIP-132 order: a comma-separated list of fields, each optionally suffixed with " desc" (default ascending). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 171: rpk ai oauth-provider update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider-update.md --- # rpk ai oauth-provider update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider-update.adoc description: Update an OAuth provider. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Update an OAuth provider. ## [](#usage)Usage ```bash rpk ai oauth-provider update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --display-name | string | Shown to users when selecting this provider. Use the provider’s brand name. | | --authorization-endpoint | string | The URL where users are redirected to grant consent. Found in your provider’s OAuth documentation. | | --token-endpoint | string | The URL where the gateway exchanges authorization codes for access tokens. | | --revocation-endpoint | string | Optional. Called when a user disconnects. Not all providers support token revocation. | | --client-secret-ref | string | Reference to a secret stored in the secret store. Not required for public clients using PKCE. | | --scopes | strings | OAuth scopes determine what data and actions the token grants access to. Set to the union of all scopes needed across MCP servers to avoid re-consent. (repeatable; replaces the full list on update). | | --grant-types | strings | Select at least one grant type. Browser Consent is the standard OAuth 2.0 flow. Token Exchange enables server-to-server delegation. (one of: oauth-grant-type-browser-consent, oauth-grant-type-token-exchange; repeatable; replaces the full list on update). | | --pkce-required | bool | Proof Key for Code Exchange adds a challenge to the authorization flow, preventing code interception attacks. Recommended for all providers. (true when set; pass --pkce-required=false to disable). | | --token-endpoint-auth-method | string | How the gateway authenticates when exchanging codes for tokens. Check your provider’s documentation for the supported method. (one of: oauth-token-endpoint-auth-method-client-secret-basic, oauth-token-endpoint-auth-method-client-secret-post, oauth-token-endpoint-auth-method-none). | | --extra-auth-params | stringArray | Additional query parameters appended to the authorization URL. (repeatable key=value; replaces the full map on update). | | --extra-token-params | stringArray | Additional parameters sent in the token exchange request body. (repeatable key=value; replaces the full map on update). | | --enabled | bool | Whether this provider accepts new connections. (true when set; pass --enabled=false to disable). | | --tags | stringArray | Tags. (repeatable key=value; replaces the full map on update). (New in 0.2.33) | | -f, --filename | string | manifest to update from (YAML or JSON; "-" for stdin); flags override file values. | | --dry-run | bool | print the request that would be sent (with the computed update mask) and exit without calling the server. | | --clear | strings | field paths to clear (added to the update mask with the field unset), for example, --clear guardrail. | | --update-mask | strings | override the inferred update mask with exactly these paths. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 172: rpk ai oauth-provider **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-oauth-provider.md --- # rpk ai oauth-provider > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai oauth-provider latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-oauth-provider page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-oauth-provider.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-oauth-provider.adoc description: Manage OAuth authorization-server configurations registered with the Redpanda AI gateway. OAuth providers describe third-party authorization servers that user-facing MCP servers can authenticate against via the device-consent flow. page-git-created-date: "2026-07-02" page-git-modified-date: "2026-08-06" --- Manage OAuth authorization-server configurations registered with the Redpanda AI gateway. OAuth providers describe third-party authorization servers that user-facing MCP servers can authenticate against via the device-consent flow. ## [](#usage)Usage ```bash rpk ai oauth-provider [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 173: rpk ai policy apply **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-apply.md --- # rpk ai policy apply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy apply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-apply page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-apply.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-apply.adoc description: "Reconcile policies from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Reconcile policies from one or more YAML manifests. For each manifest: create the resource if absent, otherwise update only the fields that are present in the manifest AND differ from the live resource. Fields you omit are left untouched; to clear a field, write it explicitly. Lists, maps and oneof variants replace wholesale. Fields that can only be set at creation time are immutable; changing one is an error. Manifests round-trip with `get -o yaml` for this resource. Pass `-f` - to read stdin. This does not delete resources absent from the manifests (no prune), and drift is detected only for the fields a manifest names; see `diff --help`. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy apply [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 174: rpk ai policy create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-create.md --- # rpk ai policy create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-create.adoc description: "Create a Cedar authorization policy: the allow/deny gate that decides WHETHER a principal may call a tool." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Create a Cedar authorization policy: the allow/deny gate that decides WHETHER a principal may call a tool. The Cedar body must contain exactly one statement; scope the resource to your MCP server, for example: ```text permit(principal, action == Action::"dataplane_adp_mcpserver_tools_call", resource == McpServer::"servicenow"); ``` Data shaping (`masking`, dropping, row filtering) is NOT configured here. It lives on the MCP server’s data policies (`rpk` ai mcp …​), not in Cedar text; the @redact\_mask / @redact\_drop annotations are gone. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --cedar | string | inline Cedar policy text. Mutually exclusive with --cedar-file. | | --cedar-file | string | path to a file containing the Cedar policy text (.cedar). Mutually exclusive with --cedar. | | --description | string | human-readable description. | | --display-name | string | human-readable display name. | | --name | string | policy id (required; lowercase letters, numbers, hyphens). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 175: rpk ai policy delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-delete.md --- # rpk ai policy delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-delete.adoc description: Delete a policy. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Delete a policy. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy delete [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --etag | string | optional etag for optimistic concurrency; delete fails if it no longer matches. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 176: rpk ai policy diff **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-diff.md --- # rpk ai policy diff > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy diff latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-diff page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-diff.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-diff.adoc description: Dry-run of apply for policies. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Dry-run of apply for policies. Prints, per manifest, whether apply would create, update (and which fields), or leave the resource unchanged. Exits non-zero when any change is pending, so CI can gate on `no drift`. > 📝 **NOTE** > > diff proves only that the fields a manifest names match live. It does not detect resources that exist live but are absent from the manifests (no prune), nor drift in fields a manifest omits. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy diff [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -f, --filename | strings | manifest file, directory, or - for stdin (repeatable). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 177: rpk ai policy get **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-get.md --- # rpk ai policy get > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy get latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-get page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-get.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-get.adoc description: Get a policy. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Get a policy. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy get [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 178: rpk ai policy list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-list.md --- # rpk ai policy list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-list.adoc description: List policies. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List policies. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy list [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 179: rpk ai policy update **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy-update.md --- # rpk ai policy update > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy update latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy-update page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy-update.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy-update.adoc description: Update a policy in place. Only the flags you pass are written; the field mask is built from the flags actually set, so omitted fields are left untouched. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Update a policy in place. Only the flags you pass are written; the field mask is built from the flags actually set, so omitted fields are left untouched. > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy update [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --cedar | string | inline Cedar policy text. Mutually exclusive with --cedar-file. | | --cedar-file | string | path to a file containing the Cedar policy text (.cedar). Mutually exclusive with --cedar. | | --description | string | new description. | | --display-name | string | new display name. | | --etag | string | optional etag for optimistic concurrency. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 180: rpk ai policy **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-policy.md --- # rpk ai policy > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai policy latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-policy page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-policy.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-policy.adoc description: Manage Cedar authorization policies (create, get, list, update, delete). page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Manage Cedar authorization policies (`create`, get, list, update, delete). > 📝 **NOTE** > > This command was introduced in ai version 0.2.26. ## [](#usage)Usage ```bash rpk ai policy [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 181: rpk ai run claude **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-run-claude.md --- # rpk ai run claude > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai run claude latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-run-claude page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-run-claude.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-run-claude.adoc description: Launch Anthropic's Claude Code CLI with its model traffic routed through the Redpanda AI gateway for the current rpk ai environment. rpk ai points ANTHROPIC_BASE_URL at the gateway's Anthropic Messages endpoint for the chosen provider and wires the gateway auth for the life of the session. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Launch Anthropic’s Claude Code CLI with its model traffic routed through the Redpanda AI gateway for the current `rpk` ai environment. `rpk` ai points ANTHROPIC\_BASE\_URL at the gateway’s Anthropic Messages endpoint for the chosen provider and wires the gateway auth for the life of the session. No token is ever written to disk. Both auth modes run in your REAL Claude Code config home, so your workspace trust, onboarding, theme, and MCP servers all apply. `rpk` ai writes nothing into `~/.claude` in either mode: - managed (api key): the gateway apiKeyHelper (`rpk ai auth token`, which Claude Code re-runs to refresh the bearer; aigw injects the upstream Anthropic key) is passed via `claude --settings` as a JSON string; off disk, merged on top of your settings, your `~/.claude/settings.json` untouched. - passthrough (enterprise/Max subscription): your existing subscription login (stored under your config home) is used. `rpk` ai only sets the gateway base URL and the X-Redpanda-Cloud-Token header (minted fresh at launch) in the environment; your subscription OAuth flows through aigw to Anthropic untouched. Pass `--claude-config-dir` to run against an isolated config home instead of your real one (`rpk` ai still never writes into it). In passthrough mode the X-Redpanda-Cloud-Token gateway JWT is set in the launched process environment, so Claude Code’s tool subprocesses (Bash, hooks, MCP) inherit it; the same Redpanda Cloud token any process running as you can already mint with `rpk ai auth token`, and Claude Code has no documented mechanism to scrub it from those subprocesses. Because passthrough uses your real config home, any auth configured in your `~/.claude/settings.json` (an apiKeyHelper, or env.ANTHROPIC\_AUTH\_TOKEN) still applies and outranks the subscription OAuth aigw needs to relay; `rpk` ai scrubs only the inherited shell env, not your on-disk settings. Anthropic and Bedrock providers are supported. A bedrock provider launches Claude Code in its native Bedrock mode pointed at the same gateway prefix; aigw signs the upstream call with the provider’s AWS credentials (SigV4), so no AWS keys ever reach your machine and the managed apiKeyHelper auth works exactly as above (passthrough does not apply; there is no Bedrock analog of a Claude subscription). Pass `-m` an inference-profile id from the provider’s model allowlist (Bedrock Anthropic models 4.6+ carry a us./eu./apac./global. prefix). Claude Code’s background (small/fast) model defaults to a Haiku-class inference profile in Bedrock mode; if the provider\`s allowlist doesn\`t include it, export ANTHROPIC\_SMALL\_FAST\_MODEL with an allowlisted id. Pass Claude Code’s own flags after a literal --: ```text rpk ai run claude -L anthropic -m claude-sonnet-4-6 -- --permission-mode plan rpk ai run claude -L bedrock -m us.anthropic.claude-sonnet-4-6 -- -p "hi" ``` ## [](#usage)Usage ```bash rpk ai run claude [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --bedrock | bool | force bedrock mode. Only needed when you have invoke-only access and rpk ai can’t read the provider to detect its type; when the provider can be read, its own type wins. | | --claude-config-dir | string | run against this CLAUDE_CONFIG_DIR instead of your real/inherited config home (rpk ai never writes into it; the managed apiKeyHelper rides in --settings). Default: your real home, so trust/onboarding/theme/MCP all apply. | | -L, --llmprovider | string | aigw LLM provider name to route through (an anthropic or bedrock provider). | | -m, --model | string | model id to use (must be in the provider’s allowlist; for bedrock providers an inference-profile id like us.anthropic.claude-sonnet-4-6); omit to let Claude Code pick its default. | | --passthrough | bool | force passthrough (enterprise/subscription) auth mode. Only needed when you have invoke-only access and rpk ai can’t read the provider to detect the mode; when the provider can be read, its own setting wins. | | --print-settings | bool | print the generated Claude Code settings.json plus launch env, then exit (no claude needed; the provider type/mode resolve via a best-effort gateway lookup; offline the dump falls back to the managed anthropic shape, or bedrock with --bedrock). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 182: rpk ai run codex **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-run-codex.md --- # rpk ai run codex > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai run codex latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-run-codex page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-run-codex.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-run-codex.adoc description: Launch the OpenAI Codex CLI with its model traffic routed through the Redpanda AI gateway for the current rpk ai environment. rpk ai generates a throwaway Codex config in a temporary CODEX_HOME, points it at the gateway's OpenAI-compatible Responses endpoint for the chosen provider, and wires Codex's bearer to rpai auth token so it refreshes itself for the life of the session. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Launch the OpenAI Codex CLI with its model traffic routed through the Redpanda AI gateway for the current `rpk` ai environment. `rpk` ai generates a throwaway Codex config in a temporary CODEX\_HOME, points it at the gateway’s OpenAI-compatible Responses endpoint for the chosen provider, and wires Codex’s bearer to `rpk ai auth token` so it refreshes itself for the life of the session. Your own `~/.codex` config is never read or modified, and no token is written to disk. The launch directory is auto-trusted under a workspace-write sandbox (approval\_policy=on-request) so the fresh CODEX\_HOME doesn’t prompt for trust on every run; pass `--no-auto-trust` to keep Codex’s normal first-run trust prompt. Only openai / openai\_compatible providers are supported (Codex speaks the OpenAI Responses API). Pass Codex’s own flags after a literal --: ```text rpk ai run codex -L openai -m gpt-5.3-codex -e high -- --ask-for-approval never ``` ## [](#usage)Usage ```bash rpk ai run codex [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --codex-home | string | persistent CODEX_HOME dir (default: a throwaway temp dir); its config.toml is regenerated each run and the dir is tightened to 0700; your real ~/.codex is refused. | | -e, --effort | string | model reasoning effort: minimal|low|medium|high (omit to let Codex use its default). | | -L, --llmprovider | string | aigw LLM provider name to route through (must be openai/openai_compatible). | | -m, --model | string | model id to use (must be in the provider’s allowlist); omit to let Codex pick its default. | | --no-auto-trust | bool | do not pre-trust the launch directory; let Codex show its normal first-run trust prompt instead (default: the launch dir is auto-trusted under a workspace-write sandbox). | | --print-config | bool | print the generated Codex config.toml and exit (no gateway call, no codex needed). | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 183: rpk ai run **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-run.md --- # rpk ai run > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai run latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-run page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-run.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-run.adoc description: Launch a third-party coding agent configured to send its model traffic through the Redpanda AI gateway for the current environment, reusing `rpk ai`'s login and auto-refreshing token. page-git-created-date: "2026-06-25" page-git-modified-date: "2026-08-06" --- Launch a third-party coding agent configured to send its model traffic through the Redpanda AI gateway for the current environment, reusing \`rpk ai’s login and auto-refreshing token. ## [](#usage)Usage ```bash rpk ai run [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 184: rpk ai uninstall **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-uninstall.md --- # rpk ai uninstall > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai uninstall latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-uninstall page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-uninstall.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-uninstall.adoc description: Uninstall the Redpanda AI CLI. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- Uninstall the Redpanda AI CLI. ## [](#usage)Usage ```bash rpk ai uninstall [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 185: rpk ai upgrade **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-upgrade.md --- # rpk ai upgrade > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai upgrade latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-upgrade page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-upgrade.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-upgrade.adoc description: Upgrade to the latest Redpanda AI CLI version. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- Upgrade to the latest Redpanda AI CLI version. ## [](#usage)Usage ```bash rpk ai upgrade [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --no-confirm | bool | Disable confirmation prompt for major version upgrades. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 186: rpk ai version **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai-version.md --- # rpk ai version > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai version latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai-version page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai-version.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai-version.adoc description: Print rpk ai version and commit. page-git-created-date: "2026-05-27" page-git-modified-date: "2026-08-06" --- Print `rpk ai version` and commit. ## [](#usage)Usage ```bash rpk ai version [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 187: rpk ai **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-ai/rpk-ai.md --- # rpk ai > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk ai latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-ai/rpk-ai page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-ai/rpk-ai.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-ai/rpk-ai.adoc description: Manage the Redpanda AI Gateway. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- > 📝 **NOTE** > > The `rpk ai` commands require an Agentic Data Plane cluster. For more information, see [Run agents on all your data, safely.](https://docs.redpanda.com/agentic-data-plane/home/). Manage the Redpanda AI Gateway. ## [](#usage)Usage ```bash rpk ai [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 188: rpk profile clear **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-clear.md --- # rpk profile clear > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile clear latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-clear page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-clear.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-clear.adoc description: Clear the current profile. This command clears the current profile, which can be useful to unset a production cluster profile. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Clear the current profile. This command clears the current profile, which can be useful to unset a production cluster profile. ## [](#usage)Usage ```bash rpk profile clear [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 189: rpk profile create **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-create.md --- # rpk profile create > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile create latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-create page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-create.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-create.adoc description: Create a new rpk profile with connection settings for a Redpanda cluster. There are multiple ways to create a profile. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Create a new `rpk` profile with connection settings for a Redpanda cluster. There are multiple ways to create a profile. A name must be provided if not using `--from-cloud` or `--from-rpk-container`. - You can use `--from-redpanda` to generate a new profile from an existing `redpanda.yaml` file. The special value `current` creates a profile from the current `redpanda.yaml` as it is loaded within `rpk`. - You can use `--from-rpk-container` to generate a profile from an existing cluster created using `rpk container start` command. The name is not needed when using this flag. - You can use `--from-profile` to generate a profile from an existing profile or from a profile in a `yaml` file. First, the filename is checked, then an existing profile name is checked. The special value `current` creates a new profile from the existing profile with any active environment variables or flags applied. - You can use `--from-cloud` to generate a profile from an existing cloud cluster ID. Note that you must be logged in with `rpk cloud login` first. The special value `prompt` will prompt to select a cloud cluster to create a profile for. - For serverless clusters that support both public and private networking, you will be prompted to select a network type unless you specify `--serverless-network`. To avoid prompts in automation, explicitly set `--serverless-network` to `public` or `private`. - You can use `--set key=value` to directly set fields. The key can either be the name of a `-X` flag or the path to the field in the profile’s YAML format. For example, using `--set tls.enabled=true` OR `--set kafka_api.tls.enabled=true` is equivalent. The `--set` flag is always applied last and can be used to set additional fields in tandem with `--from-redpanda` or `--from-cloud`. The `--set` flag supports autocompletion, suggesting the `-X` key format. If you begin writing a YAML path, the flag will suggest the rest of the path. It is recommended to always use the `--description` flag; the description is printed in the output of [`rpk profile list`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-list/). Once the command completes successfully, `rpk` switches to the newly created profile. ## [](#usage)Usage ```bash rpk profile create [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -d, --description | string | Optional description of the profile. | | --from-cloud | string | Create profile from Redpanda Cloud cluster. Automatically configures authentication and connection settings. | | --from-profile | string | Create and switch to a new profile from an existing profile or from a profile in a yaml file. | | --from-redpanda | string | Create and switch to a new profile from a redpanda.yaml file. | | --from-rpk-container | bool | Create and switch to a new profile generated from a running cluster created with rpk container. | | --serverless-network | string | Networking type for serverless clusters: public or private (if not specified, will prompt if both are available). | | -s, --set | stringArray | Set a profile configuration field. Format: key=value. Common fields: brokers, admin_api.addresses, tls.enabled. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | ## [](#suggested-reading)Suggested reading - [`rpk profile list`](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-list/) --- # Page 190: rpk profile current **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-current.md --- # rpk profile current > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile current latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-current page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-current.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-current.adoc description: Print the current profile name. This is a tiny command that simply prints the current profile name, which may be useful in scripts, or a PS1, or to confirm what you have selected. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Print the current profile name. This is a tiny command that simply prints the current profile name, which may be useful in scripts, or a PS1, or to confirm what you have selected. ## [](#usage)Usage ```bash rpk profile current [flags] ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | -n, --no-newline | bool | Do not print a newline after the profile name. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 191: rpk profile delete **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-delete.md --- # rpk profile delete > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile delete latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-delete page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-delete.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-delete.adoc description: Delete an rpk profile. Deleting a profile removes it from the rpk.yaml file. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Delete an rpk profile. Deleting a profile removes it from the `rpk.yaml` file. If the deleted profile was the selected profile, `rpk` will use in-memory defaults until a new profile is selected. ## [](#usage)Usage ```bash rpk profile delete [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 192: rpk profile edit-globals **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-edit-globals.md --- # rpk profile edit-globals > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile edit-globals latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-edit-globals page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-edit-globals.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-edit-globals.adoc description: Edit rpk globals. This command opens your default editor to edit the rpk global configurations. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Edit `rpk` globals. This command opens your default editor to edit the `rpk` global configurations. ## [](#usage)Usage ```bash rpk profile edit-globals [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 193: rpk profile edit **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-edit.md --- # rpk profile edit > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile edit latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-edit page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-edit.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-edit.adoc description: Edit an rpk profile. This command opens your default editor to edit the specified profile, or the current profile if no profile is specified. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Edit an rpk profile. This command opens your default editor to edit the specified profile, or the current profile if no profile is specified. If the profile does not exist, this command creates it and switches to it. The editor will display all available configuration fields. Fields that are not currently set are shown as comments with documentation. To set a field, uncomment it and provide a value. ## [](#usage)Usage ```bash rpk profile edit [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 194: rpk profile list **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-list.md --- # rpk profile list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-list page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-list.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-list.adoc description: List rpk profiles. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- List `rpk` profiles. ## [](#usage)Usage ```bash rpk profile list [flags] ``` ## [](#aliases)Aliases ```bash rpk profile ls ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --format | string | Output format (json,yaml,text,wide,help). (New in v26.2.1) | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 195: rpk profile print-globals **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-print-globals.md --- # rpk profile print-globals > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile print-globals latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-print-globals page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-print-globals.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-print-globals.adoc description: Print rpk global configuration. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Print `rpk` global configuration. ## [](#usage)Usage ```bash rpk profile print-globals [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 196: rpk profile print **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-print.md --- # rpk profile print > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile print latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-print page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-print.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-print.adoc description: Print rpk profile configuration. If no name is specified, this command prints the current profile as it exists in the rpk.yaml file. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Print `rpk` profile configuration. If no name is specified, this command prints the current profile as it exists in the `rpk.yaml` file. To print both the profile as it exists in the `rpk.yaml` file and the current profile as it is loaded in `rpk` with internal defaults, user-specified flags, and environment variables applied, use the `-v`/`--verbose` flag. ## [](#usage)Usage ```bash rpk profile print [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 197: rpk profile prompt **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-prompt.md --- # rpk profile prompt > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile prompt latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-prompt page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-prompt.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-prompt.adoc description: Prompt a profile name formatted for a PS1 prompt. This command prints ANSI-escaped text per your current profile's prompt field. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Prompt a profile name formatted for a PS1 prompt. This command prints ANSI-escaped text per your current profile’s `prompt` field. If the current profile does not have a prompt, this prints nothing. If the prompt is invalid, this exits 0 with no message. To validate the current prompt, use the `--validate` flag. This command may introduce other `%` variables in the future, if you want to print a `%` directly, use `%%` to escape it. To use this in zsh, be sure to add setopt PROMPT\_SUBST to your .zshrc. To edit your PS1, use something like PS1='$(`rpk profile prompt`)' in your shell rc file. ## [](#usage)Usage ```bash rpk profile prompt [flags] ``` ### [](#format)Format The `prompt` field supports space or comma separated modifiers and a quoted string that is be modified. Inside the string, the variable `%p` or `%n` refers to the profile name. As a few examples: ```text prompt: hi-white, bg-red, bold, "[%p]" prompt: hi-red "PROD" prompt: white, "dev-%n ``` If you want to have multiple formats, you can wrap each formatted section in parentheses. ```text prompt: ("--") (hi-white bg-red bold "[%p]") ``` ### [](#colors)Colors All ANSI colors are supported, with names matching the color name: `black`, `red`, `green`, `yellow`, `blue`, `magenta`, `cyan`, `white`. The `hi-` prefix indicates a high-intensity color: `hi-black`, `hi-red`, etc. The `bg-` prefix modifies the background color: `bg-black`, `bg-hi-red`, etc. ### [](#modifiers)Modifiers Four modifiers are supported, `bold`, `faint`, `underline`, and `invert`. ### [](#raw-mode)Raw mode The `raw` modifier disables ANSI color escapes entirely, outputting plain text. This is useful when your shell has issues with ANSI escape sequences affecting terminal width calculations. ```text prompt: raw, "%n" prompt: raw, "[%p]" ``` ## [](#flags)Flags | Value | Type | Description | | --- | --- | --- | | --validate | bool | Exit with an error message if the prompt is invalid. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 198: rpk profile rename-to **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-rename-to.md --- # rpk profile rename-to > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile rename-to latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-rename-to page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-rename-to.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-rename-to.adoc description: Rename the current rpk profile. This command renames the currently active profile to the specified name. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Rename the current `rpk profile`. This command renames the currently active profile to the specified name. To switch profiles first, use `rpk profile use`. ## [](#usage)Usage ```bash rpk profile rename-to [flags] ``` ## [](#aliases)Aliases ```bash rpk profile rename ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 199: rpk profile set-globals **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-set-globals.md --- # rpk profile set-globals > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile set-globals latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-set-globals page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-set-globals.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-set-globals.adoc description: Set rpk globals fields. This command takes a list of key=value pairs to write to the global config section of rpk.yaml. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Set `rpk` globals fields. This command takes a list of key=value pairs to write to the global config section of `rpk.yaml`. The globals section contains a set of settings that apply to all profiles and changes the way that `rpk` acts. For a list of global flags and what they mean, check `rpk -X help` and look for any key that begins with `globals`. This command supports autocompletion of valid keys. You can also use the format `set key value` if you intend to only set one key. ## [](#usage)Usage ```bash rpk profile set-globals [KEY=VALUE]+ [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 200: rpk profile set **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-set.md --- # rpk profile set > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile set latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-set page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-set.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-set.adoc description: Set fields in the current rpk profile. As in the create command, this command takes a list of key=value pairs to write to the current profile. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Set fields in the current `rpk profile`. As in the create command, this command takes a list of key=value pairs to write to the current profile. The key can either be the name of a `-X` flag or the path to the field in the profile’s `yaml` format. For example, using `--set` tls.enabled=true OR `--set` kafka\_api.tls.enabled=true is equivalent. The former corresponds to the `-X` flag tls.enabled, while the latter corresponds to the path kafka\_api.tls.enabled in the profile’s `yaml`. To see all available `-X` fields, run `rpk -X help`. This command supports autocompletion of valid keys, suggesting the `-X` key format. If you begin writing a YAML path, this command will suggest the rest of the path. You can also use the format `set key value` if you intend to only set one key. ## [](#usage)Usage ```bash rpk profile set [KEY=VALUE]+ [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | > ⚠️ **CAUTION** > > Profile files may contain sensitive information such as passwords or SASL credentials. Do not commit `rpk.yaml` files to version control systems like Git. --- # Page 201: rpk profile use **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-use.md --- # rpk profile use > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile use latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-use page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-use.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-use.adoc description: Switch to a different rpk profile, making it the active profile for subsequent rpk commands. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Switch to a different `rpk profile`, making it the active profile for subsequent `rpk` commands. ## [](#usage)Usage ```bash rpk profile use [flags] ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 202: rpk profile validate **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile-validate.md --- # rpk profile validate > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile validate latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile-validate page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile-validate.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile-validate.adoc description: Validate profile configuration and detect common issues. page-git-created-date: "2026-07-28" page-git-modified-date: "2026-08-06" --- Validate profile configuration and detect common issues. This command checks the current profile (or a specified profile) for: - `Cloud settings`: Whether from\_cloud matches the broker URLs - `Auth reference`: Whether the profile references a valid authentication - `Auth kind`: Whether the authentication type is correctly configured - `Auth token`: Whether the token exists and is not expired ## [](#usage)Usage ```bash rpk profile validate [flags] ``` ## [](#examples)Examples This section provides examples of how to use `rpk profile validate`. Validate the current profile: ```bash rpk profile validate ``` Validate a specific profile: ```bash rpk profile validate my-profile ``` ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 203: rpk profile **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile.md --- # rpk profile > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk profile latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-profile/rpk-profile page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-profile/rpk-profile.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-profile/rpk-profile.adoc description: Manage rpk profiles. An rpk profile talks to a single Redpanda cluster. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- Manage `rpk` profiles. An rpk profile talks to a single Redpanda cluster. You can create multiple profiles for multiple clusters and swap between them with `rpk profile use`. Multiple profiles may be useful if, for example, you use `rpk` to talk to a localhost cluster, a dev cluster, and a prod cluster, and you want to keep your configuration in one place. You can also use the RPK\_PROFILE environment variable to temporarily override the current profile without modifying your configuration. The `--profile` flag takes precedence over the environment variable if both are set. ## [](#usage)Usage ```bash rpk profile [flags] ``` ## [](#subcommands)Subcommands | Command | Description | | --- | --- | | rpk profile clear | Clear the current profile. This command clears the current profile, which can be useful to unset a production cluster profile. | | rpk profile create | Create a new rpk profile with connection settings for a Redpanda cluster. | | rpk profile current | Print the current profile name. This is a tiny command that simply prints the current profile name, which may be useful in scripts, or a PS1, or to confirm what you have selected. | | rpk profile delete | Delete an rpk profile. Deleting a profile removes it from the rpk.yaml file. | | rpk profile edit | Edit an rpk profile. This command opens your default editor to edit the specified profile, or the current profile if no profile is specified. | | rpk profile edit-globals | Edit rpk globals. This command opens your default editor to edit the rpk global configurations. | | rpk profile list | List rpk profiles. | | rpk profile print | Print rpk profile configuration. If no name is specified, this command prints the current profile as it exists in the rpk.yaml file. | | rpk profile print-globals | Print rpk global configuration. | | rpk profile prompt | Prompt a profile name formatted for a PS1 prompt. This command prints ANSI-escaped text per your current profile’s prompt field. | | rpk profile rename-to | Rename the current rpk profile. This command renames the currently active profile to the specified name. | | rpk profile set | Set fields in the current rpk profile. As in the create command, this command takes a list of key=value pairs to write to the current profile. | | rpk profile set-globals | Set rpk globals fields. This command takes a list of key=value pairs to write to the global config section of rpk.yaml. | | rpk profile use | Switch to a different rpk profile, making it the active profile for subsequent rpk commands. | | rpk profile validate | Validate profile configuration and detect common issues. | ## [](#global-flags)Global flags | Value | Type | Description | | --- | --- | --- | | --config | string | Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml. | | -X, --config-opt | stringArray | Override rpk configuration settings; -X help for detail or -X list for terser detail. | | --ignore-profile | bool | Ignore rpk.yaml and redpanda.yaml; use default settings. | | --profile | string | rpk profile to use. | | -v, --verbose | bool | Enable verbose logging. | --- # Page 204: rpk -X **URL**: https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-x-options.md --- # rpk -X > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [agentic-data-plane-full.txt](https://docs.redpanda.com/agentic-data-plane-full.txt) --- title: rpk -X latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: rpk/rpk-x-options page-component-name: agentic-data-plane page-version: master page-component-version: master page-component-title: Agentic Data Plane page-relative-src-path: rpk/rpk-x-options.adoc page-edit-url: https://github.com/redpanda-data/adp-docs/edit/main/modules/reference/pages/rpk/rpk-x-options.adoc description: Override any rpk configuration option for a single command with the -X flag, or set it persistently with an RPK_ environment variable. page-git-created-date: "2026-05-11" page-git-modified-date: "2026-08-06" --- Use the `-X` flag to override any rpk-specific configuration option for a single command, without modifying your rpk profile. Each option follows the form `key=value`: for example, `rpk -X tls.enabled=true` enables TLS for the Kafka API. Every `-X` option also has an environment-variable equivalent: prefix with `RPK_` and replace periods (`.`) with underscores (`_`). For example, `tls.enabled` becomes `RPK_TLS_ENABLED`. To list every available option, run: ```bash rpk -X list ``` Run `rpk -X help` for inline descriptions, or see the [full `rpk -X` reference](https://docs.redpanda.com/current/reference/rpk/rpk-x-options/) in the Redpanda docs. > 💡 **TIP** > > For persistent configuration across commands and sessions, prefer [rpk profiles](https://docs.redpanda.com/agentic-data-plane/reference/rpk/rpk-profile/rpk-profile/) over environment variables or `-X` flags. --- # Page 205: Manage Billing **URL**: https://docs.redpanda.com/cloud-data-platform/billing.md --- # Manage Billing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Billing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/index.adoc description: Learn about the metrics Redpanda uses to measure consumption and about subscriptions with committed use. page-git-created-date: "2024-06-06" page-git-modified-date: "2024-08-01" --- - [Billing and Support](billing/) Learn about the metrics Redpanda uses to measure consumption in Redpanda Cloud. - [Manage Payment Methods](manage-payment-methods/) Add a credit card, set a default payment method, and update billing contact information in Redpanda Cloud. - [View Billing Activity](view-billing-activity/) View charges, filter the resources breakdown, and export billing activity to CSV in Redpanda Cloud. - [Manage Billing Notifications](billing-notifications/) Manage billing notifications in Redpanda Cloud: what alerts you receive, who receives them, and how to configure your notification preferences. - AWS - [Use AWS Commitments](aws-commit/) Subscribe to Redpanda in AWS Marketplace with committed use. - [Use AWS Pay As You Go](aws-pay-as-you-go/) Subscribe to Redpanda in AWS Marketplace with pay-as-you-go billing, and cancel anytime. - Azure - [Use Azure Commitments](azure-commit/) Subscribe to Redpanda in Azure Marketplace with committed use. - GCP - [Use GCP Commitments](gcp-commit/) Subscribe to Redpanda in Google Cloud Marketplace with committed use. - [Use GCP Pay As You Go](gcp-pay-as-you-go/) Subscribe to Redpanda in Google Cloud Marketplace with pay-as-you-go billing, and cancel anytime. --- # Page 206: Use AWS Commitments **URL**: https://docs.redpanda.com/cloud-data-platform/billing/aws-commit.md --- # Use AWS Commitments > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use AWS Commitments latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: aws-commit page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: aws-commit.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/aws-commit.adoc description: Subscribe to Redpanda in AWS Marketplace with committed use. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-10-17" --- You can subscribe to Redpanda Cloud through AWS Marketplace and use your existing marketplace billing and credits to quickly provision clusters. View your bills and manage your subscription directly in the marketplace. With a usage-based billing commitment, you sign up for a minimum spend amount. Commitments are minimums: - If you use less than your committed amount, you still pay the minimum. Any unused amount on a monthly commitment rolls over to the next month until the end of your term. - If you use more than your committed amount, you can continue using Redpanda Cloud without interruption. You’re charged for any additional usage until the end of your term. > ❗ **IMPORTANT** > > When you subscribe to Redpanda Cloud through AWS Marketplace, you can only create clusters on AWS. ## [](#sign-up-in-aws-marketplace)Sign up in AWS Marketplace 1. Contact [Redpanda Sales](https://redpanda.com/contact) to request a private offer with possible discounts. 2. You will receive a private offer on AWS Marketplace. Review the policy and required terms, and click **Accept**. > 📝 **NOTE** > > If you don’t have a billing account associated with your project, you’re prompted to enable billing to link the subscription with a billing account. You are taken to the Redpanda sign-up page. 3. On the Redpanda sign-up page: - For **Email**, enter your email address to register with Redpanda. - For **Organization name**, enter a name for your new organization connected through AWS Marketplace. Redpanda organizations contain all resources, including clusters and networks. - Click **Sign up and create organization**. You will receive an email sent to the address you entered. 4. In the email, click **Verify email address**. This completes the registration and associates the email with a Redpanda account. 5. On the **Accept your invitation to sign up** page, click **Sign up** or **Log in**. You can now create resource groups, clusters, and networks in your organization. ## [](#next-steps)Next steps - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#create-a-serverless-cluster) - [Create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/#create-a-dedicated-cluster) --- # Page 207: Use AWS Pay As You Go **URL**: https://docs.redpanda.com/cloud-data-platform/billing/aws-pay-as-you-go.md --- # Use AWS Pay As You Go > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use AWS Pay As You Go latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: aws-pay-as-you-go page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: aws-pay-as-you-go.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/aws-pay-as-you-go.adoc description: Subscribe to Redpanda in AWS Marketplace with pay-as-you-go billing, and cancel anytime. page-git-created-date: "2024-09-19" page-git-modified-date: "2026-05-05" --- Subscribe to Redpanda Cloud through AWS Marketplace to quickly provision Serverless and Dedicated clusters. With a usage-based pay-as-you-go subscription, you only pay for what you use and can cancel anytime. > ❗ **IMPORTANT** > > When you sign up for Redpanda Cloud through AWS Marketplace, you can only create clusters on AWS. ## [](#sign-up-in-aws-marketplace)Sign up in AWS Marketplace 1. In the AWS Marketplace, select [**Redpanda Cloud - The proven Apache Kafka alternative (Pay as You Go)**](https://aws.amazon.com/marketplace/pp/prodview-ecbu7wwsfh644?applicationId=AWSMPContessa&ref_=beagle&sr=0-3). 2. On the **Redpanda Cloud - Pay as You Go** overview page, click **View purchase options**, then click **Subscribe**. > 📝 **NOTE** > > If you don’t have a billing account associated with your project, you’re prompted to link the subscription with a billing account. 3. On the **Subscribe to Redpanda Cloud** page, click **Set up your account**. You’re taken to the Redpanda sign-up page. 4. On the Redpanda sign-up page: - For **Email**, enter your email address to register with Redpanda. - For **Organization name**, enter a name for your new organization connected through AWS Marketplace. > 💡 **TIP** > > This process creates a new organization, even for existing Redpanda customers. Organizations contain all resources, including clusters and networks. - Click **Sign up and create organization**. You will receive an email sent to the address you entered. 5. In the email, click **Verify email address**. This associates the email with a Redpanda account. 6. On the **Accept your invitation to sign up** page, enter the credentials you want to use for Redpanda Cloud. You can now create resource groups, networks, and clusters in your organization. ## [](#next-steps)Next steps - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#create-a-serverless-cluster) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/#create-a-dedicated-cluster) --- # Page 208: Use Azure Commitments **URL**: https://docs.redpanda.com/cloud-data-platform/billing/azure-commit.md --- # Use Azure Commitments > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use Azure Commitments latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: azure-commit page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: azure-commit.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/azure-commit.adoc description: Subscribe to Redpanda in Azure Marketplace with committed use. page-git-created-date: "2024-10-30" page-git-modified-date: "2025-10-17" --- You can subscribe to Redpanda Cloud through Azure Marketplace and use your existing marketplace billing and credits to quickly provision clusters. View your bills and manage your subscription directly in the marketplace. With a usage-based billing commitment, you sign up for a monthly or an annual minimum spend amount. Commitments are minimums: - If you use less than your committed amount, you still pay the minimum. Any unused amount on a monthly commitment rolls over to the next month until the end of your term. - If you use more than your committed amount, you can continue using Redpanda Cloud without interruption. You’re charged for any additional usage until the end of your term. > ❗ **IMPORTANT** > > When you subscribe to Redpanda Cloud through Azure Marketplace, you can only create clusters on Azure. ## [](#sign-up-in-azure-marketplace)Sign up in Azure Marketplace 1. Contact [Redpanda sales](https://redpanda.com/contact) to request a private offer with possible discounts. You will receive a private offer on Azure Marketplace. This offer is associated with an Azure user account that has access to the Azure subscription used for billing. 2. In Azure Marketplace, review the policy and required terms, and click **Accept**. You are taken to the Redpanda sign-up page. 3. On the Redpanda sign-up page: - For **Email**, enter your email address to register with Redpanda. - For **Organization name**, enter a name for your new organization connected through Azure Marketplace. Redpanda organizations contain all resources, including clusters and networks. - Click **Sign up and create organization**. You will receive an email sent to the address you entered. 4. In the email, click **Verify email address**. This completes the registration and associates the email with a Redpanda account. 5. On the **Accept your invitation to sign up** page, click **Sign up** or **Log in**. You can now create resource groups, clusters, and networks in your organization. ## [](#next-steps)Next steps - [Create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/) --- # Page 209: Manage Billing Notifications **URL**: https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications.md --- # Manage Billing Notifications > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Billing Notifications latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: billing-notifications page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: billing-notifications.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/billing-notifications.adoc description: "Manage billing notifications in Redpanda Cloud: what alerts you receive, who receives them, and how to configure your notification preferences." page-topic-type: how-to personas: platform_admin, evaluator learning-objective-1: Identify the billing notifications Redpanda Cloud sends and their thresholds learning-objective-2: Configure which users in your organization receive billing notifications learning-objective-3: Opt out of billing notification emails for yourself or your organization page-git-created-date: "2026-03-27" page-git-modified-date: "2026-04-07" --- Redpanda Cloud sends email notifications to help you monitor your billing balance. Organization admins receive alerts when credit or commit balances reach spending thresholds. In this guide, you will: - Identify the billing notifications Redpanda Cloud sends and their thresholds - Configure which users in your organization receive billing notifications - Opt out of billing notification emails for yourself or your organization ## [](#what-notifications-you-receive)What notifications you receive Redpanda Cloud monitors your balance and sends a notification when it crosses each threshold. Each threshold triggers one notification. If your balance crosses the same threshold again after adding credits, you may receive another notification at that level. | Notification | Description | Thresholds | | --- | --- | --- | | Low credit balance | Sent when your pre-paid credit balance is running low. Credits are drawn down by usage, similar to a prepaid account. | 50%, 30%, 10%, 0% remaining | | Low commit balance | Sent when your contractual commit balance is running low. Commits represent a minimum spend over a contract period. | 50%, 30%, 10%, 0% remaining | Notifications are sent to email only. The subject line follows this format: `Action Required: Your Redpanda Cloud is % remaining` ## [](#who-receives-notifications)Who receives notifications All users with the **Admin** role in your organization receive billing notifications by default. To change who receives notifications, update role assignments on the **Organization IAM** page. See [Role-Based Access Control](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/) or [Group-Based Access Control](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/gbac/). ## [](#opt-out-of-notifications)Opt out of notifications ### [](#individual-opt-out)Individual opt-out To stop receiving billing notification emails: - Open any billing notification email. - Click the **Unsubscribe** or **Manage notification preferences** link at the bottom of the email. No support ticket is needed. The change takes effect within 24-48 hours. ### [](#organization-wide-opt-out)Organization-wide opt-out To disable billing notifications for all admins in your organization, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). > 📝 **NOTE** > > If billing notifications are enabled for the organization, individual admins who have not unsubscribed will continue to receive notifications. ## [](#common-questions)Common questions - I didn’t sign up for these emails. Why am I receiving them? Billing notifications are sent automatically to all organization admins. If you don’t want to receive them, click the **Unsubscribe** link at the bottom of the email. - I got an alert but I already added credits. Why? Notifications are triggered when your balance crosses a threshold. If you added credits after the threshold was crossed, the notification was already queued. If your balance later crosses the same threshold again (for example, after adding credits and then using them), you may receive another notification. - Who else in my organization is getting these? All users with the Admin role receive billing notifications. To see who has the Admin role, check the **Organization IAM** > **Users** page in Redpanda Cloud. - I unsubscribed but still received a notification. What happened? Unsubscribe requests take 24-48 hours to process. If you receive a notification during that window, it was sent before your request was fully applied. - What should I do when I get an alert? Review your current balance on the **Billing** page. You can add credits or contact your Redpanda account team to discuss your usage and plan options. - Do trial accounts get notifications? Only if the trial has promotional credits. Standard trial accounts without a credit balance do not receive billing notifications. --- # Page 210: Billing and Support **URL**: https://docs.redpanda.com/cloud-data-platform/billing/billing.md --- # Billing and Support > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Billing and Support latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: billing page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: billing.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/billing.adoc description: Learn about the metrics Redpanda uses to measure consumption in Redpanda Cloud. page-topic-type: reference personas: platform_admin, evaluator page-git-created-date: "2024-06-06" page-git-modified-date: "2026-05-22" --- Redpanda Cloud uses various [metrics](#redpanda-streaming-billing-metrics) to measure the consumption of resources. - All pricing is set in US dollars (USD). - All usage-based billing computations are conducted in Coordinated Universal Time (UTC). Billing accrues at hourly intervals. Any usage that is less than an hour is billed for the full hour. - The **Billing** page shows detailed billing activity for your organization and lets you [manage payment methods](https://docs.redpanda.com/cloud-data-platform/billing/manage-payment-methods/). Redpanda charges the credit card marked as the default. > 📝 **NOTE** > > - Redpanda Cloud can notify you when your credit or commit balance is running low. See [Manage Billing Notifications](https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications/). > > - Pricing information is available on [redpanda.com](https://www.redpanda.com/price-estimator). For questions about billing, contact [billing@redpanda.com](mailto:billing@redpanda.com). ## [](#redpanda-streaming-billing-metrics)Redpanda Streaming billing metrics ### Serverless Pricing for Serverless clusters depends on the data in, data out, data stored, partitions (virtual streams), and the time the instance is up. The cost for each Serverless metric varies based on the region you select for your cluster. | Metric | Description | | --- | --- | | Uptime | Tracks the number of hours the instance is running.NOTE: Uptime is not charged if partitions = 0 and storage = 0. This condition is met when all topics are deleted. | | Ingress | Tracks the data written into Redpanda (in GB).All Kafka protocol requests (except message headers) are counted as ingress as soon as they are read by Redpanda’s proxy process. | | Egress | Tracks the data read out of Redpanda (in GB).All Kafka protocol responses generated by the cluster (except message headers) are counted as egress as soon as the cluster processes the request, even if the client drops the connection before they are delivered. | | Partitions | Tracks the number of partitions used per hour. | | Storage | Tracks the data in object storage per hour (in GB). | See also: [Serverless limits](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#serverless-usage-limits) ### Dedicated Pricing for Dedicated clusters depends on the time the instance is up, the data in, data out, and data stored. | Metric | Description | | --- | --- | | Uptime | Tracks the number of hours the instance is running.The cost varies based on the region and tier you select for your cluster. | | Ingress | Tracks the data written into Redpanda (in GB).All Kafka protocol requests (including message headers) are counted as ingress as soon as they are read by Redpanda’s proxy process.The cost varies based on the region you select for your cluster. | | Egress | Tracks the data read out of Redpanda (in GB).All Kafka protocol responses generated by the cluster (including message headers) are counted as egress as soon as the cluster processes the request, even if the client drops the connection before they are delivered.The cost varies based on the number of availability zones (AZ) you select for your cluster. | | Storage | Tracks the usage of object storage on an hourly basis during the billing period (in GB-hours).Replication to object storage is implemented with Tiered Storage. All topics have a fixed replication factor of 3, but Redpanda counts each byte only once. | ### BYOC Pricing for BYOC clusters depends on compute, data in, data out, and data stored. The rate decreases as usage increases. | Metric | Description | | --- | --- | | Compute | Tracks the server resources (vCPU and memory) a cluster uses on an hourly basis in Redpanda units (RPUs). Where:1 RPU = 2 vCPU + 8 GB memory | | Ingress | Tracks the data written into Redpanda (in GB).All Kafka protocol requests (including message headers) are counted as ingress as soon as they are read by Redpanda’s proxy process. | | Ingress to Iceberg topics | Tracks the data written to Iceberg tables per hour (in GB).NOTE: This metric applies only if you write to Iceberg topics. This charge is in addition to the standard ingress charge. | | Egress | Tracks the data read out of Redpanda (in GB).All Kafka protocol responses generated by the cluster (including message headers) are counted as egress as soon as the cluster processes the request, even if the client drops the connection before they are delivered.The cost varies based on the number of availability zones (AZ) you select for your cluster. | | Storage | Tracks the usage of object storage on an hourly basis during the billing period (in GB-hours).Replication to object storage is implemented with Tiered Storage. All topics have a fixed replication factor of 3, but Redpanda counts each byte only once. | ## [](#redpanda-sql-billing-metrics)Redpanda SQL billing metrics Pricing for Redpanda SQL depends on the provisioned compute resources allocated to the SQL workload, measured in RPUs. The cost of an RPU can vary based on the cloud provider and region you select. | Metric | Description | | --- | --- | | Compute | Tracks the server resources (vCPU and memory) Redpanda SQL uses on an hourly basis in Redpanda units (RPUs). Where:1 RPU = 2 vCPU + 8 GB memory | > 📝 **NOTE** > > Redpanda SQL uses the same RPU definition as Redpanda Streaming BYOC clusters, but the price per RPU differs. Contact your Redpanda account team for current pricing. ## [](#redpanda-connect-billing-metrics)Redpanda Connect billing metrics Pricing per pipeline depends on the compute units you allocate. The cost of a compute unit can vary based on the cloud provider and region you select for your cluster. | Metric | Description | | --- | --- | | Compute | Tracks the server resources (vCPU and memory) a pipeline uses in compute units per hour. Where:1 compute unit = 0.1 CPU + 400 MB memory | ## [](#support-plans)Support plans All organizations in Redpanda require one of the following support plans: | Support plan | Features | | --- | --- | | Basic | Designed for non-production environmentsProvides minimal support: priority 3 tickets within 8 business hours response time and priority 4 tickets with no target response timeSupport availability is 8:00 AM to 5:00 PM Pacific Time, Monday through Friday, excluding federal US holidays | | Enterprise | Designed for production environments needing continuous availabilityP1/P2 tickets may be submittedSupport availability is 24/7, including holidays | | Premium | Designed for mission-critical workloads30-minute response times for production outagesIncludes a named Customer Success Manager to support planning and coordination, and 10 hours per month of consulting from a Solutions ArchitectRequired for deployments with BYOVPC/BYOVnet clusters | ## [](#next-steps)Next steps - [Use AWS Commitments](https://docs.redpanda.com/cloud-data-platform/billing/aws-commit/) - [Use Azure Commitments](https://docs.redpanda.com/cloud-data-platform/billing/azure-commit/) - [Use GCP Commitments](https://docs.redpanda.com/cloud-data-platform/billing/gcp-commit/) - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#create-a-serverless-cluster) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/#create-a-dedicated-cluster) - [Create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) --- # Page 211: Use GCP Commitments **URL**: https://docs.redpanda.com/cloud-data-platform/billing/gcp-commit.md --- # Use GCP Commitments > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use GCP Commitments latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: gcp-commit page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: gcp-commit.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/gcp-commit.adoc description: Subscribe to Redpanda in Google Cloud Marketplace with committed use. page-git-created-date: "2024-06-06" page-git-modified-date: "2026-05-05" --- You can subscribe to Redpanda Cloud through Google Cloud Marketplace and use your existing marketplace billing and credits to quickly provision clusters. View your bills and manage your subscription directly in the marketplace. With a usage-based billing commitment, you sign up for a monthly or an annual minimum spend amount. Commitments are minimums: - If you use less than your committed amount, you still pay the minimum. Any unused amount on a monthly commitment rolls over to the next month until the end of your term. - If you use more than your committed amount, you can continue using Redpanda Cloud without interruption. You’re charged for any additional usage until the end of your term. > ❗ **IMPORTANT** > > When you subscribe to Redpanda Cloud through Google Cloud Marketplace, you can only create clusters on GCP. ## [](#sign-up-in-google-cloud-marketplace)Sign up in Google Cloud Marketplace 1. Contact [Redpanda sales](https://redpanda.com/contact) to request a private offer with possible discounts. 2. You will receive a private offer on Google Cloud Marketplace. Review the policy and required terms, and click **Accept**. > 📝 **NOTE** > > If you don’t have a billing account associated with your project, you’re prompted to enable billing to link the subscription with a billing account. You are taken to the Redpanda sign-up page. 3. On the Redpanda sign-up page: - For **Email**, enter your email address to register with Redpanda. - For **Organization name**, enter a name for your new organization connected through Google Cloud Marketplace. Redpanda organizations contain all resources, including clusters and networks. - Click **Sign up and create organization**. You will receive an email sent to the address you entered. 4. In the email, click **Verify email address**. This completes the registration and associates the email with a Redpanda account. 5. On the **Accept your invitation to sign up** page, click **Sign up** or **Log in**. You can now create resource groups, clusters, and networks in your organization. ## [](#next-steps)Next steps - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#create-a-serverless-cluster) - [Create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/) --- # Page 212: Use GCP Pay As You Go **URL**: https://docs.redpanda.com/cloud-data-platform/billing/gcp-pay-as-you-go.md --- # Use GCP Pay As You Go > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use GCP Pay As You Go latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: gcp-pay-as-you-go page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: gcp-pay-as-you-go.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/gcp-pay-as-you-go.adoc description: Subscribe to Redpanda in Google Cloud Marketplace with pay-as-you-go billing, and cancel anytime. page-git-created-date: "2026-05-05" page-git-modified-date: "2026-05-05" --- Subscribe to Redpanda Cloud through Google Cloud Marketplace to provision Serverless and Dedicated clusters. With a usage-based pay-as-you-go subscription, you only pay for what you use and can cancel anytime. > ❗ **IMPORTANT** > > When you sign up for Redpanda Cloud through Google Cloud Marketplace, you can only create clusters on GCP. > 📝 **NOTE** > > Serverless on GCP is currently in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release. ## [](#sign-up-in-google-cloud-marketplace)Sign up in Google Cloud Marketplace 1. In the Google Cloud Marketplace, select [**Redpanda Cloud - The proven Apache Kafka alternative (Pay as You Go)**](https://console.cloud.google.com/marketplace/product/redpanda-public/redpanda-cloud-platform?project=redpanda-public). 2. On the **Redpanda Cloud - Pay as You Go** overview page, click **Subscribe**. > 📝 **NOTE** > > If you don’t have a billing account associated with your project, you’re prompted to link the subscription with a billing account. 3. On the **Subscribe to Redpanda Cloud** page, click **Set up your account**. You’re taken to the Redpanda sign-up page. 4. On the Redpanda sign-up page: - For **Email**, enter your email address to register with Redpanda. - For **Organization name**, enter a name for your new organization connected through Google Cloud Marketplace. > 💡 **TIP** > > This process creates a new organization, even for existing Redpanda customers. Organizations contain all resources, including clusters and networks. - Click **Sign up and create organization**. Redpanda sends a verification email to the address you entered. 5. In the email, click **Verify email address**. This associates the email with a Redpanda account. 6. On the **Accept your invitation to sign up** page, enter the credentials you want to use for Redpanda Cloud. You can now create resource groups, networks, and clusters in your organization. ## [](#next-steps)Next steps - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#create-a-serverless-cluster) - [Create a Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/) --- # Page 213: Manage Payment Methods **URL**: https://docs.redpanda.com/cloud-data-platform/billing/manage-payment-methods.md --- # Manage Payment Methods > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Payment Methods latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: manage-payment-methods page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: manage-payment-methods.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/manage-payment-methods.adoc description: Add a credit card, set a default payment method, and update billing contact information in Redpanda Cloud. page-topic-type: how-to personas: platform_admin, evaluator learning-objective-1: Add a credit card as a payment method in Redpanda Cloud learning-objective-2: Set a default payment method that Redpanda charges automatically learning-objective-3: Update the billing contact information for your organization page-git-created-date: "2026-05-11" page-git-modified-date: "2026-05-21" --- To pay for usage in Redpanda Cloud, you must add a credit card on the **Billing** page. The card you add is the payment method for all billable resources in your organization, including Serverless, Dedicated, and BYOC clusters, Redpanda Connect pipelines, and your support plan. The most recently added card becomes the default payment method, but you can change the default at any time. After reading this page, you will be able to: - Add a credit card as a payment method in Redpanda Cloud - Set a default payment method that Redpanda charges automatically - Update the billing contact information for your organization ## [](#prerequisites)Prerequisites - You have the **Admin** role in your Redpanda Cloud organization. See [Role-Based Access Control](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/). - You have a valid credit card. ## [](#add-a-payment-method)Add a payment method 1. Sign in to [Redpanda Cloud](https://cloud.redpanda.com). 2. From the navigation menu, click **Billing**. 3. On the **Billing** page, select the **Payment methods** tab. 4. Click **Add payment method**. 5. Enter your card details and billing address, then click **Save**. The new card appears on the **Payment methods** tab. The most recently added card becomes the default payment method unless you select a different one. > 📝 **NOTE** > > - After you add a credit card, any remaining credit balance is applied first. After that, Redpanda charges the default card on the first of each month. > > - Serverless free trials do not require a credit card to start. After your trial ends, you have a 7-day grace period to add a payment method before your clusters are suspended. See [Serverless Clusters](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/). ## [](#set-the-default-payment-method)Set the default payment method If you have more than one card on file, you can choose which one Redpanda charges: 1. Sign in to [Redpanda Cloud](https://cloud.redpanda.com). 2. From the navigation menu, click **Billing**. 3. On the **Billing** page, select the **Payment methods** tab. 4. Find the card you want to use, and mark it as the default. The card you selected is now labeled **Default payment method** and is the one Redpanda charges for usage. ## [](#remove-a-payment-method)Remove a payment method 1. Sign in to [Redpanda Cloud](https://cloud.redpanda.com). 2. From the navigation menu, click **Billing**. 3. On the **Billing** page, select the **Payment methods** tab. 4. Find the card you want to remove, and delete it. The card is removed from the **Payment methods** tab. If the card you want to remove is the default and the only card on file, add another card and set it as the default first. For help, contact [billing@redpanda.com](mailto:billing@redpanda.com). ## [](#update-billing-contact-information)Update billing contact information The billing contact is the person and address Redpanda uses for invoices and billing-related communication. It is separate from the recipients of low-balance email alerts, which are all users with the **Admin** role in your organization. See [Manage Billing Notifications](https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications/). To update the billing contact: 1. Sign in to [Redpanda Cloud](https://cloud.redpanda.com). 2. From the navigation menu, click **Billing**. 3. On the **Billing** page, select the **Settings** tab. 4. Next to **Billing contact information**, click **Edit**. 5. Update all required fields, then click **Save**. The updated billing contact appears on the **Settings** tab. ## [](#next-steps)Next steps - [Billing and Support](https://docs.redpanda.com/cloud-data-platform/billing/billing/) - [View Billing Activity](https://docs.redpanda.com/cloud-data-platform/billing/view-billing-activity/) - [Manage Billing Notifications](https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications/) --- # Page 214: View Billing Activity **URL**: https://docs.redpanda.com/cloud-data-platform/billing/view-billing-activity.md --- # View Billing Activity > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: View Billing Activity latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: view-billing-activity page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: view-billing-activity.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/billing/pages/view-billing-activity.adoc description: View charges, filter the resources breakdown, and export billing activity to CSV in Redpanda Cloud. page-topic-type: how-to personas: platform_admin, evaluator learning-objective-1: View a summary of charges for your organization learning-objective-2: Filter the per-resource breakdown of billing activity learning-objective-3: Export billing activity to a CSV file page-git-created-date: "2026-05-11" page-git-modified-date: "2026-05-21" --- The **Billing activity** tab on the **Billing** page shows a summary of charges and a per-resource breakdown for your organization. Use it to review usage for the current or a previous month, drill into per-resource costs, and export charges for record keeping or external billing systems. After reading this page, you will be able to: - View a summary of charges for your organization - Filter the per-resource breakdown of billing activity - Export billing activity to a CSV file ## [](#prerequisites)Prerequisites - You have the **Admin** role in your Redpanda Cloud organization. See [Role-Based Access Control](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/). ## [](#view-a-summary-of-charges)View a summary of charges 1. Sign in to [Redpanda Cloud](https://cloud.redpanda.com). 2. From the navigation menu, click **Billing**. 3. On the **Billing** page, select the **Billing activity** tab. 4. From the **Time range** menu, choose the period to display, for example, the current month or a previous month. The tab has three sections: - **Usage totals**: A high-level summary that subtotals usage by resource type (Dedicated, BYOC, Serverless) and shows the total amount owed for the selected time range. Expand any row to see the metrics that contribute to that subtotal. Amounts are shown before any discounts are applied. - **Resources breakdown**: A per-resource list showing the cost of each cluster, Redpanda Connect pipeline, and your support plan. Filter the list by: - Resource type: All, Dedicated, Serverless, Redpanda Connect pipeline, or Support. - Resource group: Limit results to a specific resource group. - Resource name: Search by name. - Show deleted resources: Toggle to include resources that have been deleted in the selected time range. Expand any resource to see a table of activity (for example, Compute, Ingress, Egress, Storage, Uptime, Partitions) with quantity, unit price, and amount. - **Plan details**: A side panel showing your pricing plan (for example, Pay-as-you-go) and the payment method on file. ## [](#download-charges-as-csv)Download charges as CSV To download a CSV summary of your monthly charges per resource, click the download icon next to **Resources breakdown**. You can use the exported file for record keeping or to import into your billing system. ## [](#next-steps)Next steps - [Billing and Support](https://docs.redpanda.com/cloud-data-platform/billing/billing/) - [Manage Payment Methods](https://docs.redpanda.com/cloud-data-platform/billing/manage-payment-methods/) - [Manage Billing Notifications](https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications/) --- # Page 215: Develop **URL**: https://docs.redpanda.com/cloud-data-platform/develop.md --- # Develop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Develop latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/index.adoc description: Develop doc topics. page-git-created-date: "2024-06-06" page-git-modified-date: "2024-06-07" --- - [Kafka Compatibility](kafka-clients/) Kafka clients, version 0.11 or later, are compatible with Redpanda. Validations and exceptions are listed. - [Topics](topics/) Overview of standard topics in Redpanda Cloud. - [Produce Data](produce-data/) Learn how to configure producers and idempotent producers. - [Consume Data](consume-data/) Learn about consumer offsets and follower fetching. - [Use Redpanda with the HTTP Proxy API](http-proxy/) HTTP Proxy exposes a REST API to list topics, produce events, and subscribe to events from topics using consumer groups. - [Redpanda Cloud Management MCP Server](cloud-mcp/) Manage your Redpanda Cloud clusters, topics, and users through AI agents using natural language commands. - [Data Transforms](data-transforms/) Learn about WebAssembly data transforms within Redpanda Cloud. - [Transactions](transactions/) Learn how to use transactions; for example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. - [Kafka Connect](managed-connectors/) Use Kafka Connect to stream data into and out of Redpanda. --- # Page 216: Redpanda Cloud Management MCP Server **URL**: https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp.md --- # Redpanda Cloud Management MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud Management MCP Server page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cloud-mcp/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cloud-mcp/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/cloud-mcp/index.adoc # Beta release status page-beta: "true" description: Manage your Redpanda Cloud clusters, topics, and users through AI agents using natural language commands. page-git-created-date: "2026-06-15" page-git-modified-date: "2026-06-15" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- - [Redpanda Cloud Management MCP Server](overview/) Let AI agents securely operate your Redpanda Cloud clusters, topics, and users through natural language commands. - [Redpanda Cloud Management MCP Server Quickstart](quickstart/) Connect your Claude AI agent to your Redpanda Cloud account and clusters using the Redpanda Cloud Management MCP Server. - [Configure the Redpanda Cloud Management MCP Server](configuration/) Learn how to configure the Redpanda Cloud Management MCP Server, including auto and manual client setup, enabling deletes, and security considerations. --- # Page 217: Configure the Redpanda Cloud Management MCP Server **URL**: https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/configuration.md --- # Configure the Redpanda Cloud Management MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure the Redpanda Cloud Management MCP Server page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cloud-mcp/configuration page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cloud-mcp/configuration.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/cloud-mcp/configuration.adoc # Beta release status page-beta: "true" description: Learn how to configure the Redpanda Cloud Management MCP Server, including auto and manual client setup, enabling deletes, and security considerations. page-topic-type: how-to personas: agent_developer, platform_admin learning-objective-1: Configure MCP clients learning-objective-2: Enable delete operations safely learning-objective-3: Troubleshoot common configuration issues page-git-created-date: "2026-06-15" page-git-modified-date: "2026-06-15" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- After installing the Redpanda Cloud Management MCP Server, you can configure it for different AI clients, customize security settings, and troubleshoot common issues. After reading this page, you will be able to: - Configure MCP clients - Enable delete operations safely - Troubleshoot common configuration issues ## [](#prerequisites)Prerequisites - At least version 25.2.3 of [`rpk` installed on your local machine](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) - Access to a Redpanda Cloud account - An MCP-compatible AI client such as Claude, Claude Code, or another tool that supports MCP > 💡 **TIP** > > The MCP server exposes Redpanda Cloud API endpoints for both the [Control Plane](https://docs.redpanda.com/api/doc/cloud-controlplane/) and the [Data Plane](https://docs.redpanda.com/api/doc/cloud-dataplane/). Available endpoints depend on your `rpk` version. Keep `rpk` updated to access new Redpanda Cloud features through the MCP server. New MCP endpoints are documented in Redpanda [release notes](https://github.com/redpanda-data/redpanda/releases). ## [](#install-the-integration-for-claude-or-claude-code)Install the integration for Claude or Claude Code For some supported clients, you can install and configure the MCP integration using the [`rpk cloud mcp install` command](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-mcp-install/). For Claude and Claude Code, run one of these commands: ```bash # Choose one rpk cloud mcp install --client claude rpk cloud mcp install --client claude-code ``` If you need to update the integration, re-run the install command for your client. ## [](#configure-other-mcp-clients-manually)Configure other MCP clients manually If you’re using another MCP-compatible client, manually configure it to use the Redpanda Cloud Management MCP Server. Follow these steps: Add an MCP server entry to your client’s configuration (example shown in JSON). Adjust paths for your system. ```json "mcpServers": { "redpandaCloud": { "command": "rpk", "args": [ "--config", "", (1) "cloud", "mcp", "stdio" ] } } ``` | 1 | Optional: The --config flag lets you target a specific rpk.yaml, which contains the configuration for connecting to your cluster. Always use the same configuration path as you used for rpk cloud login to ensure it has your token. Default paths vary by operating system. See the rpk cloud login reference for the default paths. | | --- | --- | You can also [start the server manually in a terminal to observe logs and troubleshoot](#local). ## [](#enable-delete-operations)Enable delete operations The server disables destructive operations by default. To allow delete operations, add `--allow-delete` to the MCP server invocation. > ⚠️ **CAUTION** > > Enabling delete operations permits actions like **deleting topics or clusters**. Restrict access to your AI client and double-check prompts. ### Auto-configured clients ```bash # Choose one rpk cloud mcp install --client claude --allow-delete rpk cloud mcp install --client claude-code --allow-delete ``` ### Manual configuration example ```json "mcpServers": { "redpandaCloud": { "command": "rpk", "args": [ "cloud", "mcp", "stdio", "--allow-delete" ] } } ``` ## [](#specify-configuration-file-paths)Specify configuration file paths All `rpk` commands accept a `--config` flag, which lets you specify the exact `rpk.yaml` configuration file to use for connecting to your Redpanda cluster. This flag overrides the default search path and ensures that the command uses the credentials and settings from the file you provide. Always use the same configuration path for both `rpk cloud login` and any MCP server setup or install commands to avoid authentication issues. By default, `rpk` searches for config files in standard locations depending on your operating system. See the [reference documentation](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-login/) for details. Use an absolute path and make sure your user has read and write permissions. > ⚠️ **CAUTION** > > The `rpk` configuration file contains your Redpanda Cloud token. Keep the file secure and never share it. For example, if you want to use a custom config path, specify it for both login and the MCP install command: ```bash rpk cloud login --config /Users//my-rpk-config.yaml rpk cloud mcp install --client claude --config /Users//my-rpk-config.yaml ``` Or for Claude Code: ```bash rpk cloud login --config /Users//my-rpk-config.yaml rpk cloud mcp install --client claude-code --config /Users//my-rpk-config.yaml ``` ## [](#remove-the-mcp-server)Remove the MCP server To remove the MCP server, delete or disable the `mcpServers.redpandaCloud` entry in your client’s config (steps vary by client). ## [](#security-considerations)Security considerations - Avoid enabling `--allow-delete` unless required. - For most local use cases, such as with Claude or Claude Code, log in with your personal Redpanda Cloud user account for better security and easier management. - If you are deploying the MCP server as part of an application or shared environment, consider using a [service account](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#authenticate-to-the-cloud-api) with tailored roles. To log in as a service account, use: ```bash rpk cloud login --client-id --client-secret --save ``` - Regularly review and rotate your credentials. ## [](#troubleshooting)Troubleshooting ### [](#verify-your-installation)Verify your installation 1. Make sure you are using at least version 25.2.3 of `rpk`. 2. If you see authentication errors, run `rpk cloud login` again. 3. Ensure you installed for the right client: ```bash rpk cloud mcp install --client claude # or rpk cloud mcp install --client claude-code ``` 4. If using another MCP client, verify your `mcpServers.redpandaCloud` entry (paths, JSON syntax, and args order). 5. Start the server manually using the [`rpk cloud mcp stdio` command](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-mcp-stdio/) (one-time login required) to verify connectivity to Redpanda Cloud endpoints: ```bash rpk cloud login rpk cloud mcp stdio ``` 1. Send the following newline-delimited JSON-RPC messages (each on its own line): ```json {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{"roots":{},"sampling":{},"elicitation":{}},"clientInfo":{"name":"ManualTest","version":"0.1.0"}}} {"jsonrpc":"2.0","method":"notifications/initialized"} {"jsonrpc":"2.0","id":2,"method":"tools/list"} ``` Expected response shapes (examples): ```json {"jsonrpc":"2.0","id":1,"result":{"capabilities":{...}}} {"jsonrpc":"2.0","id":2,"result":{"tools":[{"name":"...","description":"..."}, ...]}} ``` 2. Stop the server with `Ctrl+C`. ### [](#client-cant-find-the-mcp-server)Client can’t find the MCP server - Re-run the install for your MCP client. - Confirm the path in `--config /path/to/rpk.yaml` exists and is readable. - Double-check your client’s configuration format and syntax. ### [](#unauthorized-errors-or-token-errors)Unauthorized errors or token errors Your capabilities depend on your Redpanda Cloud account permissions. If an operation fails with a permissions error, contact your account admin. - Run `rpk cloud login` to refresh the token. - Ensure your account has the necessary permissions for the requested operation. ### [](#deletes-not-working)Deletes not working - The server disables delete operations by default. Add `--allow-delete` to the server invocation (auto or manual configuration) and restart the client. - For auto-configured clients, you may need to edit the generated config or re-run the install command and adjust the entry. --- # Page 218: Redpanda Cloud Management MCP Server **URL**: https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/overview.md --- # Redpanda Cloud Management MCP Server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud Management MCP Server page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cloud-mcp/overview page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cloud-mcp/overview.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/cloud-mcp/overview.adoc # Beta release status page-beta: "true" description: Let AI agents securely operate your Redpanda Cloud clusters, topics, and users through natural language commands. page-topic-type: overview personas: evaluator, agent_developer, platform_admin learning-objective-1: Explain what the Redpanda Cloud Management MCP Server does learning-objective-2: Identify what operations are available through MCP learning-objective-3: Identify security considerations for MCP authentication page-git-created-date: "2026-06-15" page-git-modified-date: "2026-06-15" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- The Redpanda Cloud Management MCP Server lets AI agents securely access and operate your Redpanda Cloud account and clusters through natural language commands. After reading this page, you will be able to: - Explain what the Redpanda Cloud Management MCP Server does - Identify what operations are available through MCP - Identify security considerations for MCP authentication ![A terminal window showing Claude Code invoking the Redpanda Cloud Management MCP Server to list topics in a cluster.](https://docs.redpanda.com/cloud-data-platform/shared/_images/cloud-mcp.gif) ## [](#what-you-can-do)What you can do MCP provides controlled access to: - [Control Plane](https://docs.redpanda.com/api/doc/cloud-controlplane/) APIs, such as creating a Redpanda Cloud cluster or listing clusters. - [Data Plane](https://docs.redpanda.com/api/doc/cloud-dataplane/) APIs, such as creating topics or listing topics. The MCP server runs on your computer and authenticates to Redpanda Cloud using a Redpanda Cloud token. You can do anything that’s available in the Control Plane or Data Plane APIs. Typical requests you can make to your assistant once connected include: - Create a Redpanda Cloud cluster named `dev-mcp`. - List topics in `dev-mcp`. - Create a topic `orders-raw` with 6 partitions. > 📝 **NOTE** > > The MCP server does **not** expose delete endpoints by default. You can enable delete endpoints when you create the server if you intentionally want to allow delete operations. ## [](#use-cases)Use cases - Test automation: Create short-lived clusters, create topics, and validate pipelines quickly. - Operational assistance: Inspect a cluster’s health or list topics during incidents. - Onboarding and demos: Let team members issue high-level requests without memorizing every CLI flag. ## [](#how-it-works)How it works 1. Authenticate to Redpanda Cloud and receive a token using the [`rpk cloud login` command](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-login/). 2. Configure your MCP client using the [`rpk cloud mcp install`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-mcp-install/) command. Your client then starts the server on-demand using [`rpk cloud mcp stdio`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-mcp-stdio/), authenticating with the Redpanda Cloud token from `rpk cloud login`. 3. Prompt your assistant to perform Redpanda operations. The MCP server executes them in your Redpanda Cloud account using your Redpanda Cloud token. ### [](#components)Components The Redpanda Cloud Management MCP Server requires these components: - AI client (Claude, Claude Code, or any other MCP client) that connects to the MCP server. - Redpanda CLI (`rpk`) for obtaining a token and starting the MCP server. - Redpanda Cloud account that the MCP server can connect to and issue API requests. ## [](#security-considerations)Security considerations MCP servers authenticate to Redpanda Cloud using your personal or service account credentials. However, there is **no auditing or access control** that distinguishes between actions performed by MCP servers versus direct API calls: - All API actions appear in Redpanda Cloud’s internal logs as coming from the authenticated user account, not the specific MCP server. - You cannot audit which MCP server performed which operations, as Redpanda Cloud logs are not accessible to users. - You cannot restrict specific MCP servers to only certain API endpoints or resources. ## [](#next-steps)Next steps - [Redpanda Cloud Management MCP Server Quickstart](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/quickstart/) - [Configure the Redpanda Cloud Management MCP Server](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/configuration/) > 💡 **TIP** > > The Redpanda documentation site has a read-only MCP server that provides access to Redpanda docs and examples. This server has no access to your Redpanda Cloud account or clusters. See [How to Use These Docs](https://docs.redpanda.com/home/how-to-use-these-docs/). --- # Page 219: Redpanda Cloud Management MCP Server Quickstart **URL**: https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/quickstart.md --- # Redpanda Cloud Management MCP Server Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud Management MCP Server Quickstart page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cloud-mcp/quickstart page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cloud-mcp/quickstart.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/cloud-mcp/quickstart.adoc # Beta release status page-beta: "true" description: Connect your Claude AI agent to your Redpanda Cloud account and clusters using the Redpanda Cloud Management MCP Server. page-topic-type: tutorial personas: agent_developer, platform_admin learning-objective-1: Authenticate to Redpanda Cloud with rpk learning-objective-2: Install the MCP integration for Claude learning-objective-3: Issue natural language commands to manage clusters page-git-created-date: "2026-06-15" page-git-modified-date: "2026-06-15" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- In this quickstart, you’ll get your Claude AI agent talking to Redpanda Cloud using the [Redpanda Cloud Management MCP Server](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/overview/). After completing this quickstart, you will be able to: - Authenticate to Redpanda Cloud with rpk - Install the MCP integration for Claude - Issue natural language commands to manage clusters ## [](#prerequisites)Prerequisites - At least version 25.2.3 of [`rpk` installed on your computer](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) - Access to a Redpanda Cloud account - [Claude](https://support.anthropic.com/en/articles/10065433-installing-claude-desktop) or [Claude Code](https://docs.anthropic.com/en/docs/claude-code/setup) installed > 💡 **TIP** > > For other clients, see [Configure the Redpanda Cloud Management MCP Server](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/configuration/). ## [](#set-up-the-mcp-server)Set up the MCP server 1. Verify your `rpk` version. ```bash rpk version ``` Ensure the version is at least 25.2.3. 2. Log in to Redpanda Cloud. ```bash rpk cloud login ``` A browser window opens. Sign in to grant access. After you sign in, `rpk` stores a token locally. This token is not shared with your AI agent. It is used by the MCP server to authenticate requests to your Redpanda Cloud account. 3. Install the MCP integration. Choose one client: ```bash # Claude desktop rpk cloud mcp install --client claude # Claude Code (IDE) rpk cloud mcp install --client claude-code ``` This command configures the MCP server for your client. If you need to update the integration, re-run the install command for your client. ## [](#start-prompting)Start prompting Launch Claude or Claude Code and try one of these prompts: - “Create a Redpanda Cloud cluster named `dev-mcp`.” - “List topics in `dev-mcp`.” - “Create a topic `orders-raw` with 6 partitions.” > 📝 **NOTE: Delete operations are opt-in** > > The MCP server does **not** expose API endpoints that result in delete operations by default. Use `--allow-delete` only if you intentionally want to enable delete operations. See [Enable delete operations](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/configuration/#enable_delete_operations). ## [](#next-steps)Next steps - [Configure the Redpanda Cloud Management MCP Server](https://docs.redpanda.com/cloud-data-platform/develop/cloud-mcp/configuration/) --- # Page 220: Redpanda Connect in Redpanda Cloud **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/about.md --- # Redpanda Connect in Redpanda Cloud > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Connect in Redpanda Cloud latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/about.adoc description: Learn about Redpanda Connect in Redpanda Cloud and its wide range of connectors. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect in Redpanda Cloud lets you quickly build and deploy streaming data pipelines on your clusters from a fully-integrated UI or using the [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/group/endpoint-redpanda-connect-pipeline). Choose from a [wide range of connectors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/) to suit your use case, including connectors to: - Integrate data sources ([inputs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/)) - Write to data sinks ([outputs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/)) - Transform data ([processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/)) Comprehensive data pipeline metrics are also available to help you to [monitor your data pipelines](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/monitor-connect/) and [per pipeline scaling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/resource-management/). Try this [quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/). > 💡 **TIP** > > If you’re new to Redpanda Connect, try [building and testing data pipelines locally](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/) before deploying to the Cloud. --- # Page 221: Components Catalog **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about.md --- # Components Catalog > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Components Catalog latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/about.adoc description: A searchable catalog of available Redpanda Connect components. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Use the following table to search for available inputs, outputs, and processors. Type: All Types Selected ▼ Processor Input Output Scanner Metric Cache Tracer Rate limit Buffer | Name | Connector Type | | --- | --- | | a2a_message | Processor | | amqp_0_9RabbitMQ AMQP | Input, Output | | arc | Output | | archiveZIP TAR GZIP | Processor | | avro | Processor, Scanner | | aws_bedrock_chatAmazon AWS Bedrock Chat | Processor | | aws_bedrock_embeddingsAmazon AWS Bedrock Embeddings | Processor | | aws_cloudwatch_logsAWS CloudWatch Logs Amazon CloudWatch Logs | Input | | aws_dynamodbAWS DynamoDB Amazon DynamoDB DynamoDB | Cache, Output | | aws_dynamodb_cdcAmazon DynamoDB CDC | Input | | aws_dynamodb_partiqlAmazon AWS DynamoDB PartiQL | Processor | | aws_kinesisAWS Kinesis Amazon Kinesis Kinesis | Input, Output | | aws_kinesis_firehoseAWS Kinesis Firehose Amazon Kinesis Firehose Kinesis Firehose | Output | | aws_lambdaAWS Lambda Amazon Lambda Lambda | Processor | | aws_s3AWS S3 Amazon S3 S3 Simple Storage Service | Cache, Input, Output | | aws_snsAWS SNS Amazon SNS SNS Simple Notification Service | Output | | aws_sqsAWS SQS Amazon SQS SQS Simple Queue Service | Input, Output | | azure_blob_storageAzure Blob Storage Microsoft Azure Storage | Input, Output | | azure_cosmosdbMicrosoft Azure Azure | Input, Output, Processor | | azure_data_lake_gen2Microsoft Azure Azure | Output | | azure_queue_storageAzure Queue Storage Microsoft Azure Queue | Input, Output | | azure_table_storageAzure Table Storage Microsoft Azure Table | Input, Output | | batched | Input | | benchmark | Processor | | bloblang | Processor | | bounds_check | Processor | | branch | Processor | | broker | Input, Output | | cache | Output, Processor | | cached | Processor | | catch | Processor | | chunker | Scanner | | cohere_chat | Processor | | cohere_embeddings | Processor | | cohere_rerank | Processor | | compress | Processor | | csvComma-Separated Values | Scanner | | cyborgdb | Output | | decompress | Processor, Scanner | | dedupe | Processor | | drop | Output | | drop_on | Output | | elasticsearch_v8 | Output | | fallback | Output | | for_each | Processor | | gateway | Input | | gcp_bigqueryGCP BigQuery Google BigQuery BigQuery | Output | | gcp_bigquery_selectGCP BigQuery Google Cloud GCP | Input, Processor | | gcp_bigquery_write_apiGCP BigQuery | Output | | gcp_cloud_storageGCP Cloud Storage Google Cloud Storage GCS | Cache, Input, Output | | gcp_cloudtraceGCP Cloud Trace | Tracer | | gcp_pubsubGCP PubSub Google Cloud Pub/Sub GCP Pub/Sub Google Pub/Sub | Input, Output | | gcp_spanner_cdcGoogle Cloud GCP | Input | | gcp_vertex_ai_chatGCP Vertex AI Google Cloud GCP | Processor | | gcp_vertex_ai_embeddingsGoogle Cloud GCP | Processor | | generate | Input | | git | Input | | google_drive_download | Processor | | google_drive_list_labels | Processor | | google_drive_search | Processor | | group_by | Processor | | group_by_value | Processor | | http | Processor | | http_clientHTTP REST API REST | Input, Output | | http_serverHTTP REST API REST Gateway | Input | | icebergApache Iceberg Apache Polaris AWS Glue Databricks Unity Catalog | Output | | inproc | Input, Output | | insert_part | Processor | | jiraAtlassian Jira | Input, Processor | | jmespath | Processor | | jq | Processor | | json_array | Scanner | | json_documents | Scanner | | json_schemaJSON Schema | Processor | | kafkaApache Kafka | Input, Output | | kafka_franzApache Kafka Kafka | Input, Output | | lines | Scanner | | local | Rate_limit | | log | Processor | | lru | Cache | | mapping | Processor | | memcached | Cache | | memory | Buffer, Cache | | metric | Processor | | microsoft_sql_server_cdc | Input | | mongodbMongo | Cache, Input, Output, Processor | | mongodb_cdcMongoDB CDC | Input | | mqtt | Input, Output | | multilevel | Cache | | mutation | Processor | | mysql_cdc | Input | | natsNATS.io | Input, Output | | nats_jetstreamNATS JetStream NATS | Input, Output | | nats_kvNATS KV | Cache, Input, Output, Processor | | nats_request_replyNATS Request Reply | Processor | | none | Buffer, Metric, Tracer | | noop | Cache, Processor | | open_telemetry_collectorOpenTelemetry | Metric, Tracer | | openai_chat_completion | Processor | | openai_embeddings | Processor | | openai_image_generation | Processor | | openai_speech | Processor | | openai_transcription | Processor | | openai_translation | Processor | | opensearch | Output | | oracledb_cdcOracle CDC OracleDB CDC Oracle Database CDC | Input | | otlp_grpcOpenTelemetry OTLP OTel gRPC | Input, Output | | otlp_httpOpenTelemetry OTLP OTel | Input, Output | | parallel | Processor | | parquet_decode | Processor | | parquet_encode | Processor | | parse_log | Processor | | pg_stream | | | pinecone | Output | | postgres_cdc | Input | | processors | Processor | | prometheus | Metric | | qdrant | Output, Processor | | questdb | Output | | rate_limit | Processor | | re_match | Scanner | | read_until | Input | | redis | Cache, Processor, Rate_limit | | redis_hashRedis Hash Redis | Output | | redis_listRedis List Redis Lists Redis | Input, Output | | redis_pubsubRedis PubSub Redis Pub/Sub Redis | Input, Output | | redis_scanRedis | Input | | redis_scriptRedis Script | Processor | | redis_streamsRedis Streams Redis | Input, Output | | redpanda | Cache, Input, Output, Tracer | | redpanda_common | Input, Output | | redpanda_migrator | Input, Output | | reject | Output | | reject_errored | Output | | resource | Input, Output, Processor | | retry | Output, Processor | | ristretto | Cache | | salesforce | Input | | salesforce_cdcSalesforce Salesforce CDC | Input | | salesforce_graphqlSalesforce Salesforce GraphQL | Input | | salesforce_sinkSalesforce Salesforce Sink | Output | | schema_registry | Input, Output | | schema_registry_decode | Processor | | schema_registry_encode | Processor | | select_parts | Processor | | sequence | Input | | sftp | Input, Output | | skip_bom | Scanner | | slack | Input | | slack_postSlack Post | Output | | slack_reactionSlack Reaction | Output | | slack_threadSlack Thread | Processor | | slack_usersSlack Users | Input | | sleep | Processor | | snowflake_putSnowflake | Output | | snowflake_streamingSnowflake Streaming | Output | | spicedb_watch | Input | | split | Processor | | splunk | Input | | splunk_hecSplunk | Output | | sql | Cache | | sql_driver_clickhouseClickHouse | | | sql_driver_mysqlMYSQL | | | sql_driver_oracleOracle | | | sql_driver_postgresPostgreSQL | | | sql_driver_sqliteSQLite | | | sql_insertSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Output, Processor | | sql_rawSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Output, Processor | | sql_selectSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Processor | | string_split | Processor | | switch | Output, Processor, Scanner | | sync_response | Output, Processor | | system_window | Buffer | | tar | Scanner | | text_chunker | Processor | | timeplus | Input, Output | | to_the_end | Scanner | | try | Processor | | try_catch | Processor | | ttlru | Cache | | unarchiveZIP TAR GZIP Archive | Processor | | while | Processor | | workflow | Processor | | xml | Processor | ## [](#about-components)About Components Every Redpanda Connect pipeline has at least one [input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/), an optional [buffer](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/about/), an [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/) and any number of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/): ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: type: none pipeline: processors: - mapping: | message = this meta.link_count = links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` These are the main components within Redpanda Connect and they provide the majority of useful behavior. ## [](#observability-components)Observability components There are also the observability components: [logger](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/logger/about/), [metrics](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/), and [tracing](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/about/), which allow you to specify how Redpanda Connect exposes observability data. ```yaml http: address: 0.0.0.0:4195 enabled: true debug_endpoints: false logger: format: json level: WARN metrics: statsd: address: localhost:8125 flush_period: 100ms tracer: jaeger: agent_address: localhost:6831 ``` ## [](#resource-components)Resource components Finally, there are [caches](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) and [rate limits](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/). These are components that are referenced by core components and can be shared. ```yaml input: http_client: # This is an input url: TODO rate_limit: foo_ratelimit # This is a reference to a rate limit pipeline: processors: - cache: # This is a processor resource: baz_cache # This is a reference to a cache operator: add key: '${! json("id") }' value: "x" - mapping: root = if errored() { deleted() } rate_limit_resources: - label: foo_ratelimit local: count: 500 interval: 1s cache_resources: - label: baz_cache memcached: addresses: [ localhost:11211 ] ``` It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the [`resource` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/resource/), [`resource` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/resource/) and [`resource` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/resource/) respectively. For more information about any of these component types check out their sections: - [inputs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/) - [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) - [outputs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/) - [buffers](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/about/) - [metrics](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/) - [tracers](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/about/) - [logger](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/logger/about/) - [caches](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) - [rate limits](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) --- # Page 222: Buffers **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/about.md --- # Buffers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Buffers latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/buffers/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/buffers/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/buffers/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect uses a transaction model internally for guaranteeing delivery of messages, this means that a message from an input is not acknowledged (or its offset committed, etc) until that message has been processed and either intentionally deleted or successfully delivered to all outputs. This transaction model makes Redpanda Connect safe to deploy in scenarios where data loss is unacceptable. However, sometimes it’s useful to customize the way in which messages are delivered, and this is where buffers come in. A buffer is an optional component type that comes immediately after the input layer and can be used as a way of decoupling the transaction model from components downstream such as the processing layer and outputs. This is considered an advanced component as most users will likely not benefit from a buffer, but they enable you to do things like group messages using window algorithms or intentionally weaken the delivery guarantees of the pipeline depending on the buffer you choose. Since buffers are able to modify (or disable) the transaction model within Redpanda Connect it is important that when you choose a buffer you read its documentation to understand the implication it will have on delivery guarantees. --- # Page 223: memory **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/memory.md --- # memory > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: memory latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/buffers/memory page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/buffers/memory.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/buffers/memory.adoc description: Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly. #### Common ```yml buffers: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml buffers: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This buffer is appropriate when consuming messages from inputs that do not gracefully handle back pressure and where delivery guarantees aren’t critical. This buffer has a configurable limit, where consumption will be stopped with back pressure upstream if the total size of messages in the buffer reaches this amount. Since this calculation is only an estimate, and the real size of messages in RAM is always higher, it is recommended to set the limit significantly below the amount of RAM available. ## [](#delivery-guarantees)Delivery guarantees This buffer intentionally weakens the delivery guarantees of the pipeline and therefore should never be used in places where data loss is unacceptable. ## [](#batching)Batching It is possible to batch up messages sent from this buffer using a [batch policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/#batch-policy). ## [](#fields)Fields ### [](#batch_policy)`batch_policy` Optionally configure a policy to flush buffered messages in batches. **Type**: `object` ### [](#batch_policy-byte_size)`batch_policy.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batch_policy-check)`batch_policy.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batch_policy-count)`batch_policy.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batch_policy-enabled)`batch_policy.enabled` Whether to batch messages as they are flushed. **Type**: `bool` **Default**: `false` ### [](#batch_policy-period)`batch_policy.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batch_policy-processors)`batch_policy.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#limit)`limit` The maximum buffer size (in bytes) to allow before applying backpressure upstream. **Type**: `int` **Default**: `524288000` --- # Page 224: none **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/none.md --- # none > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: none latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/buffers/none page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/buffers/none.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/buffers/none.adoc description: Do not buffer messages. This is the default and most resilient configuration. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Do not buffer messages. This is the default and most resilient configuration. ```yml # Config fields, showing default values buffer: none: {} ``` Selecting no buffer means the output layer is directly coupled with the input layer. This is the safest and lowest latency option since acknowledgements from at-least-once protocols can be propagated all the way from the output protocol to the input protocol. If the output layer is hit with back pressure it will propagate all the way to the input layer, and further up the data stream. If you need to relieve your pipeline of this back pressure consider using a more robust buffering solution such as Kafka before resorting to alternatives. --- # Page 225: system_window **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/system_window.md --- # system_window > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: system_window latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/buffers/system_window page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/buffers/system_window.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/buffers/system_window.adoc description: Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock. #### Common ```yml buffers: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` #### Advanced ```yml buffers: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` A window is a grouping of messages that fit within a discrete measure of time following the system clock. Messages are allocated to a window either by the processing time (the time at which they’re ingested) or by the event time, and this is controlled via the [`timestamp_mapping` field](#timestamp_mapping). In tumbling mode (default) the beginning of a window immediately follows the end of a prior window. When the buffer is initialized the first window to be created and populated is aligned against the zeroth minute of the zeroth hour of the day by default, and may therefore be open for a shorter period than the specified size. A window is flushed only once the system clock surpasses its scheduled end. If an [`allowed_lateness`](#allowed_lateness) is specified then the window will not be flushed until the scheduled end plus that length of time. When a message is added to a window it has a metadata field `window_end_timestamp` added to it containing the timestamp of the end of the window as an RFC3339 string. ## [](#sliding-windows)Sliding windows Sliding windows begin from an offset of the prior windows' beginning rather than its end, and therefore messages may belong to multiple windows. In order to produce sliding windows specify a [`slide` duration](#slide). ## [](#back-pressure)Back pressure If back pressure is applied to this buffer either due to output services being unavailable or resources being saturated, windows older than the current and last according to the system clock will be dropped in order to prevent unbounded resource usage. This means you should ensure that under the worst case scenario you have enough system memory to store two windows' worth of data at a given time (plus extra for redundancy and other services). If messages could potentially arrive with event timestamps in the future (according to the system clock) then you should also factor in these extra messages in memory usage estimates. ## [](#delivery-guarantees)Delivery guarantees This buffer honours the transaction model within Redpanda Connect in order to ensure that messages are not acknowledged until they are either intentionally dropped or successfully delivered to outputs. However, since messages belonging to an expired window are intentionally dropped there are circumstances where not all messages entering the system will be delivered. When this buffer is configured with a slide duration it is possible for messages to belong to multiple windows, and therefore be delivered multiple times. In this case the first time the message is delivered it will be acked (or nacked) and subsequent deliveries of the same message will be a "best attempt". During graceful termination if the current window is partially populated with messages they will be nacked such that they are re-consumed the next time the service starts. ## [](#examples)Examples ### Counting Passengers at Traffic Given a stream of messages relating to cars passing through various traffic lights of the form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` We can use a window buffer in order to create periodic messages summarizing the traffic for a period of time of this form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "total_cars": 15, "passengers": 43 } ``` With the following config: ```yaml buffer: system_window: timestamp_mapping: root = this.created_at size: 1h pipeline: processors: # Group messages of the window into batches of common traffic light IDs - group_by_value: value: '${! json("traffic_light") }' # Reduce each batch to a single message by deleting indexes > 0, and # aggregate the car and passenger counts. - mapping: | root = if batch_index() == 0 { { "traffic_light": this.traffic_light, "created_at": meta("window_end_timestamp"), "total_cars": json("registration_plate").from_all().unique().length(), "passengers": json("passengers").from_all().sum(), } } else { deleted() } ``` ## [](#fields)Fields ### [](#timestamp_mapping)`timestamp_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) applied to each message during ingestion that provides the timestamp to use for allocating it a window. By default the function `now()` is used in order to generate a fresh timestamp at the time of ingestion (the processing time), whereas this mapping can instead extract a timestamp from the message itself (the event time). The timestamp value assigned to `root` must either be a numerical unix time in seconds (with up to nanosecond precision via decimals), or a string in ISO 8601 format. If the mapping fails or provides an invalid result the message will be dropped (with logging to describe the problem). **Type**: `string` **Default**: `"root = now()"` ```yml # Examples timestamp_mapping: root = this.created_at timestamp_mapping: root = meta("kafka_timestamp_unix").number() ``` ### [](#size)`size` A duration string describing the size of each window. By default windows are aligned to the zeroth minute and zeroth hour on the UTC clock, meaning windows of 1 hour duration will match the turn of each hour in the day, this can be adjusted with the `offset` field. **Type**: `string` ```yml # Examples size: 30s size: 10m ``` ### [](#slide)`slide` An optional duration string describing by how much time the beginning of each window should be offset from the beginning of the previous, and therefore creates sliding windows instead of tumbling. When specified this duration must be smaller than the `size` of the window. **Type**: `string` **Default**: `""` ```yml # Examples slide: 30s slide: 10m ``` ### [](#offset)`offset` An optional duration string to offset the beginning of each window by, otherwise they are aligned to the zeroth minute and zeroth hour on the UTC clock. The offset cannot be a larger or equal measure to the window size or the slide. **Type**: `string` **Default**: `""` ```yml # Examples offset: -6h offset: 30m ``` ### [](#allowed_lateness)`allowed_lateness` An optional duration string describing the length of time to wait after a window has ended before flushing it, allowing late arrivals to be included. Since this windowing buffer uses the system clock an allowed lateness can improve the matching of messages when using event time. **Type**: `string` **Default**: `""` ```yml # Examples allowed_lateness: 10s allowed_lateness: 1m ``` --- # Page 226: Caches **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about.md --- # Caches > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Caches latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A cache is a key/value store which can be used by certain components for applications such as deduplication or data joins. Caches are configured as a named resource: ```yaml cache_resources: - label: foobar memcached: addresses: - localhost:11211 default_ttl: 60s ``` > It’s possible to layer caches with read-through and write-through behavior using the [`multilevel` cache](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/multilevel/). And then any components that use caches have a field `resource` that specifies the cache resource: ```yaml pipeline: processors: - cache: resource: foobar operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } ``` For the simple case where you wish to store messages in a cache as an output destination for your pipeline check out the [`cache` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/cache/). To see examples of more advanced uses of caches such as hydration and deduplication check out the [`cache` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cache/). --- # Page 227: aws_dynamodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/aws_dynamodb.md --- # aws_dynamodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_dynamodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/aws_dynamodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/aws_dynamodb.adoc description: Stores key/value pairs as a single document in a DynamoDB table. The key is stored as a string value and used as the table hash key. The value is stored as a binary value using the data_key field name. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores key/value pairs as a single document in a DynamoDB table. The key is stored as a string value and used as the table hash key. The value is stored as a binary value using the `data_key` field name. #### Common ```yml caches: aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) ``` #### Advanced ```yml caches: aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) consistent_read: false default_ttl: "" # No default (optional) ttl_key: "" # No default (optional) retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` A prefix can be specified to allow multiple cache types to share a single DynamoDB table. An optional TTL duration (`ttl`) and field (`ttl_key`) can be specified if the backing table has TTL enabled. Strong read consistency can be enabled using the `consistent_read` configuration field. ## [](#fields)Fields ### [](#consistent_read)`consistent_read` Whether to use strongly consistent reads on Get commands. **Type**: `bool` **Default**: `false` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#data_key)`data_key` The key of the table column to store item values within. **Type**: `string` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. A `ttl_key` must be specified in order to set item TTLs. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#hash_key)`hash_key` The key of the table column to store item keys within. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#table)`table` The table to store items in. **Type**: `string` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. **Type**: `string` --- # Page 228: aws_s3 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/aws_s3.md --- # aws_s3 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_s3 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/aws_s3 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/aws_s3.adoc description: Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket. #### Common ```yml caches: aws_s3: bucket: "" # No default (required) content_type: application/octet-stream ``` #### Advanced ```yml caches: aws_s3: bucket: "" # No default (required) content_type: application/octet-stream force_path_style_urls: false retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` It is not possible to atomically upload S3 objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The S3 bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` The content type to set for each item. **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 229: gcp_cloud_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/gcp_cloud_storage.md --- # gcp_cloud_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_cloud_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/gcp_cloud_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/gcp_cloud_storage.adoc description: Use a Google Cloud Storage bucket as a cache. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Use a Google Cloud Storage bucket as a cache. ```yml caches: gcp_cloud_storage: bucket: "" # No default (required) content_type: "" # No default (optional) credentials_json: "" ``` It is not possible to atomically upload cloud storage objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The Google Cloud Storage bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` Optional field to explicitly set the Content-Type. **Type**: `string` ### [](#credentials_json)`credentials_json` An optional field to set Google Service Account Credentials json. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` --- # Page 230: lru **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/lru.md --- # lru > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: lru latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/lru page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/lru.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/lru.adoc description: Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts. #### Common ```yml caches: lru: cap: 1000 init_values: {} ``` #### Advanced ```yml caches: lru: cap: 1000 init_values: {} algorithm: standard two_queues_recent_ratio: 0.25 two_queues_ghost_ratio: 0.5 optimistic: false ``` This provides the lru package which implements a fixed-size thread safe LRU cache. It uses the package [`lru`](https://github.com/hashicorp/golang-lru/v2) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache lru: cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. ## [](#fields)Fields ### [](#algorithm)`algorithm` the lru cache implementation **Type**: `string` **Default**: `standard` | Option | Summary | | --- | --- | | arc | is an adaptive replacement cache. It tracks recent evictions as well as recent usage in both the frequent and recent caches. Its computational overhead is comparable to two_queues, but the memory overhead is linear with the size of the cache. ARC has been patented by IBM. | | standard | is a simple LRU cache. It is based on the LRU implementation in groupcache | | two_queues | tracks frequently used and recently used entries separately. This avoids a burst of accesses from taking out frequently used entries, at the cost of about 2x computational overhead and some extra bookkeeping. | ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1000` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The lru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#two_queues_ghost_ratio)`two_queues_ghost_ratio` is the default ratio of ghost entries kept to track entries recently evicted on two\_queues cache. **Type**: `float` **Default**: `0.5` ### [](#two_queues_recent_ratio)`two_queues_recent_ratio` is the ratio of the two\_queues cache dedicated to recently added entries that have only been accessed once. **Type**: `float` **Default**: `0.25` --- # Page 231: memcached **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/memcached.md --- # memcached > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: memcached latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/memcached page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/memcached.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/memcached.adoc description: Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different namespaces. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different namespaces. #### Common ```yml caches: memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s ``` #### Advanced ```yml caches: memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of addresses of memcached servers to use. **Type**: `array` ### [](#default_ttl)`default_ttl` A default TTL to set for items, calculated from the moment the item is cached. **Type**: `string` **Default**: `300s` ### [](#prefix)`prefix` An optional string to prefix item keys with in order to prevent collisions with similar services. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 232: memory **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/memory.md --- # memory > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: memory latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/memory page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/memory.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/memory.adoc description: Stores key/value pairs in a map held in memory. This cache is therefore reset every time the service restarts. Each item in the cache has a TTL set from the moment it was last edited, after which it will be removed during the next compaction. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores key/value pairs in a map held in memory. This cache is therefore reset every time the service restarts. Each item in the cache has a TTL set from the moment it was last edited, after which it will be removed during the next compaction. #### Common ```yml caches: memory: default_ttl: 5m compaction_interval: 60s init_values: {} ``` #### Advanced ```yml caches: memory: default_ttl: 5m compaction_interval: 60s init_values: {} shards: 1 ``` The compaction interval determines how often the cache is cleared of expired items, and this process is only triggered on writes to the cache. Access to the cache is blocked during this process. Item expiry can be disabled entirely by setting the `compaction_interval` to an empty string. The field `init_values` can be used to prepopulate the memory cache with any number of key/value pairs which are exempt from TTLs: ```yaml cache_resources: - label: foocache memory: default_ttl: 60s init_values: foo: bar ``` These values can be overridden during execution, at which point the configured TTL is respected as usual. ## [](#fields)Fields ### [](#compaction_interval)`compaction_interval` The period of time to wait before each compaction, at which point expired items are removed. This field can be set to an empty string in order to disable compactions/expiry entirely. **Type**: `string` **Default**: `60s` ### [](#default_ttl)`default_ttl` The default TTL of each item. After this period an item will be eligible for removal during the next compaction. **Type**: `string` **Default**: `5m` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#shards)`shards` A number of logical shards to spread keys across, increasing the shards can have a performance benefit when processing a large number of keys. **Type**: `int` **Default**: `1` --- # Page 233: mongodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/mongodb.md --- # mongodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mongodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/mongodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/mongodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/mongodb.adoc description: Use a MongoDB instance as a cache. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- Use a MongoDB instance as a cache. #### Common ```yml caches: mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` #### Advanced ```yml caches: mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#key_field)`key_field` The field in the document that is used as the key. **Type**: `string` ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` ### [](#value_field)`value_field` The field in the document that is used as the value. **Type**: `string` --- # Page 234: multilevel **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/multilevel.md --- # multilevel > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: multilevel latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/multilevel page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/multilevel.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/multilevel.adoc description: Combines multiple caches as levels, performing read-through and write-through operations across them. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Combines multiple caches as levels, performing read-through and write-through operations across them. ```yml caches: multilevel: - label: "" memory: default_ttl: 5m compaction_interval: 60s - label: "" redis: url: redis://localhost:6379 expiration: 24h ``` ## [](#examples)Examples ### [](#hot-and-cold-cache)Hot and cold cache The multilevel cache is useful for reducing traffic against a remote cache by routing it through a local cache. In the following example requests will only go through to the memcached server if the local memory cache is missing the key. ```yaml pipeline: processors: - branch: processors: - cache: resource: leveled operator: get key: ${! json("key") } - catch: - mapping: 'root = {"err":error()}' result_map: 'root.result = this' cache_resources: - label: leveled multilevel: [ hot, cold ] - label: hot memory: default_ttl: 60s - label: cold memcached: addresses: [ TODO:11211 ] default_ttl: 60s ``` --- # Page 235: nats_kv **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/nats_kv.md --- # nats_kv > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_kv latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/nats_kv page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/nats_kv.adoc description: Cache key/values in a NATS key-value bucket. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Cache key/value pairs in a NATS key-value bucket. #### Common ```yml caches: nats_kv: urls: [] # No default (required) bucket: "" # No default (required) ``` #### Advanced ```yml caches: nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` The NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of an user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plain text user JWT (given along with the corresponding user NKey Seed). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plain text user NKey Seed (given along with the corresponding user JWT). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 236: noop **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/noop.md --- # noop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: noop latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/noop page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/noop.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/noop.adoc description: Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option. Introduced in version 4.27.0. ```yml caches: noop: {} ``` --- # Page 237: redis **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/redis.md --- # redis > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/redis page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/redis.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/redis.adoc description: Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration. #### Common ```yml caches: redis: url: "" # No default (required) prefix: "" # No default (optional) ``` #### Advanced ```yml caches: redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] prefix: "" # No default (optional) default_ttl: "" # No default (optional) retries: initial_interval: 500ms max_interval: 1s max_elapsed_time: 5s ``` ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#prefix)`prefix` An optional string to prefix item keys with in order to prevent collisions with similar services. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `5s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `1s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 238: redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/redpanda.md --- # redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/redpanda page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/redpanda.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/redpanda.adoc description: A Kafka cache using the Franz Kafka client library. page-git-created-date: "2025-07-08" page-git-modified-date: "2026-08-11" --- A Kafka cache implemented using the [Franz Kafka client library](https://github.com/twmb/franz-go). #### Common ```yaml caches: redpanda: seed_brokers: [] # No default (required) topic: "" # No default (required) ``` #### Advanced ```yaml caches: redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) allow_auto_topic_creation: true ``` A cache that stores data in a Kafka topic. This cache is useful for data that is written frequently and queried infrequently. Reads from the cache require scanning the entire topic partition. If you expect frequent access, consider placing an in-memory caching layer in front of this one. Because only the latest values are needed, configure compaction for topics used as caches so that reads are less expensive when topics are rescanned. See [Compaction Settings](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings/). The cache does not have any TTL mechanisms. Use the Kafka topic retention policies to manage TTL. ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#conn_idle_timeout)`conn_idle_timeout` The amount of time that connections can remain idle before they are closed. **Type**: `string` **Default**: `20s` ### [](#metadata_max_age)`metadata_max_age` The maximum age of metadata before it is refreshed. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. This buffer helps prevent premature timeouts, especially for requests that already define their own timeout values. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more SASL authentication methods. Each method is tried in the order specified. If the broker supports the first mechanism, outgoing client connections use that mechanism. If the first mechanism fails, the client will use the first supported mechanism. If the broker does not support any client mechanisms, connections will fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when [`sasl.mechanism`](#sasl-mechanism) is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. For more information, see the [credentials for AWS](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/) guide. **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` The credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming the specified role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM-based authentication as specified by the aws-msk-iam-auth Java library. | | OAUTHBEARER | OAuth Bearer authentication. | | PLAIN | PLAIN mechanism for plaintext password authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM authentication as specified in RFC5802. | | none | Disable SASL authentication. | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to. Items containing commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to store data in. **Type**: `string` --- # Page 239: ristretto **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/ristretto.md --- # ristretto > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: ristretto latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/ristretto page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/ristretto.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/ristretto.adoc description: Stores key/value pairs in a map held in the memory-bound Ristretto cache. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores key/value pairs in a map held in the memory-bound [Ristretto cache](https://github.com/dgraph-io/ristretto). #### Common ```yml caches: ristretto: default_ttl: "" ``` #### Advanced ```yml caches: ristretto: default_ttl: "" get_retries: enabled: false initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` This cache is more efficient and appropriate for high-volume use cases than the standard memory cache. However, the add command is non-atomic, and therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#default_ttl)`default_ttl` A default TTL to set for items, calculated from the moment the item is cached. Set to an empty string or zero duration to disable TTLs. **Type**: `string` **Default**: `""` ```yaml # Examples: default_ttl: 5m # --- default_ttl: 60s ``` ### [](#get_retries)`get_retries` Determines how and whether get attempts should be retried if the key is not found. Ristretto is a concurrent cache that does not immediately reflect writes, and so it can sometimes be useful to enable retries at the cost of speed in cases where the key is expected to exist. **Type**: `object` ### [](#get_retries-enabled)`get_retries.enabled` Whether retries should be enabled. **Type**: `bool` **Default**: `false` ### [](#get_retries-initial_interval)`get_retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#get_retries-max_elapsed_time)`get_retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#get_retries-max_interval)`get_retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 240: sql **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/sql.md --- # sql > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/sql page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/sql.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/sql.adoc description: Uses an SQL database table as a destination for storing cache key/value items. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- Uses an SQL database table as a destination for storing cache key/value items. #### Common ```yml caches: sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) ``` #### Advanced ```yml caches: sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` Each cache key/value pair will exist as a row within the specified table. Currently only the key and value columns are set, and therefore any other columns present within the target table must allow NULL values if this cache is going to be used for set and add operations. Cache operations are translated into SQL statements as follows: ## [](#get)Get All `get` operations are performed with a traditional `select` statement. ## [](#delete)Delete All `delete` operations are performed with a traditional `delete` statement. ## [](#set)Set The `set` operation is performed with a traditional `insert` statement. This will behave as an `add` operation by default, and so ideally needs to be adapted in order to provide updates instead of failing on collision s. Since different SQL engines implement upserts differently it is necessary to specify a `set_suffix` that modifies an `insert` statement in order to perform updates on conflict. ## [](#add)Add The `add` operation is performed with a traditional `insert` statement. ## [](#fields)Fields ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#key_column)`key_column` The name of a column to be used for storing cache item keys. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: key_column: foo ``` ### [](#set_suffix)`set_suffix` An optional suffix to append to each insert query for a cache `set` operation. This should modify an insert statement into an upsert appropriate for the given SQL engine. **Type**: `string` ```yaml # Examples: set_suffix: ON DUPLICATE KEY UPDATE bar=VALUES(bar) # --- set_suffix: ON CONFLICT (foo) DO UPDATE SET bar=excluded.bar # --- set_suffix: ON CONFLICT (foo) DO NOTHING ``` ### [](#table)`table` The table to insert/read/delete cache items. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#value_column)`value_column` The name of a column to be used for storing cache item values. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: value_column: bar ``` --- # Page 241: ttlru **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/ttlru.md --- # ttlru > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: ttlru latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/caches/ttlru page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/caches/ttlru.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/caches/ttlru.adoc description: Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts. #### Common ```yml caches: ttlru: cap: 1024 default_ttl: 5m0s init_values: {} ``` #### Advanced ```yml caches: ttlru: cap: 1024 default_ttl: 5m0s ttl: "" # No default (optional) init_values: {} optimistic: false ``` The cache ttlru provides a simple, goroutine safe, cache with a fixed number of entries. Each entry has a per-cache defined TTL. This TTL is reset on both modification and access of the value. As a result, if the cache is full, and no items have expired, when adding a new item, the item with the soonest expiration will be evicted. It uses the package [`expirable`](https://github.com/hashicorp/golang-lru/tree/main/expirable) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache ttlru: default_ttl: '5m' cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. ## [](#fields)Fields ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1024` ### [](#default_ttl)`default_ttl` The cache ttl of each element **Type**: `string` **Default**: `5m0s` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `object` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The ttlru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#ttl)`ttl` Deprecated. Please use `default_ttl` field **Type**: `string` --- # Page 242: Inputs **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about.md --- # Inputs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Inputs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- An input is a source of data piped through an array of optional [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/): ```yaml input: label: my_redis_input redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group # Optional list of processing steps processors: - mapping: | root.document = this.without("links") root.link_count = this.links.length() ``` Some inputs have a logical end, when this happens the input gracefully terminates and Redpanda Connect will shut itself down once all messages have been processed fully. It’s also possible to specify a logical end for an input that otherwise doesn’t have one with the [`read_until` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/read_until/), which checks a condition against each consumed message in order to determine whether it should be the last. ## [](#brokering)Brokering Only one input is configured at the root of a Redpanda Connect config. However, the root input can be a [broker](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/broker/) which combines multiple inputs and merges the streams: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup - redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group ``` ## [](#labels)Labels Inputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple inputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/). ### [](#sequential-reads)Sequential reads Sometimes it’s useful to consume a sequence of inputs, where an input is only consumed once its predecessor is drained fully, you can achieve this with the [`sequence` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/). ## [](#generating-messages)Generating messages It’s possible to generate data with Redpanda Connect using the [`generate` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/generate/), which is also a convenient way to trigger scheduled pipelines. --- # Page 243: amqp_0_9 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/amqp_0_9.md --- # amqp_0_9 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: amqp_0_9 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/amqp_0_9 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/amqp_0_9.adoc description: Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml inputs: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) consumer_tag: "" prefetch_count: 10 ``` #### Advanced ```yml inputs: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) queue_declare: enabled: false durable: true auto_delete: false arguments: "" # No default (optional) bindings_declare: [] # No default (optional) consumer_tag: "" auto_ack: false nack_reject_patterns: [] prefetch_count: 10 prefetch_size: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `amqp_content_type` - `amqp_content_encoding` - `amqp_delivery_mode` - `amqp_priority` - `amqp_correlation_id` - `amqp_reply_to` - `amqp_expiration` - `amqp_message_id` - `amqp_timestamp` - `amqp_type` - `amqp_user_id` - `amqp_app_id` - `amqp_consumer_tag` - `amqp_delivery_tag` - `amqp_redelivered` - `amqp_exchange` - `amqp_routing_key` - All existing message headers, including nested headers prefixed with the key of their respective parent. You can access these metadata fields using [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_ack)`auto_ack` Set to `true` to automatically acknowledge messages as soon as they are consumed rather than waiting for acknowledgments from downstream. This can improve throughput and prevent the pipeline from becoming blocked, but delivery guarantees are lost. **Type**: `bool` **Default**: `false` ### [](#bindings_declare)`bindings_declare[]` Passively declares the bindings of the target queue to make sure they exist and are configured correctly. If the bindings exist, then the passive declaration verifies that fields specified in this object match them. **Type**: `array` ```yaml # Examples: bindings_declare: - exchange: foo key: bar ``` ### [](#bindings_declare-exchange)`bindings_declare[].exchange` The exchange of the declared binding. **Type**: `string` **Default**: `""` ### [](#bindings_declare-key)`bindings_declare[].key` The key of the declared binding. **Type**: `string` **Default**: `""` ### [](#consumer_tag)`consumer_tag` A consumer tag to uniquely identify the consumer. **Type**: `string` **Default**: `""` ### [](#nack_reject_patterns)`nack_reject_patterns[]` A list of regular expression patterns to match against errors in messages that Redpanda Connect fails to deliver. When a message has an error that matches a pattern, it is dropped or delivered to a dead-letter queue (if a queue has been configured). By default, failed messages are negatively acknowledged (nacked) and requeued. **Type**: `array` **Default**: `[]` ```yaml # Examples: nack_reject_patterns: - "^reject me please:.+$" ``` ### [](#prefetch_count)`prefetch_count` The maximum number of pending messages at a given time. **Type**: `int` **Default**: `10` ### [](#prefetch_size)`prefetch_size` The maximum size of pending messages (in bytes) at a given time. **Type**: `int` **Default**: `0` ### [](#queue)`queue` An AMQP queue to consume from. **Type**: `string` ### [](#queue_declare)`queue_declare` Passively declares the [target queue](#queue) to make sure a queue with the specified name exists and is configured correctly. If the queue exists, then the passive declaration verifies that fields specified in this object match the its properties. **Type**: `object` ### [](#queue_declare-arguments)`queue_declare.arguments` Arguments for server-specific implementations of the queue (optional). You can use arguments to configure additional parameters for queue types that require them. For more information about available arguments, see the [RabbitMQ Client Library](https://github.com/rabbitmq/amqp091-go/blob/b3d409fe92c34bea04d8123a136384c85e8dc431/types.go#L282-L362). | Argument | Description | Accepted values | | --- | --- | --- | | x-queue-type | Declares the type of queue. | Options: classic (default), quorum, stream, drop-head, reject-publish, and reject-publish-dlx. | | x-max-length | The maximum number of messages in the queue. | A non-negative integer. | | x-max-length-bytes | The maximum size of messages (in bytes) in the queue. | A non-negative integer. | | x-overflow | Sets the queue’s overflow behavior. | Options: drop-head (default), reject-publish, reject-publish-dlx. | | x-message-ttl | The duration (in milliseconds) that messages remain in the queue before they expire and are discarded. | A string that represents the number of milliseconds. For example, 60000 retains messages for one minute. | | x-expires | The duration after which the queue automatically expires. | A positive integer. | | x-max-age | The duration (in configurable units) that streamed messages are retained on disk before they are discarded. | Options: Y, M, D, h, m, s. For example, 7D retains messages for a week. | | x-stream-max-segment-size-bytes | The maximum size (in bytes) of the segment files held on disk. | A positive integer. Default: 500000000 (approximately 500 MB). | | x-queue-version | The version of the classic queue to use. | Options: 1 or 2. | | x-consumer-timeout | The duration (in milliseconds) that a consumer can remain idle before it is automatically canceled. | A positive integer that represents the number of milliseconds. For example, 60000 sets a timeout duration of one minute. | | x-single-active-consumer | When set to true, a single consumer receives messages from the queue even when multiple consumers are subscribed to it. | A boolean. | **Type**: `object` ```yaml # Examples: arguments: x-max-length: 1000 x-max-length-bytes: 4096 x-queue-type: quorum ``` ### [](#queue_declare-auto_delete)`queue_declare.auto_delete` Whether the declared queue auto-deletes when there are no active consumers. **Type**: `bool` **Default**: `false` ### [](#queue_declare-durable)`queue_declare.durable` Whether the declared queue is durable. **Type**: `bool` **Default**: `true` ### [](#queue_declare-enabled)`queue_declare.enabled` Whether to enable queue declaration. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. This input attempts to connect to each URL in the list, in order, until a successful connection is established. It then continues to use that URL until the connection is closed. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 244: aws_cloudwatch_logs **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_cloudwatch_logs.md --- # aws_cloudwatch_logs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_cloudwatch_logs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/aws_cloudwatch_logs page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/aws_cloudwatch_logs.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/aws_cloudwatch_logs.adoc description: Consumes log events from AWS CloudWatch Logs. page-git-created-date: "2026-03-13" page-git-modified-date: "2026-08-11" --- Consumes log events from AWS CloudWatch Logs. #### Common ```yml inputs: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s limit: 1000 structured_log: true api_timeout: 30s auto_replay_nacks: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Polls CloudWatch Log Groups for log events. Supports filtering by log streams, CloudWatch filter patterns, and configurable start times. Each log event becomes a separate message with metadata including the log group name, log stream name, timestamp, and ingestion time. > ❗ **IMPORTANT** > > This input provides at-least-once delivery. It tracks its position in memory only, so if the process restarts, it resumes from the configured `start_time` (or the beginning if not set). Duplicates can occur across restarts. For exactly-once outcomes, implement idempotent or deduplicated downstream processing. ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `cloudwatch_log_group`: The name of the log group. - `cloudwatch_log_stream`: The name of the log stream. - `cloudwatch_timestamp`: The timestamp of the log event (Unix milliseconds). - `cloudwatch_ingestion_time`: The ingestion timestamp (Unix milliseconds). - `cloudwatch_event_id`: The unique event ID. You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#api_timeout)`api_timeout` The maximum time to wait for an API request to complete. **Type**: `string` **Default**: `30s` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#filter_pattern)`filter_pattern` An optional CloudWatch Logs filter pattern to apply when querying log events. For syntax details, see the [CloudWatch Logs filter and pattern syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) documentation. **Type**: `string` ```yaml # Examples: filter_pattern: [ERROR] ``` ### [](#limit)`limit` The maximum number of log events to return in a single API call. Valid range: 1-10000. **Type**: `int` **Default**: `1000` ### [](#log_group_name)`log_group_name` The name of the CloudWatch Log Group to consume from. **Type**: `string` ```yaml # Examples: log_group_name: my-app-logs ``` ### [](#log_stream_names)`log_stream_names[]` An optional list of log stream names to consume from. If not set, events from all streams in the log group will be consumed. **Type**: `array` ```yaml # Examples: log_stream_names: - stream-1 - stream-2 ``` ### [](#log_stream_prefix)`log_stream_prefix` An optional log stream name prefix to filter streams. Only streams starting with this prefix will be consumed. **Type**: `string` ```yaml # Examples: log_stream_prefix: prod- ``` ### [](#poll_interval)`poll_interval` The interval at which to poll for new log events. **Type**: `string` **Default**: `5s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#start_time)`start_time` The time to start consuming log events from. Can be an RFC3339 timestamp (for example, `2024-01-01T00:00:00Z`) or the string `now` to start consuming from the current time. If not set, starts from the beginning of available logs. **Type**: `string` ```yaml # Examples: start_time: 2024-01-01T00:00:00Z # --- start_time: now ``` ### [](#structured_log)`structured_log` Whether to output log events as structured JSON objects with all metadata fields, or as plain text messages with metadata stored in Redpanda Connect message metadata. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 245: aws_dynamodb_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_dynamodb_cdc.md --- # aws_dynamodb_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_dynamodb_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/aws_dynamodb_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/aws_dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/aws_dynamodb_cdc.adoc description: Reads change data capture (CDC) events from DynamoDB Streams. page-topic-type: reference personas: data_engineer, streaming_developer, platform_operator learning-objective-1: Look up configuration options for DynamoDB CDC streaming learning-objective-2: Find metadata fields available for message processing learning-objective-3: Identify checkpointing and performance tuning settings page-git-created-date: "2026-03-04" page-git-modified-date: "2026-08-11" --- Stream item-level changes from DynamoDB tables using DynamoDB Streams. This input automatically manages shards, checkpoints progress for recovery, and processes multiple shards concurrently. Use this reference to: - Look up configuration options for DynamoDB CDC streaming - Find metadata fields available for message processing - Identify checkpointing and performance tuning settings ### Common ```yml inputs: label: "" aws_dynamodb_cdc: tables: [] checkpoint_table: redpanda_dynamodb_checkpoints checkpoint_namespace: "" start_from: trim_horizon snapshot_mode: none ``` ### Advanced ```yml inputs: label: "" aws_dynamodb_cdc: tables: [] table_discovery_mode: single table_tag_filter: "" table_discovery_interval: 5m checkpoint_table: redpanda_dynamodb_checkpoints checkpoint_namespace: "" global_table: false global_table_replicas: [] batch_size: 1000 poll_interval: 1s start_from: trim_horizon checkpoint_limit: 1000 max_tracked_shards: 10000 throttle_backoff: 100ms snapshot_mode: none snapshot_segments: 1 snapshot_batch_size: 100 snapshot_throttle: 100ms snapshot_deduplicate: true snapshot_buffer_size: 100000 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#prerequisites)Prerequisites The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled. You can enable streams with one of these view types: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images ## [](#checkpointing)Checkpointing Checkpoints are stored in a separate DynamoDB table (configured via `checkpoint_table`). This table is created automatically if it does not exist. On restart, the input resumes from the last checkpointed position for each shard. ### [](#share-a-checkpoint-table-across-pipelines)Share a checkpoint table across pipelines By default, checkpoints are keyed by stream and shard only, so multiple pipelines that read the same stream and share one `checkpoint_table` overwrite each other’s positions, which causes skipped or duplicated events. This is common when several developers or environments test against the same table. Set `checkpoint_namespace` to give each pipeline its own isolated set of checkpoints within a shared table. Redpanda Connect prefixes the namespace to the checkpoint key, so pipelines with different namespaces never collide: ```yaml # Alice's pipeline input: aws_dynamodb_cdc: tables: [ orders ] region: us-east-1 checkpoint_table: shared_dynamodb_checkpoints checkpoint_namespace: dev-alice start_from: trim_horizon ``` ```yaml # Bob's pipeline: same stream and checkpoint table, isolated by namespace input: aws_dynamodb_cdc: tables: [ orders ] region: us-east-1 checkpoint_table: shared_dynamodb_checkpoints checkpoint_namespace: dev-bob start_from: trim_horizon ``` The `checkpoint_namespace` field requires Redpanda Connect 4.101.0 or later. It is backward compatible: leaving it unset (the default) keeps the existing checkpoint keys unchanged, so existing deployments are unaffected. The value cannot contain a `#` character. > 📝 **NOTE** > > A namespace provides isolation, not coordination. Two pipelines that use the same `checkpoint_namespace` and `checkpoint_table` still overwrite each other’s checkpoints. To run readers independently, give each one a distinct namespace. > ⚠️ **WARNING** > > Changing or removing `checkpoint_namespace` makes the pipeline read checkpoints under the new key. If no checkpoints exist yet for that namespace, for example the first time you set it or when you switch to a namespace that has never been used, the pipeline has nothing to resume from and starts at `start_from` (with `start_from: trim_horizon`, this replays all records still available in the stream’s 24-hour retention window). Switching back to a previously used namespace resumes from that namespace’s last checkpoints. ## [](#alternative-components)Alternative components For better performance and longer retention (up to 1 year vs 24 hours), consider using Kinesis Data Streams for DynamoDB with the `aws_kinesis` input instead. ## [](#message-structure)Message structure Each CDC event is delivered as a JSON message with the following structure. Use these fields in your Bloblang mappings with `this.`: ```json { "eventID": "abc123-", (1) "eventName": "INSERT | MODIFY | REMOVE", (2) "eventSource": "aws:dynamodb", "awsRegion": "us-east-1", "tableName": "my-table", (3) "dynamodb": { "keys": { (4) "pk": "user#123", "sk": "profile" }, "newImage": { (5) "pk": "user#123", "sk": "profile", "name": "Alice", "email": "alice@example.com" }, "oldImage": { (6) "pk": "user#123", "sk": "profile", "name": "Alice Smith" }, "sequenceNumber": "12345678901234567890", (7) "sizeBytes": 256, "streamViewType": "NEW_AND_OLD_IMAGES" } } ``` | 1 | Unique identifier for this change event. | | --- | --- | | 2 | Type of change: INSERT (new item), MODIFY (updated item), or REMOVE (deleted item). | | 3 | Name of the source DynamoDB table. | | 4 | Primary key attributes of the changed item. Always present. | | 5 | Item state after the change. Present for INSERT and MODIFY events (requires NEW_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 6 | Item state before the change. Present for MODIFY and REMOVE events (requires OLD_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 7 | Position of this record in the shard, used for ordering and checkpointing. | > 📝 **NOTE** > > DynamoDB attribute values are automatically unmarshalled from DynamoDB’s type format (`{"S": "value"}`) to plain values (`"value"`). ### [](#example-mapping)Example mapping ```yaml pipeline: processors: - mapping: | root.event_type = this.eventName root.table = this.tableName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `dynamodb_shard_id`: The shard ID from which the record was read - `dynamodb_sequence_number`: The sequence number of the record in the stream - `dynamodb_event_name`: The type of change: INSERT, MODIFY, or REMOVE - `dynamodb_table`: The name of the DynamoDB table ## [](#metrics)Metrics This input emits the following metrics: - `dynamodb_cdc_shards_tracked`: Total number of shards being tracked (gauge) - `dynamodb_cdc_shards_active`: Number of shards currently being read from (gauge) ## [](#fields)Fields ### [](#batch_size)`batch_size` Maximum number of records to read per shard in a single request. Valid range: 1-1000. **Type**: `int` **Default**: `1000` ### [](#checkpoint_limit)`checkpoint_limit` Maximum number of unacknowledged messages before forcing a checkpoint update. Lower values provide better recovery guarantees but increase write overhead. **Type**: `int` **Default**: `1000` ### [](#checkpoint_namespace)`checkpoint_namespace` Isolates this pipeline’s checkpoints within a shared `checkpoint_table` by prefixing the namespace to the checkpoint key. Use this so that multiple pipelines reading the same stream can share one checkpoint table without overwriting each other’s positions, for example per-developer or per-environment test pipelines. Leave empty (the default) to keep the original checkpoint keys unchanged. A namespace isolates readers but does not coordinate them: pipelines that share the same namespace still collide. Changing or removing the namespace changes the checkpoint key. If no checkpoints exist yet under the new key, the pipeline starts from `start_from`. Switching back to a previously used namespace resumes from that namespace’s last checkpoints. The value cannot contain a `#` character. **Type**: `string` **Default**: `""` ### [](#checkpoint_table)`checkpoint_table` DynamoDB table name for storing checkpoints. Will be created if it doesn’t exist. **Type**: `string` **Default**: `redpanda_dynamodb_checkpoints` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#global_table)`global_table` Provision the checkpoint table as a DynamoDB Global Table (v2) so checkpoints replicate across regions. Requires `global_table_replicas`. When the table is auto-created it is created as a global table; when it already exists, its replicas are reconciled (missing regions are added by calling `UpdateTable`). The existing table must have been created in global mode (`TableId` hash key). Enabling this against a pre-existing non-global checkpoint table fails fast with a clear error. **Type**: `bool` **Default**: `false` ### [](#global_table_replicas)`global_table_replicas[]` Regions other than this pipeline’s own region to replicate the checkpoint table to. The pipeline’s own region is always included. Required when `global_table` is true. Applied both when the checkpoint table is created and, for an existing global table, when reconciling replicas (missing regions are added; this list is not used to remove regions). **Type**: `array` **Default**: `[]` ### [](#max_tracked_shards)`max_tracked_shards` Maximum number of shards to track simultaneously. Prevents memory issues with extremely large tables. **Type**: `int` **Default**: `10000` ### [](#poll_interval)`poll_interval` Time to wait between polling attempts when no records are available. **Type**: `string` **Default**: `1s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#snapshot_batch_size)`snapshot_batch_size` Records per scan request during snapshot. Maximum 1000. Lower values provide better backpressure control but require more API calls. **Type**: `int` **Default**: `100` ### [](#snapshot_buffer_size)`snapshot_buffer_size` Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted. **Type**: `int` **Default**: `100000` ### [](#snapshot_deduplicate)`snapshot_deduplicate` Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot. If buffer is exceeded, deduplication is disabled to prevent data loss. **Type**: `bool` **Default**: `true` ### [](#snapshot_mode)`snapshot_mode` `none`: Streams CDC events only (default). `snapshot_only`: Performs a one-time full table scan with no ongoing streaming. `snapshot_and_cdc`: Scans the entire table, then streams changes. **Type**: `string` **Default**: `none` **Options**: `none`, `snapshot_only`, `snapshot_and_cdc` ### [](#snapshot_segments)`snapshot_segments` Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more Read Capacity Units (RCUs). A lower value is safer to start with. **Type**: `int` **Default**: `1` ### [](#snapshot_throttle)`snapshot_throttle` Minimum time between scan requests per segment. Use this to limit Read Capacity Unit (RCU) consumption during snapshot. **Type**: `string` **Default**: `100ms` ### [](#start_from)`start_from` Where to start reading when no checkpoint exists. `trim_horizon` starts from the oldest available record, `latest` starts from new records. **Type**: `string` **Default**: `trim_horizon` **Options**: `trim_horizon`, `latest` ### [](#table_discovery_interval)`table_discovery_interval` Interval for rescanning and discovering new tables when using `tag` or `includelist` mode. Set to 0 to disable periodic rescanning. **Type**: `string` **Default**: `5m` ### [](#table_discovery_mode)`table_discovery_mode` `single`: Streams from tables specified in the `tables` list. `tag`: Auto-discovers tables by tags (ignores the `tables` field). `includelist`: Streams from tables in the `tables` list. Use `single` instead; `includelist` is kept for backward compatibility. **Type**: `string` **Default**: `single` **Options**: `single`, `tag`, `includelist` ### [](#table_tag_filter)`table_tag_filter` Multi-tag filter in the format `key1:v1,v2;key2:v3,v4`. Matches tables where (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4). Required when `table_discovery_mode` is `tag`. **Type**: `string` **Default**: `""` ### [](#tables)`tables[]` List of table names to stream from. For single table mode, provide one table. For multi-table mode, provide multiple tables. **Type**: `array` **Default**: `[]` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#throttle_backoff)`throttle_backoff` Time to wait when applying backpressure due to too many in-flight messages. **Type**: `string` **Default**: `100ms` ## [](#examples)Examples ### [](#consume-cdc-events)Consume CDC events Read change events from a DynamoDB table with streams enabled. ```yaml input: aws_dynamodb_cdc: tables: [my-table] region: us-east-1 ``` ### [](#start-from-latest)Start from latest Only process new changes, ignoring existing stream data. ```yaml input: aws_dynamodb_cdc: tables: [orders] start_from: latest region: us-west-2 ``` ### [](#snapshot-and-cdc)Snapshot and CDC Scan all existing records, then stream ongoing changes. ```yaml input: aws_dynamodb_cdc: tables: [products] snapshot_mode: snapshot_and_cdc snapshot_segments: 5 region: us-east-1 ``` ### [](#auto-discover-tables-by-tag)Auto-discover tables by tag Automatically discover and stream from all tables with a specific tag. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "stream-enabled:true" table_discovery_interval: 5m region: us-east-1 ``` ### [](#auto-discover-tables-by-multiple-tags)Auto-discover tables by multiple tags Discover tables matching multiple tag criteria with OR logic per key, AND logic across keys. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "environment:prod,staging;team:data,analytics" table_discovery_interval: 5m region: us-east-1 # Matches tables with: (environment=prod OR environment=staging) AND (team=data OR team=analytics) ``` ### [](#stream-from-multiple-specific-tables)Stream from multiple specific tables Stream from an explicit list of tables simultaneously. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: includelist tables: - orders - customers - products region: us-west-2 ``` ## [](#suggested-reading)Suggested reading For common patterns including filtering events, routing to Kafka or S3, and detecting changed fields, see the [DynamoDB CDC Patterns](https://docs.redpanda.com/connect/cookbooks/dynamodb_cdc/) cookbook. --- # Page 246: aws_kinesis **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_kinesis.md --- # aws_kinesis > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_kinesis latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/aws_kinesis page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/aws_kinesis.adoc description: Receive messages from one or more Kinesis streams. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Receive messages from one or more Kinesis streams. #### Common ```yml inputs: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false billing_mode: PAY_PER_REQUEST read_capacity_units: 0 write_capacity_units: 0 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s start_from_oldest: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false billing_mode: PAY_PER_REQUEST read_capacity_units: 0 write_capacity_units: 0 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s rebalance_period: 30s lease_period: 30s start_from_oldest: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Consumes messages from one or more Kinesis streams either by automatically balancing shards across other instances of this input, or by consuming shards listed explicitly. The latest message sequence consumed by this input is stored within a [DynamoDB table](#table-schema), which allows it to resume at the correct sequence of the shard during restarts. This table is also used for coordination across distributed inputs when shard balancing. Redpanda Connect will not store a consumed sequence unless it is acknowledged at the output level, which ensures at-least-once delivery guarantees. ## [](#ordering)Ordering By default messages of a shard can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#table-schema)Table schema It’s possible to configure Redpanda Connect to create the DynamoDB table required for coordination if it does not already exist. However, if you wish to create this yourself (recommended) then create a table with a string HASH key `StreamID` and a string RANGE key `ShardID`. ## [](#batching)Batching Use the `batching` fields to configure an optional [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/#batch-policy). Each stream shard will be batched separately in order to ensure that acknowledgements aren’t contaminated. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum gap between the in flight sequence versus the latest acknowledged sequence at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual shards. Any given sequence will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#commit_period)`commit_period` The period of time between each update to the checkpoint table. **Type**: `string` **Default**: `5s` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#dynamodb)`dynamodb` Determines the table used for storing and accessing the latest consumed sequence for shards, and for coordinating balanced consumers of streams. **Type**: `object` ### [](#dynamodb-billing_mode)`dynamodb.billing_mode` When creating the table determines the billing mode. **Type**: `string` **Default**: `PAY_PER_REQUEST` **Options**: `PROVISIONED`, `PAY_PER_REQUEST` ### [](#dynamodb-create)`dynamodb.create` Whether, if the table does not exist, it should be created. **Type**: `bool` **Default**: `false` ### [](#dynamodb-credentials)`dynamodb.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#dynamodb-credentials-from_ec2_role)`dynamodb.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#dynamodb-credentials-id)`dynamodb.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#dynamodb-credentials-profile)`dynamodb.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#dynamodb-credentials-role)`dynamodb.credentials.role` The role ARN to assume. **Type**: `string` ### [](#dynamodb-credentials-role_external_id)`dynamodb.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#dynamodb-credentials-secret)`dynamodb.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#dynamodb-credentials-token)`dynamodb.credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#dynamodb-endpoint)`dynamodb.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#dynamodb-read_capacity_units)`dynamodb.read_capacity_units` Set the provisioned read capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#dynamodb-region)`dynamodb.region` The AWS region to target. **Type**: `string` ### [](#dynamodb-table)`dynamodb.table` The name of the table to access. **Type**: `string` **Default**: `""` ### [](#dynamodb-tcp)`dynamodb.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#dynamodb-tcp-connect_timeout)`dynamodb.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-tcp-keep_alive)`dynamodb.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#dynamodb-tcp-keep_alive-count)`dynamodb.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#dynamodb-tcp-keep_alive-idle)`dynamodb.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-keep_alive-interval)`dynamodb.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-tcp_user_timeout)`dynamodb.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-write_capacity_units)`dynamodb.write_capacity_units` Set the provisioned write capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#lease_period)`lease_period` The period of time after which a client that has failed to update a shard checkpoint is assumed to be inactive. **Type**: `string` **Default**: `30s` ### [](#rebalance_period)`rebalance_period` The period of time between each attempt to rebalance shards across clients. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#start_from_oldest)`start_from_oldest` Whether to consume from the oldest message when a sequence does not yet exist for the stream. **Type**: `bool` **Default**: `true` ### [](#steal_grace_period)`steal_grace_period` Determines how long beyond the next commit period a client will wait when stealing a shard for the current owner to store a checkpoint. A longer value increases the time taken to balance shards but reduces the likelihood of processing duplicate messages. **Type**: `string` **Default**: `2s` ### [](#streams)`streams[]` One or more Kinesis data streams to consume from. Streams can either be specified by their name or full ARN. Shards of a stream are automatically balanced across consumers by coordinating through the provided DynamoDB table. Multiple comma separated streams can be listed in a single element. Shards are automatically distributed across consumers of a stream by coordinating through the provided DynamoDB table. Alternatively, it’s possible to specify an explicit shard to consume from with a colon after the stream name, e.g. `foo:0` would consume the shard `0` of the stream `foo`. **Type**: `array` ```yaml # Examples: streams: - foo - "arn:aws:kinesis:*:111122223333:stream/my-stream" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 247: aws_s3 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_s3.md --- # aws_s3 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_s3 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/aws_s3 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/aws_s3.adoc description: Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in realtime. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in real time. #### Common ```yml inputs: label: "" aws_s3: bucket: "" prefix: "" scanner: to_the_end: {} sqs: url: "" endpoint: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" delay_period: "" max_messages: 10 wait_time_seconds: 0 nack_visibility_timeout: 0 ``` #### Advanced ```yml inputs: label: "" aws_s3: bucket: "" prefix: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) force_path_style_urls: false delete_objects: false scanner: to_the_end: {} sqs: url: "" endpoint: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" delay_period: "" max_messages: 10 wait_time_seconds: 0 nack_visibility_timeout: 0 ``` ## [](#stream-objects-on-upload-with-sqs)Stream objects on upload with SQS A common pattern for consuming S3 objects is to emit upload notification events from the bucket either directly to an SQS queue, or to an SNS topic that is consumed by an SQS queue, and then have your consumer listen for events that prompt it to download the newly uploaded objects. More information about this pattern and how to set it up can be found in the [Amazon S3 docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html). Redpanda Connect is able to follow this pattern when you configure an `sqs.url`, where it consumes events from SQS and downloads only the object keys contained in those events. For this to work, Redpanda Connect needs to know where within the event the key and bucket names can be found, specified as [dot paths](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) with the fields `sqs.key_path` and `sqs.bucket_path`. The default values for these fields should already be correct when following the guide above. If your notification events are being routed to SQS via an SNS topic, the events are enveloped by SNS, in which case you also need to specify the field `sqs.envelope_path`, which in the case of SNS to SQS will usually be `Message`. When using SQS, make sure you have sensible values for `sqs.max_messages` and also the visibility timeout of the queue itself. When Redpanda Connect consumes an S3 object the SQS message that triggered it is not deleted until the S3 object has been sent onwards. This ensures at-least-once crash resiliency, but also means that if the S3 object takes longer to process than the visibility timeout of your queue, then the same objects might be processed multiple times. ## [](#download-large-files)Download large files When downloading large files, process them in streamed parts to avoid loading the entire file into memory at once. To do this, specify a [`scanner`](#scanner) that determines how to break the input into smaller individual messages. ## [](#bucket-and-prefix)Bucket and prefix The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. The `prefix` field accepts a single string. To consume from multiple prefixes in the same bucket, use multiple `aws_s3` inputs in a [`broker` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/broker/): ```yaml input: broker: inputs: - aws_s3: bucket: my-bucket prefix: logs/app1/ - aws_s3: bucket: my-bucket prefix: logs/app2/ ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml input: aws_s3: bucket: r2-bucket endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `s3_key` - `s3_bucket` - `s3_last_modified_unix` - `s3_last_modified` (RFC3339) - `s3_content_type` - `s3_content_encoding` - `s3_version_id` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). Note that user defined metadata is case insensitive within AWS, and it is likely that the keys will be received in a capitalized form, if you wish to make them consistent you can map all metadata keys to lower or uppercase using a Bloblang mapping such as `meta = meta().map_each_key(key → key.lowercase())`. ## [](#fields)Fields ### [](#bucket)`bucket` The bucket to consume from. If the field `sqs.url` is specified this field is optional. **Type**: `string` **Default**: `""` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the bucket once they are processed. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs for downloading keys, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed when walking a bucket. **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#sqs)`sqs` Consume SQS messages in order to trigger key downloads. **Type**: `object` ### [](#sqs-bucket_path)`sqs.bucket_path` A [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) whereby the bucket name can be found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.bucket.name` ### [](#sqs-delay_period)`sqs.delay_period` An optional period of time to wait from when a notification was originally sent to when the target key download is attempted. **Type**: `string` **Default**: `""` ```yaml # Examples: delay_period: 10s # --- delay_period: 5m ``` ### [](#sqs-endpoint)`sqs.endpoint` A custom endpoint to use when connecting to SQS. **Type**: `string` **Default**: `""` ### [](#sqs-envelope_path)`sqs.envelope_path` A [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) of a field to extract an enveloped JSON payload for further extracting the key and bucket from SQS messages. This is specifically useful when subscribing an SQS queue to an SNS topic that receives bucket events. **Type**: `string` **Default**: `""` ```yaml # Examples: envelope_path: Message ``` ### [](#sqs-key_path)`sqs.key_path` A [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) whereby object keys are found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.object.key` ### [](#sqs-max_messages)`sqs.max_messages` The maximum number of SQS messages to consume from each request. **Type**: `int` **Default**: `10` ### [](#sqs-nack_visibility_timeout)`sqs.nack_visibility_timeout` Custom SQS Nack Visibility timeout in seconds. Default is 0 **Type**: `int` **Default**: `0` ### [](#sqs-url)`sqs.url` An optional SQS URL to connect to. When specified this queue will control which objects are downloaded. **Type**: `string` **Default**: `""` ### [](#sqs-wait_time_seconds)`sqs.wait_time_seconds` Whether to set the wait time. Enabling this activates long-polling. Valid values: 0 to 20. **Type**: `int` **Default**: `0` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 248: aws_sqs **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_sqs.md --- # aws_sqs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_sqs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/aws_sqs page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/aws_sqs.adoc description: Consume messages from an AWS SQS URL. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consume messages from an AWS SQS URL. #### Common ```yml inputs: label: "" aws_sqs: url: "" # No default (required) max_outstanding_messages: 1000 ``` #### Advanced ```yml inputs: label: "" aws_sqs: url: "" # No default (required) delete_message: true reset_visibility: true max_number_of_messages: 10 max_outstanding_messages: 1000 wait_time_seconds: 0 message_timeout: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level, which allows you to transfer data across accounts. To find out more, see [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sqs_message_id` - `sqs_receipt_handle` - `sqs_approximate_receive_count` - All message attributes You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delete_message)`delete_message` Whether to delete the consumed message when it’s acknowledged. Set to `false` to handle the deletion using a different mechanism. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_number_of_messages)`max_number_of_messages` The maximum number of messages that Redpanda Connect can return each time it polls the SQS URL. Enter values from `1` to `10` only. **Type**: `int` **Default**: `10` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of pending messages that Redpanda Connect can have in flight at the same time. **Type**: `int` **Default**: `1000` ### [](#message_timeout)`message_timeout` The maximum time allowed to process a received message before Redpanda Connect refreshes the [receipt handle](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-message-identifiers.html), and the message becomes visible in the queue again. Redpanda Connect attempts to refresh the receipt handle after half of the timeout has elapsed. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#reset_visibility)`reset_visibility` Whether to set the visibility timeout of the consumed message to zero if Redpanda Connect receives a negative acknowledgement. Set to `false` to use the [queue’s visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) for each message rather than releasing the message immediately for reprocessing. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The SQS URL to consume from. **Type**: `string` ### [](#wait_time_seconds)`wait_time_seconds` Whether to set a wait time (in seconds). Enter values from `1` to `20` to enable wait times and to activate [log polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) for queued messages. **Type**: `int` **Default**: `0` --- # Page 249: azure_blob_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/azure_blob_storage.md --- # azure_blob_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_blob_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/azure_blob_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/azure_blob_storage.adoc description: Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. #### Common ```yml inputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} targets_input: "" # No default (optional) ``` #### Advanced ```yml inputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} delete_objects: false targets_input: "" # No default (optional) ``` Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#download-large-files)Download large files When downloading large files it’s often necessary to process it in streamed parts in order to avoid loading the entire file in memory at a given time. In order to do this a [`scanner`](#scanner) can be specified that determines how to break the input into smaller individual messages. ## [](#stream-new-files)Stream new files By default this input will consume all files found within the target container and will then gracefully terminate. This is referred to as a "batch" mode of operation. However, it’s possible to instead configure a container as [an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and then use this as a [`targets_input`](#targets_input), in which case new files are consumed as they’re uploaded and Redpanda Connect will continue listening for and downloading files as they arrive. This is referred to as a "streamed" mode of operation. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `blob_storage_key` - `blob_storage_container` - `blob_storage_last_modified` - `blob_storage_last_modified_unix` - `blob_storage_content_type` - `blob_storage_content_encoding` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#container)`container` The name of the container from which to download blobs. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the blob once they are processed. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#targets_input)`targets_input` > ⚠️ **CAUTION** > > This is an experimental field that provides an optional source of download targets, configured as a [regular Redpanda Connect input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/). Each message yielded by this input should be a single structured object containing a field `name`, which represents the blob to be downloaded. This requires setting up [Azure Blob Storage as an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and an associated event handler that a Redpanda Connect input can read from. For example, use either one of the following: - [Azure Event Hubs](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-hubs) using the `kafka` input - [Namespace topics](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-grid-namespace-topic) using the `mqtt` input **Type**: `input` ```yaml # Examples: targets_input: mqtt: topics: - some-topic urls: - example.westeurope-1.ts.eventgrid.azure.net:8883 processors: - unarchive: format: json_array - mapping: |- if this.eventType == "Microsoft.Storage.BlobCreated" { root.name = this.data.url.parse_url().path.trim_prefix("/foocontainer/") } else { root = deleted() } ``` --- # Page 250: azure_cosmosdb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/azure_cosmosdb.md --- # azure_cosmosdb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_cosmosdb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/azure_cosmosdb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/azure_cosmosdb.adoc description: Executes a SQL query against Azure CosmosDB and creates a batch of messages from each page of items. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a SQL query against [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction) and creates a batch of messages from each page of items. ### Common ```yml inputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml inputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) batch_count: -1 auto_replay_nacks: true ``` ## [](#cross-partition-queries)Cross-partition queries Cross-partition queries are currently not supported by the underlying driver. For every query, the PartitionKey values must be known in advance and specified in the config. [See details](https://github.com/Azure/azure-sdk-for-go/issues/18578#issuecomment-1222510989). ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: - `activity_id` - `request_charge` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#query-container)Query container Execute a parametrized SQL query to select documents from a container. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish AS b WHERE b.species = @species args_mapping: | root = [ { "Name": "@species", "Value": "smooth-head" }, ] ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that, for each message, creates a list of arguments to use with the query. **Type**: `string` ```yaml # Examples: args_mapping: |- root = [ { "Name": "@name", "Value": "benthos" }, ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` The maximum number of messages that should be accumulated into each batch. Use '-1' specify dynamic page size. **Type**: `int` **Default**: `-1` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = now().ts_format("2006-01-02") ``` ### [](#query)`query` The query to execute **Type**: `string` ```yaml # Examples: query: SELECT c.foo FROM testcontainer AS c WHERE c.bar = "baz" AND c.timestamp < @timestamp ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 251: azure_queue_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/azure_queue_storage.md --- # azure_queue_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_queue_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/azure_queue_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/azure_queue_storage.adoc description: Dequeue objects from an Azure Storage Queue. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Dequeue objects from an Azure Storage Queue. #### Common ```yml inputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) dequeue_visibility_timeout: 30s max_in_flight: 10 track_properties: false ``` This input adds the following metadata fields to each message: ```none - queue_storage_insertion_time - queue_storage_queue_name - queue_storage_message_lag (if 'track_properties' set to true) - All user defined queue metadata ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. ## [](#fields)Fields ### [](#dequeue_visibility_timeout)`dequeue_visibility_timeout` The timeout duration until a dequeued message gets visible again, 30s by default **Type**: `string` **Default**: `30s` ### [](#max_in_flight)`max_in_flight` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `10` ### [](#queue_name)`queue_name` The name of the source storage queue. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: queue_name: foo_queue # --- queue_name: ${! env("MESSAGE_TYPE").lowercase() } ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#track_properties)`track_properties` If set to `true` the queue is polled on each read request for information such as the queue message lag. These properties are added to consumed messages as metadata, but will also have a negative performance impact. **Type**: `bool` **Default**: `false` --- # Page 252: azure_table_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/azure_table_storage.md --- # azure_table_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_table_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/azure_table_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/azure_table_storage.adoc description: Queries an Azure Storage Account Table, optionally with multiple filters. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Queries an Azure Storage Account Table, optionally with multiple filters. #### Common ```yml inputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) filter: "" select: "" page_size: 1000 ``` Queries an Azure Storage Account Table, optionally with multiple filters. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `table_storage_name` - `row_num` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#filter)`filter` OData filter expression. Is not set all rows are returned. Valid operators are `eq, ne, gt, lt, ge and le` **Type**: `string` **Default**: `""` ```yaml # Examples: filter: PartitionKey eq 'foo' and RowKey gt '1000' ``` ### [](#page_size)`page_size` Maximum number of records to return on each page. **Type**: `int` **Default**: `1000` ### [](#select)`select` Select expression using OData notation. Limits the columns on each record to just those requested. **Type**: `string` **Default**: `""` ```yaml # Examples: select: PartitionKey,RowKey,Foo,Bar,Timestamp ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to read messages from. **Type**: `string` ```yaml # Examples: table_name: Foo ``` --- # Page 253: batched **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/batched.md --- # batched > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: batched latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/batched page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/batched.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/batched.adoc description: Consumes data from a child input and applies a batching policy to the stream. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consumes data from a child input and applies a batching policy to the stream. #### Common ```yml inputs: label: "" batched: child: "" # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" batched: child: "" # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Batching at the input level is sometimes useful for processing across micro-batches, and can also sometimes be a useful performance trick. However, most inputs are fine without it so unless you have a specific plan for batching this component is not worth using. ## [](#fields)Fields ### [](#child)`child` The child input. **Type**: `input` ### [](#policy)`policy` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: policy: byte_size: 5000 count: 0 period: 1s # --- policy: count: 10 period: 1s # --- policy: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#policy-byte_size)`policy.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#policy-check)`policy.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#policy-count)`policy.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#policy-period)`policy.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#policy-processors)`policy.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` --- # Page 254: broker **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/broker.md --- # broker > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: broker latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/broker page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/broker.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/broker.adoc description: Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel. #### Common ```yml inputs: label: "" broker: inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" broker: copies: 1 inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` A broker type is configured with its own list of input configurations and a field to specify how many copies of the list of inputs should be created. Adding more input types allows you to combine streams from multiple sources into one. For example, reading from both RabbitMQ and Kafka: ```yaml input: broker: copies: 1 inputs: - amqp_0_9: urls: - amqp://guest:guest@localhost:5672/ consumer_tag: benthos-consumer queue: benthos-queue # Optional list of input specific processing steps processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() - kafka: addresses: - localhost:9092 client_id: benthos_kafka_input consumer_group: benthos_consumer_group topics: [ benthos_stream:0 ] ``` If the number of copies is greater than zero the list will be copied that number of times. For example, if your inputs were of type foo and bar, with 'copies' set to '2', you would end up with two 'foo' inputs and two 'bar' inputs. ## [](#batching)Batching It’s possible to configure a [batch policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/#batch-policy) with a broker using the `batching` fields. When doing this the feeds from all child inputs are combined. Some inputs do not support broker based batching and specify this in their documentation. ## [](#processors)Processors It is possible to configure [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) at the broker level, where they will be applied to _all_ child inputs, as well as on the individual child inputs. If you have processors at both the broker level _and_ on child inputs then the broker processors will be applied _after_ the child nodes processors. ## [](#fields)Fields ### [](#batching-2)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` Whatever is specified within `inputs` will be created this many times. **Type**: `int` **Default**: `1` ### [](#inputs)`inputs[]` A list of inputs to create. **Type**: `array` --- # Page 255: gateway **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gateway.md --- # gateway > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gateway latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/gateway page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/gateway.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/gateway.adoc description: Receive messages delivered over HTTP. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-08-11" --- The `gateway` input is a Cloud-only component that receives messages over HTTP and injects them into a running Redpanda Connect pipeline. It’s ideal for: - Receiving webhook events from third-party services - Accepting real-time telemetry or sensor data over HTTP - Building lightweight ingest endpoints for client apps For on-premises or self-managed deployments, use the [`http_server`](https://docs.redpanda.com/connect/components/inputs/http_server/) input instead. This component is fully managed and available in the following Redpanda Cloud deployment types: - **Serverless** - **Dedicated** - **Bring Your Own Cloud (BYOC)** When a pipeline with a `gateway` input is deployed, Redpanda Cloud provisions a secure URL that you can use to send HTTP requests. You can post raw payloads, JSON messages, or stream events in real time. Authentication and access control are handled through standard Redpanda Cloud API tokens. For more information, see [Cloud API Authentication](https://docs.redpanda.com/api/doc/cloud-dataplane/authentication). Network access: - On **public clusters** (Serverless and Dedicated), the gateway URL is accessible over the public internet. - On **private clusters** (BYOC), the gateway is accessible only from within your configured VPC. #### Common ```yaml input: label: "" gateway: path: / rate_limit: "" ``` #### Advanced ```yaml input: label: "" gateway: path: / rate_limit: "" sync_response: status: "200" headers: Content-Type: application/octet-stream metadata_headers: include_prefixes: [] include_patterns: [] ``` The field `rate_limit` allows you to specify an optional [`rate_limit` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) that applies to all HTTP requests. When the rate limit is breached, HTTP requests return a 429 response with a Retry-After header. ## [](#responses)Responses You can also return a response for each message received using [synchronous responses](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses/). When doing so, you can customize headers using the `sync_response.headers` field, which supports [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) in the value based on the response message contents. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `http_server_user_agent` - `http_server_request_path` - `http_server_verb` - `http_server_remote_ip` - All headers (only first values are taken) - All query parameters - All path parameters - All cookies You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#path)`path` The endpoint path to listen for data delivery requests. **Type**: `string` **Default**: `/` ### [](#rate_limit)`rate_limit` An optional [rate limit](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle requests by. **Type**: `string` **Default**: `""` ### [](#sync_response)`sync_response` Customize messages returned using [synchronous responses](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses/). **Type**: `object` ### [](#sync_response-headers)`sync_response.headers` Specify headers to return with synchronous responses. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: ```yaml Content-Type: "application/octet-stream" ``` ### [](#sync_response-metadata_headers)`sync_response.metadata_headers` Specify criteria for which metadata values are added to the response as headers. **Type**: `object` ### [](#sync_response-metadata_headers-include_patterns)`sync_response.metadata_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#sync_response-metadata_headers-include_prefixes)`sync_response.metadata_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#sync_response-status)`sync_response.status` Specify the status code to return with synchronous responses. This is a string value, which allows you to customize it based on resulting payloads and their metadata. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `200` ```yaml # Examples: status: ${! json("status") } # --- status: ${! meta("status") } ``` ### [](#tcp)`tcp` Customize messages returned via [synchronous responses](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses/). **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#ingest-a-real-time-stream-of-sensor-data)Ingest a real-time stream of sensor data Use the `gateway` input to stream telemetry data from edge devices or browser clients that connect over HTTP. Suppose a client connects and sends JSON-encoded sensor readings like this: ```json { "sensor_id": "temp-001", "value": 22.5, "unit": "C" } { "sensor_id": "temp-001", "value": 22.8, "unit": "C" } { "sensor_id": "temp-001", "value": 23.1, "unit": "C" } ``` Redpanda Connect treats each line as an individual message. The following pipeline sets up a `gateway` input to handle these connections and logs each message: ```yaml input: label: sensor_stream gateway: path: /ws/sensors rate_limit: "" pipeline: processors: - log: level: INFO message: "Received reading from ${! json(\"sensor_id\") }: ${! json(\"value\") } ${! json(\"unit\") }" ``` This configuration: - Accepts HTTP connections on `/ws/sensors` - Receives a stream of messages over a single connection - Logs each message using Bloblang interpolation You can replace the `log` processor with any downstream output, such as Redpanda or Amazon S3, to persist or analyze the data in real time. --- # Page 256: gcp_bigquery_select **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gcp_bigquery_select.md --- # gcp_bigquery_select > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_bigquery_select latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/gcp_bigquery_select page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/gcp_bigquery_select.adoc description: Executes a SELECT query against BigQuery and creates a message for each row received. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a `SELECT` query against BigQuery and creates a message for each row received. ```yml inputs: label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) auto_replay_nacks: true job_labels: {} priority: "" args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` Once the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to execute). ## [](#examples)Examples ### [](#word-counts)Word counts Here we query the public corpus of Shakespeare’s works to generate a stream of the top 10 words that are 3 or more characters long: ```yaml input: gcp_bigquery_select: project: sample-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: length(word) >= ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: | root = [ 3 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A list of labels to add to the query job. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#priority)`priority` The priority with which to schedule the query. **Type**: `string` **Default**: `""` ### [](#project)`project` GCP project where the query job will execute. **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` Fully-qualified BigQuery table name to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 257: gcp_cloud_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gcp_cloud_storage.md --- # gcp_cloud_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_cloud_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/gcp_cloud_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/gcp_cloud_storage.adoc description: Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix. #### Common ```yml inputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} ``` #### Advanced ```yml inputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} delete_objects: false ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `gcs_key` - `gcs_bucket` - `gcs_last_modified` - `gcs_last_modified_unix` - `gcs_content_type` - `gcs_content_encoding` - All user defined metadata You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#bucket)`bucket` The name of the bucket from which to download objects. **Type**: `string` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the bucket once they are processed. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` Optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 258: gcp_pubsub **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gcp_pubsub.md --- # gcp_pubsub > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_pubsub latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/gcp_pubsub page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/gcp_pubsub.adoc description: Consumes messages from a GCP Cloud Pub/Sub subscription. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consumes messages from a GCP Cloud Pub/Sub subscription. #### Common ```yml inputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 ``` #### Advanced ```yml inputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 create_subscription: enabled: false topic: "" ``` For information on how to set up credentials see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `gcp_pubsub_publish_time_unix` - The time at which the message was published to the topic. - `gcp_pubsub_delivery_attempt` - When dead lettering is enabled, this is set to the number of times PubSub has attempted to deliver a message. - `gcp_pubsub_message_id` - The unique identifier of the message. - `gcp_pubsub_ordering_key` - The ordering key of the message. - All message attributes You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#create_subscription)`create_subscription` Allows you to configure the input subscription and creates if it doesn’t exist. **Type**: `object` ### [](#create_subscription-enabled)`create_subscription.enabled` Whether to configure subscription or not. **Type**: `bool` **Default**: `false` ### [](#create_subscription-topic)`create_subscription.topic` Defines the topic that the subscription should be vinculated to. **Type**: `string` **Default**: `""` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#endpoint)`endpoint` An optional endpoint to override the default of `pubsub.googleapis.com:443`. This can be used to connect to a region specific pubsub endpoint. For a list of valid values, see [this document](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#max_outstanding_bytes)`max_outstanding_bytes` The maximum number of outstanding pending messages to be consumed measured in bytes. **Type**: `int` **Default**: `1000000000` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of outstanding pending messages to be consumed at a given time. **Type**: `int` **Default**: `1000` ### [](#project)`project` The project ID of the target subscription. **Type**: `string` ### [](#subscription)`subscription` The target subscription ID. **Type**: `string` ### [](#sync)`sync` Enable synchronous pull mode. **Type**: `bool` **Default**: `false` --- # Page 259: gcp_spanner_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gcp_spanner_cdc.md --- # gcp_spanner_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_spanner_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/gcp_spanner_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/gcp_spanner_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/gcp_spanner_cdc.adoc description: Creates an input that consumes from a spanner change stream. page-git-created-date: "2025-07-08" page-git-modified-date: "2026-08-11" --- Creates an input that consumes from a spanner change stream. #### Common ```yaml inputs: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yaml inputs: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" heartbeat_interval: 10s metadata_table: "" min_watermark_cache_ttl: 5s allowed_mod_types: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true ``` Consumes change records from a Google Cloud Spanner change stream. This input allows you to track and process database changes in real-time, making it useful for data replication, event-driven architectures, and maintaining derived data stores. The input reads from a specified change stream within a Spanner database and converts each change record into a message. The message payload contains the change records in JSON format, and metadata is added with details about the Spanner instance, database, and stream. Change streams provide a way to track mutations to your Spanner database tables. For more information about Spanner change streams, refer to the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams). ## [](#fields)Fields ### [](#allowed_mod_types)`allowed_mod_types[]` List of modification types to process. If not specified, all modification types are processed. Allowed values: INSERT, UPDATE, DELETE **Type**: `array` ```yaml # Examples: allowed_mod_types: - INSERT - UPDATE - DELETE ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is passed on for processing or delivery (flushed). When the combined size of all messages in the batch exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `batching.count` or `batching.period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch should be flushed. Set the value to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch should be flushed regardless of its size. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. For example, `1s` flushes a batch after one second. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so any attempt to split it into smaller batches with these processors will be ignored. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials_json)`credentials_json` Base64-encoded JSON credentials file for authenticating to GCP with a service account. If not provided, Application Default Credentials (ADC) is used. For more information about how to create a service account and obtain the credentials JSON, see the [Google Cloud documentation](https://cloud.google.com/docs/authentication/getting-started). **Type**: `string` **Default**: `""` ### [](#database_id)`database_id` The ID of the Spanner database to read from. This is the name of the database as it appears in the Spanner console or API. For more information about how to create a Spanner database, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-databases). **Type**: `string` ### [](#end_timestamp)`end_timestamp` The timestamp at which to stop reading change records from the change stream. This is an optional field that allows you to limit the range of change records processed by the input. The timestamp should be in RFC3339 format, such as `2023-10-01T00:00:00Z`. If not provided, the input reads all available change records up to the current time. **Type**: `string` **Default**: `""` ```yaml # Examples: end_timestamp: 2022-01-01T00:00:00Z ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval at which to send heartbeat messages to the output. Heartbeat messages are sent to indicate that the input is still active and processing changes. This can help prevent timeouts in downstream systems. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. For example, `1s` sends a heartbeat every second. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` The ID of the Spanner instance to read from. This is the name of the instance as it appears in the Spanner console or API. For more information about how to create a Spanner instance, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-instances). **Type**: `string` ### [](#metadata_table)`metadata_table` The table to store metadata in (default: `cdc_metadata_`). **Type**: `string` **Default**: `""` ### [](#min_watermark_cache_ttl)`min_watermark_cache_ttl` Sets how frequently to query Spanner for the minimum watermark. **Type**: `string` **Default**: `5s` ### [](#project_id)`project_id` The ID of the GCP project that contains the Spanner instance and database. This is the name of the project as it appears in the GCP console or API. For more information about how to create a GCP project, see the [Google Cloud documentation](https://cloud.google.com/resource-manager/docs/creating-managing-projects). **Type**: `string` ### [](#start_timestamp)`start_timestamp` The timestamp at which to start reading change records from the change stream. This is an optional field that allows you to limit the range of change records processed by the input. The timestamp should be in RFC3339 format, such as `2023-10-01T00:00:00Z` (default: current time). **Type**: `string` **Default**: `""` ```yaml # Examples: start_timestamp: 2022-01-01T00:00:00Z ``` ### [](#stream_id)`stream_id` The name of the change stream to track. The stream must exist in the Spanner database. To create a change stream, follow the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams/manage). **Type**: `string` --- # Page 260: generate **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/generate.md --- # generate > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: generate latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/generate page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/generate.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/generate.adoc description: Generates messages at a given interval using a Bloblang mapping executed without a context. This allows you to generate messages for testing your pipeline configs. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates messages at a given interval using a [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) mapping executed without a context. This allows you to generate messages for testing your pipeline configs. #### Common ```yml inputs: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#cron-scheduled-processing)Cron Scheduled Processing A common use case for the generate input is to trigger processors on a schedule so that the processors themselves can behave similarly to an input. The following configuration reads rows from a PostgreSQL table every 5 minutes. ```yaml input: generate: interval: '@every 5m' mapping: 'root = {}' processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: foo columns: [ "*" ] ``` ### [](#generate-100-rows)Generate 100 Rows The generate input can be used as a convenient way to generate test data. The following example generates 100 rows of structured data by setting an explicit count. The interval field is set to empty, which means data is generated as fast as the downstream components can consume it. ```yaml input: generate: count: 100 interval: "" mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "is yummy" } } else { { "type": "bar", "bar": "is gross" } } ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` The number of generated messages that should be accumulated into each batch flushed at the specified interval. **Type**: `int` **Default**: `1` ### [](#count)`count` An optional number of messages to generate, if set above 0 the specified number of messages is generated and then the input will shut down. **Type**: `int` **Default**: `0` ### [](#interval)`interval` The time interval at which messages should be generated, expressed either as a duration string or as a cron expression. If set to an empty string messages will be generated as fast as downstream services can process them. Cron expressions can specify a timezone by prefixing the expression with `TZ=`, where the location name corresponds to a file within the IANA Time Zone database. **Type**: `string` **Default**: `1s` ```yaml # Examples: interval: 5s # --- interval: 1m # --- interval: 1h # --- interval: @every 1s # --- interval: 0,30 */2 * * * * # --- interval: TZ=Europe/London 30 3-6,20-23 * * * ``` ### [](#mapping)`mapping` A [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) mapping to use for generating messages. **Type**: `string` ```yaml # Examples: mapping: root = "hello world" # --- mapping: root = {"test":"message","id":uuid_v4()} ``` --- # Page 261: git **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/git.md --- # git > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: git latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/git page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/git.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/git.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Clones a Git repository, reads its contents, then polls for new commits at a configurable interval. Any updates are emitted as new messages. ```yml inputs: label: "" git: repository_url: "" # No default (required) branch: main poll_interval: 10s include_patterns: [] exclude_patterns: [] max_file_size: 10485760 checkpoint_cache: "" # No default (optional) checkpoint_key: git_last_commit auth: basic: username: "" password: "" ssh_key: private_key_path: "" private_key: "" passphrase: "" token: value: "" auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `git_file_path` - `git_file_size` - `git_file_mode` - `git_file_modified` - `git_commit` - `git_mime_type` - `git_is_binary` - `git_encoding` (present if the file was base64 encoded) - `git_deleted` (only present if the file was deleted) You can access these metadata fields using function interpolation. ## [](#fields)Fields ### [](#auth)`auth` Options for authenticating with your Git repository. **Type**: `object` ### [](#auth-basic)`auth.basic` Allows you to specify basic authentication. **Type**: `object` ### [](#auth-basic-password)`auth.basic.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#auth-basic-username)`auth.basic.username` The username to use for authentication. **Type**: `string` **Default**: `""` ### [](#auth-ssh_key)`auth.ssh_key` Allows you to specify SSH key authentication. **Type**: `object` ### [](#auth-ssh_key-passphrase)`auth.ssh_key.passphrase` The passphrase for your SSH private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key)`auth.ssh_key.private_key` Your private SSH key. When using encrypted keys, you must also set a value for [`private_key_passphrase`](#auth-ssh_key-passphrase). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key_path)`auth.ssh_key.private_key_path` The path to your private SSH key file. When using encrypted keys, you must also set a value for [`private_key_passphrase`](#auth-ssh_key-passphrase). **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Allows you to specify token-based authentication. **Type**: `object` ### [](#auth-token-value)`auth.token.value` The token value to use for token-based authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#branch)`branch` The repository branch to check out. **Type**: `string` **Default**: `main` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) resource to store the last processed commit hash. After a restart, Redpanda Connect can then continue processing changes from where it left off, avoiding the need to reprocess all detected updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key to use when storing the last processed commit hash in the cache. **Type**: `string` **Default**: `git_last_commit` ### [](#exclude_patterns)`exclude_patterns[]` A list of file patterns to exclude. For example, you could choose not to read content from certain Git directories or image files: `'.git/**', '**/*.png'`. These patterns take precedence over `include_patterns`. The following patterns are supported: - Glob patterns: **, `/`**`*/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. **Type**: `array` **Default**: `[]` ### [](#include_patterns)`include_patterns[]` A list of file patterns to read from. For example, you could read content from only Markdown and YAML files: `'***/**.md', 'configs/*.yaml'`. The following patterns are supported: - Glob patterns: **, `/`**`*/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. If this field is left empty, all files are read from. **Type**: `array` **Default**: `[]` ### [](#max_file_size)`max_file_size` The maximum size of files to read from (in bytes). Files that exceed this limit are skipped. Set to `0` for unlimited file sizes. **Type**: `int` **Default**: `10485760` ### [](#poll_interval)`poll_interval` How frequently this input polls the Git repository for changes. **Type**: `string` **Default**: `10s` ```yaml # Examples: poll_interval: 10s ``` ### [](#repository_url)`repository_url` The URL of the Git repository to clone. **Type**: `string` ```yaml # Examples: repository_url: https://github.com/username/repo.git ``` --- # Page 262: http_client **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/http_client.md --- # http_client > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: http_client latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/http_client page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/http_client.adoc page-git-created-date: "2025-03-04" page-git-modified-date: "2026-05-26" --- Connects to a server and continuously requests single messages. #### Common ```yml inputs: label: "" http_client: url: "" # No default (required) verb: GET headers: {} rate_limit: "" # No default (optional) timeout: 5s payload: "" # No default (optional) stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" http_client: url: "" # No default (required) verb: GET headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false payload: "" # No default (optional) drop_empty_bodies: true stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). You can also add [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) to the [`url`](#url) and [`headers`](#headers) fields to implement basic pagination, such as page numbers or tokens, where subsequent requests need to include data from previously-consumed responses. Example: ```yaml input: http_client: url: >- https://api.example.com/search?query=allmyfoos&start_time=${! ( (timestamp_unix()-300).ts_format("2006-01-02T15:04:05Z","UTC").escape_url_query() ) }${! ("&next_token="+this.meta.next_token.not_null()) | "" } verb: GET rate_limit: schedule_searches oauth2: enabled: true token_url: https://api.example.com/oauth2/token client_key: "${EXAMPLE_KEY}" client_secret: "${EXAMPLE_SECRET}" rate_limit_resources: - label: schedule_searches local: count: 1 interval: 30s ``` > 💡 **TIP** > > If pagination requires more complex logic, consider using the [`http` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/http/) combined with a [`generate` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/generate/), which allows you to schedule the processor. ## [](#streaming-messages)Streaming messages If you [enable streaming](#stream-enabled), Redpanda Connect consumes the body of the server response as a continuous stream of data, and breaks the stream down into smaller, logical messages using the [specified scanner](#stream-scanner). This functionality allows you to consume APIs that provide long-lived streamed data feeds, such as stock market feeds. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure, and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. **Type**: `bool` **Default**: `false` ### [](#drop_empty_bodies)`drop_empty_bodies` Whether to drop empty payloads received from the target server. **Type**: `bool` **Default**: `true` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure, where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether or not to transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys to add to the HTTP request as headers (optional). **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#payload)`payload` A payload to deliver for each request (optional). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#stream)`stream` Enables streaming mode, where the HTTP connection remains open and messages are processed line-by-line. **Type**: `object` ### [](#stream-enabled)`stream.enabled` Enables streaming mode. **Type**: `bool` **Default**: `false` ### [](#stream-reconnect)`stream.reconnect` Whether to automatically reestablish the HTTP connection if it is lost. **Type**: `bool` **Default**: `true` ### [](#stream-scanner)`stream.scanner` The [scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `GET` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 263: http_server **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/http_server.md --- # http_server > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: http_server latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/http_server page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/http_server.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/http_server.adoc description: Receive messages POSTed over HTTP(S). HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. page-git-created-date: "2026-02-18" page-git-modified-date: "2026-05-26" --- Receive messages sent over HTTP using POST requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. #### Common ```yml inputs: label: "" http_server: address: "" path: /post ws_path: /post/ws allowed_verbs: - "POST" timeout: 5s rate_limit: "" ``` #### Advanced ```yml inputs: label: "" http_server: address: "" path: /post ws_path: /post/ws ws_welcome_message: "" ws_rate_limit_message: "" allowed_verbs: - "POST" timeout: 5s rate_limit: "" cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] sync_response: status: 200 headers: Content-Type: "application/octet-stream" metadata_headers: include_prefixes: [] include_patterns: [] tcp: reuse_addr: false reuse_port: false ``` The field `rate_limit` allows you to specify an optional [`rate_limit` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/), which will be applied to each HTTP request made and each websocket payload received. When the rate limit is breached HTTP requests will have a 429 response returned with a Retry-After header. Websocket payloads will be dropped and an optional response payload will be sent as per `ws_rate_limit_message`. ## [](#responses)Responses It’s possible to return a response for each message received using [synchronous responses](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses/). When doing so you can customize headers with the `sync_response` field `headers`, which can also use [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) in the value based on the response message contents. ## [](#endpoints)Endpoints The following fields specify endpoints that are registered for sending messages, and support path parameters of the form `/{foo}`, which are added to ingested messages as metadata. A path ending in `/` will match against all extensions of that path: ### [](#path-defaults-to-post)`path` (defaults to `/post`) This endpoint expects POST requests where the entire request body is consumed as a single message. If the request contains a multipart `content-type` header as per [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html) then the multiple parts are consumed as a batch of messages, where each body part is a message of the batch. ### [](#ws_path-defaults-to-postws)`ws_path` (defaults to `/post/ws`) Creates a websocket connection, where payloads received on the socket are passed through the pipeline as a batch of one message. > ⚠️ **CAUTION: Endpoint caveats** > > Endpoint caveats > > Components within a Redpanda Connect config will register their respective endpoints in a non-deterministic order. This means that establishing precedence of endpoints that are registered via multiple `http_server` inputs or outputs (either within brokers or from cohabiting streams) is not possible in a predictable way. > > This ambiguity makes it difficult to ensure that paths which are both a subset of a path registered by a separate component, and end in a slash (`/`) and will therefore match against all extensions of that path, do not prevent the more specific path from matching against requests. > > It is therefore recommended that you ensure paths of separate components do not collide unless they are explicitly non-competing. > > For example, if you were to deploy two separate `http_server` inputs, one with a path `/foo/` and the other with a path `/foo/bar`, it would not be possible to ensure that the path `/foo/` does not swallow requests made to `/foo/bar`. You may specify an optional `ws_welcome_message`, which is a static payload to be sent to all clients once a websocket connection is first established. It’s also possible to specify a `ws_rate_limit_message`, which is a static payload to be sent to clients that have triggered the servers rate limit. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `http_server_user_agent` - `http_server_request_path` - `http_server_verb` - `http_server_remote_ip` - All headers (only first values are taken) - All query parameters - All path parameters - All cookies If HTTPS is enabled, the following fields are added as well: - `http_server_tls_version` - `http_server_tls_subject` - `http_server_tls_cipher_suite` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#path-switching)Path Switching This example shows an `http_server` input that captures all requests and processes them by switching on that path: ```yaml input: http_server: path: / allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - switch: - check: '@http_server_request_path == "/foo"' processors: - mapping: | root.title = "You Got Fooed!" root.result = content().string().uppercase() - check: '@http_server_request_path == "/bar"' processors: - mapping: 'root.title = "Bar Is Slow"' - sleep: # Simulate a slow endpoint duration: 1s ``` ### [](#mock-oauth-2-0-server)Mock OAuth 2.0 Server This example shows an `http_server` input that mocks an OAuth 2.0 Client Credentials flow server at the endpoint `/oauth2_test`: ```yaml input: http_server: path: /oauth2_test allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - log: message: "Received request" level: INFO fields_mapping: | root = @ root.body = content().string() - mapping: | root.access_token = "MTQ0NjJkZmQ5OTM2NDE1ZTZjNGZmZjI3" root.token_type = "Bearer" root.expires_in = 3600 - sync_response: {} - mapping: 'root = deleted()' ``` ## [](#fields)Fields ### [](#address)`address` An alternative address to host from. If left empty the service wide address is used. **Type**: `string` **Default**: `""` ### [](#allowed_verbs)`allowed_verbs[]` An array of verbs that are allowed for the `path` endpoint. **Type**: `array` **Default**: ```yaml - "POST" ``` ### [](#cert_file)`cert_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#cors)`cors` Adds Cross-Origin Resource Sharing headers. Only valid with a custom `address`. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#key_file)`key_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#path)`path` The endpoint path to listen for POST requests. **Type**: `string` **Default**: `/post` ### [](#rate_limit)`rate_limit` An optional [rate limit](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle requests by. **Type**: `string` **Default**: `""` ### [](#sync_response)`sync_response` Customize messages returned via [synchronous responses](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses/). **Type**: `object` ### [](#sync_response-headers)`sync_response.headers` Specify headers to return with synchronous responses. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: ```yaml Content-Type: "application/octet-stream" ``` ### [](#sync_response-metadata_headers)`sync_response.metadata_headers` Specify criteria for which metadata values are added to the response as headers. **Type**: `object` ### [](#sync_response-metadata_headers-include_patterns)`sync_response.metadata_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#sync_response-metadata_headers-include_prefixes)`sync_response.metadata_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#sync_response-status)`sync_response.status` Specify the status code to return with synchronous responses. This is a string value, which allows you to customize it based on resulting payloads and their metadata. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `200` ```yaml # Examples: status: ${! json("status") } # --- status: ${! meta("status") } ``` ### [](#tcp)`tcp` TCP listener configuration for the HTTP server. Only valid with a custom `address`. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#timeout)`timeout` Timeout for requests. If a consumed messages takes longer than this to be delivered the connection is closed, but the message may still be delivered. **Type**: `string` **Default**: `5s` ### [](#ws_path)`ws_path` The endpoint path to create websocket connections from. **Type**: `string` **Default**: `/post/ws` ### [](#ws_rate_limit_message)`ws_rate_limit_message` An optional message to delivery to websocket connections that are rate limited. **Type**: `string` **Default**: `""` ### [](#ws_welcome_message)`ws_welcome_message` An optional message to deliver to fresh websocket connections. **Type**: `string` **Default**: `""` --- # Page 264: inproc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/inproc.md --- # inproc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: inproc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/inproc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/inproc.adoc description: Directly connects to an output within the same Redpanda Connect process by a chosen ID, to link isolated streams when running in streams mode. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- ```yml inputs: label: "" inproc: "" ``` Directly connect to an output within a Redpanda Connect process by referencing it by a chosen ID. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 265: jira **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/jira.md --- # jira > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: jira latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/jira page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/jira.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/jira.adoc description: Streams Jira issues, comments, or changelog entries via JQL with incremental polling. page-git-created-date: "2026-07-23" page-git-modified-date: "2026-08-11" --- Streams Jira issues, comments, or changelog entries via JQL with incremental polling. Periodically queries Jira’s REST API using a JQL filter and emits one message per resource. The cursor (max issue `updated` timestamp, plus the set of issue versions already emitted at the boundary) is persisted via the configured cache resource after every fully-acknowledged page, so progress survives restarts — including mid-backfill — and boundary issues are not re-emitted on every poll. Authentication uses API token (email + token) basic auth. The `backoff` settings govern the adaptive backoff applied to 429 responses; retries of 502/503/504 responses use a fixed three-attempt policy. Each message body is the raw JSON of the resource. Metadata fields: - `jira_id` - issue key (issues) / comment ID / changelog history ID - `jira_issue_key` - parent issue key (omitted for resource=issues) - `jira_project` - project key - `jira_updated` - RFC 3339 timestamp of the resource - `jira_event_type` - "issue" / "comment" / "changelog" - `jira_self` - Jira API URL of the resource Limitations (v1): OAuth and the worklogs resource are not yet supported. For resource=comments and resource=changelog, only the first page of child resources (up to ~50 comments or ~100 changelog entries per issue update) is emitted; a WARN is logged when truncation is detected. To fetch the full child set for issues that exceed this limit, query the Jira REST API directly. #### Common ```yml inputs: label: "" jira: auth: email: "" # No default (required) api_token: "" # No default (required) resource: issues jql: "" fields: - "*all" expand: [] page_size: 50 poll_interval: 60s cursor: cache: "" # No default (required) key: "" overlap: 60s auto_replay_nacks: true base_url: "" # No default (required) timeout: 5s ``` #### Advanced ```yml inputs: label: "" jira: auth: email: "" # No default (required) api_token: "" # No default (required) resource: issues jql: "" fields: - "*all" expand: [] page_size: 50 poll_interval: 60s cursor: cache: "" # No default (required) key: "" overlap: 60s auto_replay_nacks: true base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#auth)`auth` API token authentication. **Type**: `object` ### [](#auth-api_token)`auth.api_token` Jira API token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-email)`auth.email` Email or username of the Jira account. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` Base URL of the target service (e.g., [https://api.example.com](https://api.example.com)). TLS is enabled automatically for https URLs. **Type**: `string` ### [](#cursor)`cursor` Cursor checkpoint storage. **Type**: `object` ### [](#cursor-cache)`cursor.cache` Name of a cache resource used to persist the cursor. **Type**: `string` ### [](#cursor-key)`cursor.key` Cache key. Defaults to `redpanda_connect_jira_input_`. **Type**: `string` **Default**: `""` ### [](#cursor-overlap)`cursor.overlap` Widens `updated >= cursor - overlap` to absorb minute-boundary effects. Jira JQL’s `updated` operator has minute precision, so this should be set to at least 1m to have an effect. **Type**: `string` **Default**: `60s` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#expand)`expand[]` Jira `expand` query parameter. The input automatically adds `changelog` when resource=changelog. **Type**: `array` **Default**: `[]` ### [](#fields-2)`fields[]` Jira `fields` query parameter - narrow this for throughput. **Type**: `array` **Default**: ```yaml - "*all" ``` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#jql)`jql` Jira JQL filter. The input appends an `updated >= cursor` predicate and `ORDER BY updated ASC, key ASC`. Empty means all issues visible to the principal. **Type**: `string` **Default**: `""` ### [](#page_size)`page_size` Issues per Jira page (Jira max 100). **Type**: `int` **Default**: `50` ### [](#poll_interval)`poll_interval` Time to wait between polls once the input has caught up. Minimum 10s. **Type**: `string` **Default**: `60s` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#resource)`resource` Which Jira resource to emit. **Type**: `string` **Default**: `issues` **Options**: `issues`, `comments`, `changelog` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` --- # Page 266: kafka_franz **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka_franz.md --- # kafka_franz > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: kafka_franz latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/kafka_franz page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/kafka_franz.adoc description: A Kafka input using the Franz Kafka client library. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. A Kafka input using the [Franz Kafka client library](https://github.com/twmb/franz-go). #### Common ```yml inputs: label: "" kafka_franz: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input often out-performs the traditional `kafka` input and provides more useful logs and error messages. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/) that applies to individual topic partitions in order to batch messages together before flushing them for processing. Batching can be beneficial for performance as well as useful for windowed processing, and doing so this way preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#multi_header)`multi_header` Decode headers into lists to allow the handling of multiple values with the same key. **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 267: kafka **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka.md --- # kafka > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: kafka latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/kafka page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/kafka.adoc description: Connects to Kafka brokers and consumes one or more topics. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. Connects to Kafka brokers and consumes one or more topics. #### Common ```yml inputs: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) consumer_group: "" checkpoint_limit: 1024 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" consumer_group: "" client_id: benthos instance_id: "" # No default (optional) rack_id: "" start_from_oldest: true checkpoint_limit: 1024 auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) commit_period: 1s max_processing_period: 100ms extract_tracing_map: "" # No default (optional) group: session_timeout: 10s heartbeat_interval: 3s rebalance_timeout: 60s fetch_buffer_cap: 256 multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Offsets are managed within Kafka under the specified consumer group, and partitions for each topic are automatically balanced across members of the consumer group. The Kafka input allows parallel processing of messages from different topic partitions, and messages of the same topic partition are processed with a maximum parallelism determined by the field [`checkpoint_limit`](#checkpoint_limit). To enforce ordered processing of partition messages, set the [`checkpoint_limit`](#checkpoint_limit) to `1`, which makes sure that a message is only processed after the previous message is delivered. Batching messages before processing can be enabled using the [`batching`](#batching) field, and this batching is performed per-partition such that messages of a batch will always originate from the same partition. This batching mechanism is capable of creating batches of greater size than the [`checkpoint_limit`](#checkpoint_limit), in which case the next batch will only be created upon delivery of the current one. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All existing message headers (version 0.11+) The field `kafka_lag` is the calculated difference between the high water mark offset of the partition at the time of ingestion and the current message offset. You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#ordering)Ordering By default messages of a topic partition can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#troubleshooting)Troubleshooting If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer [`kafka_franz` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka_franz/). - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable. ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages of the same topic and partition that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual partitions. Any given offset will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the connection. This field can be explicitly made empty in order to disable stored offsets for the consumed topic partitions. **Type**: `string` **Default**: `""` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_buffer_cap)`fetch_buffer_cap` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `256` ### [](#group)`group` Tuning parameters for consumer group synchronization. **Type**: `object` ### [](#group-heartbeat_interval)`group.heartbeat_interval` A period in which heartbeats should be sent out. **Type**: `string` **Default**: `3s` ### [](#group-rebalance_timeout)`group.rebalance_timeout` A period after which rebalancing is abandoned if unresolved. **Type**: `string` **Default**: `60s` ### [](#group-session_timeout)`group.session_timeout` A period after which a consumer of the group is kicked after no heartbeats. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to help brokers identify each input after restarts and prevent unnecessary rebalances. **Type**: `string` ### [](#max_processing_period)`max_processing_period` A maximum estimate for the time taken to process a message, this is used for tuning consumer group synchronization. **Type**: `string` **Default**: `100ms` ### [](#multi_header)`multi_header` Decode headers into lists to allow handling of multiple values with the same key **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#start_from_oldest)`start_from_oldest` Determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. The setting is applied when creating a new consumer group or the saved offset no longer exists. **Type**: `bool` **Default**: `true` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. Multiple comma separated topics can be listed in a single element. Partitions are automatically distributed across consumers of a topic. Alternatively, it’s possible to specify explicit partitions to consume from with a colon after the topic name, e.g. `foo:0` would consume the partition 0 of the topic foo. This syntax supports ranges, e.g. `foo:0-10` would consume partitions 0 through to 10 inclusive. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` --- # Page 268: microsoft_sql_server_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/microsoft_sql_server_cdc.md --- # microsoft_sql_server_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: microsoft_sql_server_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/microsoft_sql_server_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/microsoft_sql_server_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/microsoft_sql_server_cdc.adoc description: Enables Change Data Capture by consuming from Microsoft SQL Server's change tables. page-git-created-date: "2025-10-24" page-git-modified-date: "2026-08-11" --- Enables Change Data Capture by consuming from Microsoft SQL Server’s change tables. #### Common ```yaml inputs: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yaml inputs: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Streams changes from a Microsoft SQL Server database for Change Data Capture (CDC). Additionally, if `stream_snapshot` is set to true, then the existing data in the database is also streamed too. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `database_schema` (The database schema for the table where the message originates from) - `schema` (The table schema in benthos common schema format, compatible with processors like parquet\_encode) - `table` (Name of the table that the message originated from) - `operation` (Type of operation that generated the message: "read", "delete", "insert", or "update\_before" and "update\_after". "read" is from messages that are read in the initial snapshot phase.) - `lsn` (the Log Sequence Number in Microsoft SQL Server) ## [](#permissions)Permissions To use the default Microsoft SQL Server cache, the user must have permissions to create tables and stored procedures. Refer to [`checkpoint_cache_table_name`](#checkpoint_cache_table_name) for additional details. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to store the current Log Sequence Number (LSN) position. This enables the connector to resume from the last processed position after restarts, preventing data loss and duplicate processing. The cache stores the highest LSN that has been successfully delivered downstream. **Type**: `string` ### [](#checkpoint_cache_connection_string)`checkpoint_cache_connection_string` An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, this creates the checkpoint cache table on the remote server instead of the source database. If `checkpoint_cache` is also set, that takes precedence. **Type**: `string` ```yaml # Examples: checkpoint_cache_connection_string: sqlserver://username:password@remotehost/instance?param1=value¶m2=value ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. **Type**: `string` **Default**: `microsoft_sql_server_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The multipart identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed Log Sequence Number (LSN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire change table. **Type**: `string` **Default**: `rpcn.CdcCheckpointCache` ```yaml # Examples: checkpoint_cache_table_name: dbo.checkpoint_cache ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed concurrently before applying back pressure. Higher values enable better parallelization and batching but increase memory usage. Messages are processed in LSN order, and a given LSN is only acknowledged after all previous LSNs have been successfully delivered, ensuring at-least-once guarantees. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string for the Microsoft SQL Server database. Use the format `sqlserver://username:password@host/instance?param1=value¶m2=value`. For Windows Authentication, use `sqlserver://host/instance?trusted_connection=yes`. Include additional parameters like `TrustServerCertificate=true` for self-signed certificates or `encrypt=disable` to disable encryption. **Type**: `string` ```yaml # Examples: connection_string: sqlserver://username:password@host/instance?param1=value¶m2=value ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude from CDC streaming. Use this to filter out specific tables from the include patterns. Table names should follow the `schema.table` format. Exclude patterns are applied after include patterns, allowing you to include broad patterns while excluding specific tables. **Type**: `array` ```yaml # Examples: exclude: dbo.privatetable ``` ### [](#include)`include[]` Regular expressions for tables to include in CDC streaming. Specify table names using the format `schema.table` (such as `dbo.orders`, `sales.customers`). Each pattern is treated as a regular expression, allowing wildcards and pattern matching. All specified tables must have CDC enabled in SQL Server. **Type**: `array` ```yaml # Examples: include: dbo.products ``` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies a number of tables that will be processed in parallel during the snapshot processing stage. **Type**: `int` **Default**: `1` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows to stream in a single batch during the initial snapshot phase. Larger batch sizes can improve throughput for initial data loads but may increase memory usage. This setting only applies when `stream_snapshot` is enabled. **Type**: `int` **Default**: `1000` ### [](#stream_backoff_interval)`stream_backoff_interval` The time interval to wait between polling attempts when no new CDC data is available. For low-traffic tables, increasing this value reduces database load and network traffic. Use Go duration format like `5s`, `30s`, or `1m`. Shorter intervals provide lower latency for new changes but increase server load. **Type**: `string` **Default**: `5s` ```yaml # Examples: stream_backoff_interval: 5s # --- stream_backoff_interval: 1m ``` ### [](#stream_snapshot)`stream_snapshot` Whether to stream a snapshot of all existing data before streaming CDC changes. When enabled, the connector first queries all existing table data, then switches to streaming incremental changes from the transaction log. Set to `false` to start streaming only new changes from the current LSN position. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` --- # Page 269: mongodb_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/mongodb_cdc.md --- # mongodb_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mongodb_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/mongodb_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/mongodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/mongodb_cdc.adoc page-git-created-date: "2025-03-11" page-git-modified-date: "2026-05-26" --- Streams data changes from a MongoDB replica set, using MongoDB’s [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data updates. #### Common ```yml inputs: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 snapshot_auto_bucket_sharding: false document_mode: update_lookup json_marshal_mode: canonical app_name: benthos auto_replay_nacks: true ``` ## [](#prerequisites)Prerequisites - MongoDB version 6 or later - Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see [Redpanda Cloud Networking](https://docs.redpanda.com/cloud-data-platform/networking/). - A MongoDB database running as a [replica set](https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb) or in a [sharded cluster](https://www.mongodb.com/docs/manual/sharding/) using replica set [protocol version 1](https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion). - A MongoDB database using the [WiredTiger](https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger) storage engine. ## [](#enable-connectivity-from-cloud-based-data-sources-byoc)Enable connectivity from cloud-based data sources (BYOC) To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source. ## [](#data-capture-method)Data capture method The `mongodb_cdc` input uses [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and [required collections](#collections). For more information, see [`document_mode` options](#document_mode) and the [MongoDB documentation](https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre—​and-post-images). ## [](#data-replication)Data replication Redpanda Connect allows you to specify which [database collections](#collections) in your source database to receive changes from. You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded [operations log (oplog)](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections: 1. Records the latest oplog position. 2. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing: 1. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `false`, the internal `$splitVector` command is used to compute shards. 2. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `true`, the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. 3. This input then uses the number of connections specified in [`snapshot-parallelism`](#snapshot_parallelism) to read the selected collections. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the [`checkpoint_cache`](#checkpoint_cache). 4. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#metadata)Metadata Each message emitted by this plugin has the following metadata: - `operation`: either "insert", "replace", "delete" or "update" for changes streamed. Documents from the initial snapshot have the operation set to "read". - `collection`: the collection the document was written to. - `operation_time`: the oplog time for when this operation occurred. - `schema`: the collection schema in benthos common schema format (set as immutable metadata). Extracted from the collection’s `$jsonSchema` validator if available, otherwise inferred from the first document seen. Not present on messages where no schema could be determined (e.g. deletes without pre-images when no prior schema is cached). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates. **Type**: `string` ### [](#checkpoint_interval)`checkpoint_interval` The interval between writing checkpoints to the cache. **Type**: `string` **Default**: `5s` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the oplog position in [`checkpoint_cache`](#checkpoint_cache). If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mongodb_cdc_checkpoint` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1000` ### [](#collections)`collections[]` A list of collections to stream changes from. Specify each collection name as a separate item. **Type**: `array` ### [](#database)`database` The name of the MongoDB database to stream changes from. **Type**: `string` ### [](#document_mode)`document_mode` The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes. **Type**: `string` **Default**: `update_lookup` | Option | Summary | | --- | --- | | partial_update | In this mode update operations only have a description of the update operation, which follows the following schema: { "_id": , "operations": [ # type == set means that the value was updated like so: # root.foo."bar.baz" = "world" {"path": ["foo", "bar.baz"], "type": "set", "value":"world"}, # type == unset means that the value was deleted like so: # root.qux = deleted() {"path": ["qux"], "type": "unset", "value": null}, # type == truncatedArray means that the array at that path was truncated to value number of elements # root.array = this.array.slice(2) {"path": ["array"], "type": "truncatedArray", "value": 2} ] } | | pre_and_post_images | Uses pre and post image collection to emit the full documents for update and delete operations. To use and configure this mode see the setup steps in the ^MongoDB documentation. | | update_lookup | In this mode insert, replace and update operations have the full document emitted and deletes only have the _id field populated. Documents updates lookup the full document. This corresponds to the updateLookup option, see the ^MongoDB documentation for more information. | ### [](#json_marshal_mode)`json_marshal_mode` Controls the format used to convert a message from BSON to JSON when it is received by Redpanda Connect. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#read_batch_size)`read_batch_size` The number of documents to fetch in each message batch from MongoDB. **Type**: `int` **Default**: `1000` ### [](#read_max_wait)`read_max_wait` The maximum duration MongoDB waits to accumulate the [`read_batch_size`](#read_batch_size) documents on a change stream before returning the batch to Redpanda Connect. **Type**: `string` **Default**: `1s` ### [](#snapshot_auto_bucket_sharding)`snapshot_auto_bucket_sharding` Uses the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments: - Set this field to to `true`. - Set `stream_snapshot` to `true`. - Increase `snapshot_parallelism` to a value greater than `1`. **Type**: `bool` **Default**: `false` ### [](#snapshot_parallelism)`snapshot_parallelism` Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing. This field is only applicable when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 270: mongodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/mongodb.md --- # mongodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mongodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/mongodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/mongodb.adoc description: Executes a query and creates a message for each document received. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- Executes a query and creates a message for each document received. #### Common ```yml inputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true batch_size: "" # No default (optional) sort: "" # No default (optional) limit: "" # No default (optional) ``` #### Advanced ```yml inputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: find json_marshal_mode: canonical query: "" # No default (required) auto_replay_nacks: true batch_size: "" # No default (optional) sort: "" # No default (optional) limit: "" # No default (optional) ``` Once the documents from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to execute). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` A explicit number of documents to batch up before flushing them for processing. Must be greater than `0`. Operations: `find`, `aggregate` **Type**: `int` ```yaml # Examples: batch_size: 1000 ``` ### [](#collection)`collection` The collection to select from. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#json_marshal_mode)`json_marshal_mode` The json\_marshal\_mode setting is optional and controls the format of the output message. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#limit)`limit` An explicit maximum number of documents to return. Operations: `find` **Type**: `int` ### [](#operation)`operation` The mongodb operation to perform. **Type**: `string` **Default**: `find` **Options**: `find`, `aggregate` ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#query)`query` Bloblang expression describing MongoDB query. **Type**: `string` ```yaml # Examples: query: |- root.from = {"$lte": timestamp_unix()} root.to = {"$gte": timestamp_unix()} ``` ### [](#sort)`sort` An object specifying fields to sort by, and the respective sort order (`1` ascending, `-1` descending). Note: The driver currently appears to support only one sorting key. Operations: `find` **Type**: `object` ```yaml # Examples: sort: name: 1 # --- sort: age: -1 ``` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 271: mqtt **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/mqtt.md --- # mqtt > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mqtt latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/mqtt page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/mqtt.adoc description: Subscribe to topics on MQTT brokers. page-git-created-date: "2024-11-07" page-git-modified-date: "2026-05-26" --- Subscribe to topics on MQTT brokers. #### Common ```yml inputs: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topics: [] # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (required) qos: 1 clean_session: true auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `mqtt_duplicate` - `mqtt_qos` - `mqtt_retained` - `mqtt_topic` - `mqtt_message_id` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#clean_session)`clean_session` Set whether the connection is non-persistent. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#qos)`qos` The level of delivery guarantee to enforce. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. **Type**: `array` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` --- # Page 272: mysql_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/mysql_cdc.md --- # mysql_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mysql_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/mysql_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/mysql_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/mysql_cdc.adoc page-git-created-date: "2025-02-20" page-git-modified-date: "2026-05-26" --- Streams data changes from a MySQL database, using MySQL’s binary log to capture data updates. This input is built on the [`mysql-canal` library](https://github.com/go-mysql-org/go-mysql?tab=readme-ov-file#replication) but uses a custom approach for streaming historical data. #### Common ```yml inputs: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 stream_snapshot: "" # No default (required) max_parallel_snapshot_tables: 1 auto_replay_nacks: true checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 max_reconnect_attempts: 10 stream_snapshot: "" # No default (required) max_parallel_snapshot_tables: 1 auto_replay_nacks: true checkpoint_limit: 1024 tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (required) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `mysql_cdc` input uses MySQL’s [binary log (`binlog`)](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) to capture changes made to a MySQL database in real time and streams them to Redpanda Connect. Redpanda Connect allows you to specify which [database tables](#tables) in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from. ## [](#prerequisites)Prerequisites - MySQL version 8 or later - Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see [Redpanda Cloud Networking](https://docs.redpanda.com/cloud-data-platform/networking/). - A MySQL instance with binary logging enabled ### [](#configuration-resources)Configuration resources #### Cloud platforms - [Change data capture on Amazon RDS for MySQL](https://aws.amazon.com/blogs/database/enable-change-data-capture-on-amazon-rds-for-mysql-applications-that-are-using-xa-transactions/) - [Azure MySQL Database (CDC)](https://learn.microsoft.com/en-us/fabric/real-time-hub/add-source-mysql-database-cdc) - [Google Cloud SQL for MySQL](https://cloud.google.com/datastream/docs/configure-cloudsql-mysql) #### Self-hosted MySQL - [Binary Logging Options and Variables](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html) ## [](#choose-a-replication-mode)Choose a replication mode You can run the `mysql_cdc` input in one of two modes, depending on whether you need a snapshot of existing data. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected tables and streams the contents before processing changes from the last recorded binlog position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest binlog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MySQL database and does the following to capture a snapshot of all data in the selected tables: 1. Executes the `FLUSH TABLES WITH READ LOCK` query to write any outstanding table updates to disk, and locks the tables. 2. Runs the `START TRANSACTION WITH CONSISTENT SNAPSHOT` statement to create a new transaction with a consistent view of all data, capturing the state of the database at the moment the transaction started. 3. Reads the current binlog position. 4. Runs the `UNLOCK TABLES` statement to release the database. 5. Preserves the initial transaction for data integrity. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current binlog position in the [`checkpoint_cache`](#checkpoint_cache). After the snapshot is taken, the input executes SELECT statements to extract data from the selected tables in two stages: 1. The input finds the primary keys of a table. 2. It selects the data ordered by primary key. Finally, the input uses the stored binlog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MySQL database and starts processing data changes from the latest binlog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last binlog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#binlog-rotation)Binlog rotation While the `mysql_cdc` input is streaming changes to Redpanda Connect, your MySQL server may rotate the binlog file. When this occurs, Redpanda Connect flushes the existing message batch and stores the new binlog position so that it can resume processing using the latest offset. ## [](#data-mappings)Data mappings The following table shows how selected MySQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | MySQL data type | Bloblang value | | --- | --- | | TEXT, VARCHAR | A string value, for example: "this data" | | BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB | An array of byte values, for example: [byte1,byte2,byte3] | | DECIMAL, NUMERIC, TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT, YEAR | A standard numeric type, for example: 123 | | FLOAT, DOUBLE | A 64-bit decimal (float64), for example: 123.1234 | | DATETIME, TIMESTAMP | A Bloblang timestamp, for example:1257894000000 2009-11-10 23:00:00 +0000 UTC | | SET | An array of strings, for example: ["apple", "banana", "orange"] | | JSON | A map object of the JSON, for example: {"red": 1, "blue": 2, "green": 3} | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `operation`: The type of operation (insert, update, delete, or read for snapshot messages) - `table`: The name of the table - `binlog_position`: The binlog position (for CDC messages only, not set for snapshot messages) - `schema`: The table schema in benthos common schema format, compatible with processors like parquet\_encode ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for MySQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for MySQL. When enabled, an IAM authentication token is generated and used as the password. When using IAM authentication ensure `max_reconnect_attempts` is set to a low value to ensure it can refresh credentials. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The MySQL endpoint hostname (e.g., mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The ID of credentials to use. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the MySQL instance is located. If no region is specified then the environment default will be used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID for the role assumption. Only used with the `role` field. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles can be assumed in sequence, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#aws-token)`aws.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` Specify a `cache` resource to store the binlog position of the most recent data update delivered to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this last known position, avoiding the need to reprocess all table updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the binlog position in [`checkpoint_cache`](#checkpoint_cache). If you have multiple `mysql_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mysql_binlog_position` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given binlog position is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the MySQL database from which you want to stream updates. Use the format `user:password@tcp(localhost:3306)/database`. **Type**: `string` ```yaml # Examples: dsn: user:password@tcp(localhost:3306)/database ``` ### [](#flavor)`flavor` The type of MySQL database to connect to. **Type**: `string` **Default**: `mysql` | Option | Summary | | --- | --- | | mariadb | MariaDB flavored databases. | | mysql | MySQL flavored databases. | ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies the number of tables that will be snapshotted in parallel. **Type**: `int` **Default**: `1` ### [](#max_reconnect_attempts)`max_reconnect_attempts` The maximum number of attempts the MySQL driver will try to re-establish a broken connection before Connect attempts reconnection. A zero or negative number means infinite retry attempts. **Type**: `int` **Default**: `10` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of table rows to fetch in each batch when taking a snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key. **Type**: `bool` ### [](#tables)`tables[]` A list of the database table names to stream changes from. Specify each table name as a separate item. **Type**: `array` ```yaml # Examples: tables: - table1 - table2 ``` ### [](#tls)`tls` Using this field overrides the SSL/TLS settings in the environment and DSN. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 273: nats_jetstream **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/nats_jetstream.md --- # nats_jetstream > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_jetstream latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/nats_jetstream page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/nats_jetstream.adoc description: Reads messages from NATS JetStream subjects. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Reads messages from NATS JetStream subjects. #### Common ```yml inputs: label: "" nats_jetstream: urls: [] # No default (required) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: "" # No default (optional) deliver: all ``` #### Advanced ```yml inputs: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: "" # No default (optional) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: "" # No default (optional) create_stream: false deliver: all ack_wait: 30s max_ack_pending: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consume-mirrored-streams)Consume mirrored streams When a stream being consumed is mirrored in a different JetStream domain, the stream cannot be resolved from the subject name alone. You must specify the stream name as well as the subject (if applicable). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_sequence_stream` - `nats_sequence_consumer` - `nats_num_delivered` - `nats_num_pending` - `nats_domain` - `nats_timestamp_unix_nano` - `nats_consumer` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#ack_wait)`ack_wait` The maximum amount of time NATS server should wait for an ack from consumer. **Type**: `string` **Default**: `30s` ```yaml # Examples: ack_wait: 100ms # --- ack_wait: 5m ``` ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#bind)`bind` Indicates that the subscription should use an existing consumer. **Type**: `bool` ### [](#create_stream)`create_stream` Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set). **Type**: `bool` **Default**: `false` ### [](#deliver)`deliver` Determines which messages to deliver when consuming without a durable subscriber. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Deliver all available messages. | | last | Deliver starting with the last published messages. | | last_per_subject | Deliver starting with the last published message per subject. | | new | Deliver starting from now, not taking into account any previous messages. | ### [](#durable)`durable` Preserve the state of your consumer under a durable name. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_ack_pending)`max_ack_pending` The maximum number of outstanding acks to be allowed before consuming is halted. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#queue)`queue` An optional queue group to consume as. **Type**: `string` ### [](#stream)`stream` A stream to consume from. Either a subject or stream must be specified. **Type**: `string` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. Either a subject or stream must be specified. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 274: nats_kv **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/nats_kv.md --- # nats_kv > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_kv latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/nats_kv page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/nats_kv.adoc description: Watches for updates in a NATS key-value bucket. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Watches for updates in a NATS key-value bucket. #### Common ```yml inputs: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: > auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) key: > auto_replay_nacks: true ignore_deletes: false include_history: false meta_only: false tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_delta` - `nats_kv_operation` - `nats_kv_created` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#ignore_deletes)`ignore_deletes` Do not send delete markers as messages. **Type**: `bool` **Default**: `false` ### [](#include_history)`include_history` Include all the history per key, not just the last one. **Type**: `bool` **Default**: `false` ### [](#key)`key` Key to watch for updates, can include wildcards. **Type**: `string` **Default**: `>` ```yaml # Examples: key: foo.bar.baz # --- key: foo.*.baz # --- key: foo.bar.* # --- key: foo.> ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#meta_only)`meta_only` Retrieve only the metadata of the entry **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 275: nats **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/nats.md --- # nats > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/nats page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/nats.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/nats.adoc description: Subscribe to a NATS subject. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Subscribe to a NATS subject. #### Common ```yml inputs: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true ``` #### Advanced ```yml inputs: label: "" nats: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true nak_delay: "" # No default (optional) prefetch_count: 500000 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_reply_subject` - All message headers (when supported by the connection) You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#nak_delay)`nak_delay` An optional delay duration on redelivering a message when negatively acknowledged. **Type**: `string` ```yaml # Examples: nak_delay: 1m ``` ### [](#prefetch_count)`prefetch_count` The maximum number of messages to pull at a time. **Type**: `int` **Default**: `500000` ### [](#queue)`queue` An optional queue group to consume as. **Type**: `string` ### [](#send_ack)`send_ack` Whether an automatic acknowledgment is sent as a reply to each message. When enabled, these replies are sent only when data has been delivered to all outputs. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. Either a subject or stream must be specified. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 276: oracledb_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/oracledb_cdc.md --- # oracledb_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: oracledb_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/oracledb_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/oracledb_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/oracledb_cdc.adoc description: Enables Change Data Capture by consuming from OracleDB. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-08-11" --- Enables Change Data Capture by consuming from OracleDB. Streams changes from an Oracle database for Change Data Capture (CDC). Use the `snapshot_mode` field to control whether existing data is captured in an initial snapshot before streaming changes. #### Common ```yml inputs: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) snapshot_mode: "" # No default (optional) max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 min_scn_window_size: 1000 max_scn_window_size: 100000 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true transaction_cache: "" # No default (optional) transaction_cache_key: oracledb_cdc snapshot_filters: "" # No default (optional) include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 pdb_name: "" # No default (optional) auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) snapshot_mode: "" # No default (optional) max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 min_scn_window_size: 1000 max_scn_window_size: 100000 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true transaction_cache: "" # No default (optional) transaction_cache_key: oracledb_cdc snapshot_filters: "" # No default (optional) include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 pdb_name: "" # No default (optional) auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `database_schema`: The database schema for the table where the message originates from. - `table_name`: Name of the table that the message originated from. - `operation`: Type of operation that generated the message: "read", "delete", "insert", or "update". "read" is from messages that are read in the initial snapshot phase. - `scn`: The System Change Number in Oracle. Messages published as part of a snapshot will contain Oracle’s current SCN captured at time of snapshot. - `transaction_id`: The Oracle transaction ID in `USN.SLOT.SEQ` format, identifying the transaction that produced the change. Not present on snapshot (`read`) messages. - `source_ts_ms`: The timestamp of when Oracle wrote the change record into the redo log, expressed as milliseconds since the Unix epoch. This reflects the database server’s wall-clock time at the moment the DML executed, not the transaction commit time. - `commit_ts_ms`: The timestamp of the transaction commit, expressed as milliseconds since the Unix epoch. Sourced from `V$LOGMNR_CONTENTS.TIMESTAMP` on the COMMIT redo record — this is Oracle’s wall-clock time when the commit was written to the redo log, not a dedicated commit-timestamp column. For snapshot (`read`) messages, this reflects Oracle’s `SYSTIMESTAMP` at the moment the snapshot SCN was captured, so all snapshot messages share the same value. - `schema`: The table schema, for use with schema-aware downstream processors such as `schema_registry_encode`. When new columns are detected in CDC events, the schema is automatically refreshed from the Oracle catalog. Dropped columns are reflected after a connector restart. > 📝 **NOTE** > > `source_ts_ms` is not present on snapshot (`read`) messages. ## [](#permissions)Permissions When using the default Oracle-based cache, the Connect user requires permission to create tables and stored procedures, and the rpcn schema must already exist. See `checkpoint_cache_table_name` for more information. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to use for storing the current System Change Number (SCN) that has been successfully delivered. This allows Redpanda Connect to continue from that SCN upon restart, rather than consume the entire state of OracleDB redo logs. If not set, the default Oracle-based cache is used. See `checkpoint_cache_table_name` for more information. **Type**: `string` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. **Type**: `string` **Default**: `oracledb_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed System Change Number (SCN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire redo log. When `pdb_name` is set and this field is left at its default value, the table name is automatically derived per PDB (e.g. `RPCN.CDC_CHECKPOINT_MYPDB`) to avoid SCN collisions between pipelines monitoring different PDBs. Set this field explicitly to opt out of that auto-derivation. **Type**: `string` **Default**: `RPCN.CDC_CHECKPOINT_CACHE` ```yaml # Examples: checkpoint_cache_table_name: RPCN.CHECKPOINT_CACHE ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level. Any given System Change Number (SCN) will not be acknowledged unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string of the Oracle database to connect to. You can supply additional connection options as URL query parameters, for example: `oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true`. **Type**: `string` ```yaml # Examples: connection_string: oracle://username:password@host:port/service_name # --- connection_string: oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude. **Type**: `array` ```yaml # Examples: exclude: SCHEMA.PRIVATETABLE ``` ### [](#include)`include[]` Regular expressions for tables to include. **Type**: `array` ```yaml # Examples: include: SCHEMA.PRODUCTS ``` ### [](#logminer)`logminer` LogMiner configuration settings. **Type**: `object` ### [](#logminer-backoff_interval)`logminer.backoff_interval` The interval between attempts to check for new changes once all data is processed. For low traffic tables increasing this value can reduce network traffic to the server. **Type**: `string` **Default**: `5s` ```yaml # Examples: backoff_interval: 5s # --- backoff_interval: 1m ``` ### [](#logminer-lob_enabled)`logminer.lob_enabled` When enabled, large object (CLOB, BLOB) columns are included in both snapshot and streaming change events. When disabled, these columns are still present but contain no values. Enabling this option introduces additional performance overhead and increases memory requirements. **Type**: `bool` **Default**: `true` ### [](#logminer-max_scn_window_size)`logminer.max_scn_window_size` The maximum SCN range that can be mined in a single cycle. The window starts at scn\_window\_size and grows by scn\_window\_size each cycle that ends at the cap (backlog present), up to this limit. It shrinks by the same step each cycle that catches up to the database. This allows the connector to automatically mine larger windows during heavy backlog and smaller windows during steady state. **Type**: `int` **Default**: `100000` ### [](#logminer-max_transaction_events)`logminer.max_transaction_events` The maximum number of events that can be buffered for a single transaction. If a transaction exceeds this limit it is discarded and its events will not be emitted. Set to 0 to disable the limit. **Type**: `int` **Default**: `0` ### [](#logminer-min_scn_window_size)`logminer.min_scn_window_size` The minimum SCN gap required before starting a new LogMiner session. When the gap between the connector’s current position and the database’s current SCN is smaller than this value, the mining cycle is skipped and the connector backs off instead. This prevents excessive LogMiner start/stop cycles on low-traffic databases where Oracle background activity advances the SCN without producing relevant events. Set to 0 to disable. **Type**: `int` **Default**: `1000` ### [](#logminer-mining_interval)`logminer.mining_interval` The interval between mining cycles during normal operation. Controls how frequently LogMiner polls for new changes when not caught up. **Type**: `string` **Default**: `300ms` ```yaml # Examples: mining_interval: 100ms # --- mining_interval: 1s ``` ### [](#logminer-scn_window_size)`logminer.scn_window_size` The SCN range to mine per cycle. Each cycle reads changes between the current SCN and current SCN + scn\_window\_size. Smaller values mean more frequent queries with lower memory usage but higher overhead; larger values reduce query frequency and improve throughput at the cost of higher memory usage per cycle. **Type**: `int` **Default**: `20000` ### [](#logminer-strategy)`logminer.strategy` Controls how LogMiner retrieves data dictionary information. `online_catalog` uses the current data dictionary for best performance but cannot capture DDL changes. Currently, only `online_catalog` is supported. **Type**: `string` **Default**: `online_catalog` ### [](#logminer-transaction_cache)`logminer.transaction_cache` A [cache resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to use for buffering in-flight transactions. When set, DML events are serialized and stored in the named cache rather than held in memory, reducing connector memory usage for workloads with large or long-running transactions. If not set, an in-memory buffer is used. Each in-flight transaction is stored as N+1 cache entries: one metadata key holding the transaction ID, start SCN, and event count; and one event key per DML event. A transaction with 1000 events occupies 1001 cache entries. Each AddEvent call writes exactly two keys regardless of how many events the transaction has already accumulated. This cache is designed for low-latency stores with cheap per-operation cost. Redis and Memcached are the recommended backends. The built-in `memory:{}` cache works but provides no durability across restarts. High-latency or per-request-cost stores such as S3 or DynamoDB are not recommended - a transaction with 1000 events generates approximately 3000 cache operations across its lifetime, and because LogMiner processes events on a single goroutine, per-call latency directly reduces throughput. A backend that causes timeouts or errors will also cause the mining cycle to restart from an earlier checkpoint SCN, which can result in duplicate event delivery. **Type**: `string` ### [](#logminer-transaction_cache_key)`logminer.transaction_cache_key` The key prefix used when storing transactions in `transaction_cache`. An alternative prefix must be set if multiple `oracledb_cdc` inputs share the same cache resource, since Oracle transaction IDs (USN.SLOT.SEQ) are only unique within a single Oracle instance and would otherwise collide. **Type**: `string` **Default**: `oracledb_cdc` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies a number of tables that will be processed in parallel during the snapshot processing stage. **Type**: `int` **Default**: `1` ### [](#pdb_name)`pdb_name` The name of the pluggable database (PDB) to monitor. When connecting to a CDB root, LogMiner output is scoped to this PDB via SRC\_CON\_NAME filtering and catalog queries use ALTER SESSION SET CONTAINER to switch context. Requires GRANT SET CONTAINER TO CONTAINER=ALL. **Type**: `string` ### [](#snapshot_filters)`snapshot_filters` A map of fully-qualified table names (for example `SCHEMA.TABLE`) to SQL `SELECT` queries that override the default snapshot query for each table. Use this to filter or shape the rows captured during the initial snapshot. Each query must project every column of the table’s primary key (all columns of a composite key), even if it otherwise selects only a subset of columns. During a snapshot, Redpanda Connect pages through a table’s rows by filtering and sorting on the full primary key against the query’s own result set. If a primary key column isn’t projected, the snapshot fails part-way through, after the first batch of rows is read. **Type**: `object` ```yaml # Examples: snapshot_filters: TESTDB.PRODUCTS: SELECT * FROM TESTDB.PRODUCTS WHERE ID > 1000 TESTDB.USERS: SELECT * FROM TESTDB.USERS ``` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows to be streamed in a single batch when taking a snapshot. **Type**: `int` **Default**: `1000` ### [](#snapshot_mode)`snapshot_mode` Controls snapshot behavior. `none` (default) skips snapshotting and starts streaming from the current SCN. `snapshot_only` performs a full snapshot, persists the SCN checkpoint, then stops without streaming. `snapshot_and_stream` performs a full snapshot then transitions to streaming. **Type**: `string` **Options**: `none`, `snapshot_only`, `snapshot_and_stream` ### [](#wallet_password)`wallet_password` Password for the `ewallet.p12` PKCS#12 wallet file. Only use this when the wallet directory contains `ewallet.p12` rather than `cwallet.sso`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#wallet_path)`wallet_path` Path to the Oracle Wallet directory. When set, this automatically enables SSL. The directory must contain either `cwallet.sso` (auto-login, does not require a password) or `ewallet.p12` (requires `wallet_password`). **Type**: `string` ```yaml # Examples: wallet_path: /opt/oracle/wallet ``` --- # Page 277: otlp_grpc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/otlp_grpc.md --- # otlp_grpc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: otlp_grpc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/otlp_grpc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/otlp_grpc.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-08-11" --- Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Exposes an OpenTelemetry Collector gRPC receiver that accepts traces, logs, and metrics via gRPC. Telemetry data is received in OTLP protobuf format and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml inputs: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 rate_limit: "" ``` #### Advanced ```yml inputs: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 tls: enabled: false cert_file: "" key_file: "" auth_token: "" max_recv_msg_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#protocols)Protocols This input supports OTLP/gRPC on the default port 4317 using the standard OTLP protobuf format for all signal types (traces, logs, metrics). ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the gRPC metadata. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" ) exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint("localhost:4317"), otlptracegrpc.WithInsecure(), // or WithTLSCredentials() for TLS otlptracegrpc.WithHeaders(map[string]string{ "authorization": "Bearer your-token-here", }), ) ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a ResourceExhausted gRPC status code. ## [](#fields)Fields ### [](#address)`address` The address to listen on for gRPC connections. **Type**: `string` **Default**: `0.0.0.0:4317` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'authorization: Bearer ' metadata. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` Encoding format for messages in the batch. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_recv_msg_size)`max_recv_msg_size` Maximum size of gRPC messages to receive in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` An optional rate limit resource to throttle requests. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for gRPC. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` --- # Page 278: otlp_http **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/otlp_http.md --- # otlp_http > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: otlp_http latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/otlp_http page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/otlp_http.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-08-11" --- Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Exposes an OpenTelemetry Collector HTTP receiver that accepts traces, logs, and metrics via HTTP. Telemetry data is received in OTLP format (both protobuf and JSON) at standard OTLP endpoints and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml inputs: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 rate_limit: "" ``` #### Advanced ```yml inputs: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 tls: enabled: false cert_file: "" key_file: "" auth_token: "" read_timeout: 10s write_timeout: 10s max_body_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#endpoints)Endpoints This input exposes the following standard OTLP HTTP endpoints: - `/v1/traces` - OpenTelemetry traces - `/v1/logs` - OpenTelemetry logs - `/v1/metrics` - OpenTelemetry metrics ## [](#protocols)Protocols This input supports OTLP/HTTP on the default port 4318. It accepts both: - `application/x-protobuf` - OTLP protobuf format - `application/json` - OTLP JSON format ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the HTTP Authorization header. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" ) exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint("localhost:4318"), otlptracehttp.WithInsecure(), // or WithTLSClientConfig() for TLS otlptracehttp.WithHeaders(map[string]string{ "Authorization": "Bearer your-token-here", }), ) ``` ### [](#curl-example)cURL example ```bash curl -X POST http://localhost:4318/v1/traces \ -H "Content-Type: application/x-protobuf" \ -H "Authorization: Bearer your-token-here" \ --data-binary @traces.pb ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a 429 (Too Many Requests) response. ## [](#fields)Fields ### [](#address)`address` The address to listen on for HTTP connections. **Type**: `string` **Default**: `0.0.0.0:4318` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'Authorization: Bearer ' header. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` Encoding format for messages in the batch. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_body_size)`max_body_size` Maximum size of HTTP request body in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` An optional rate limit resource to throttle requests. **Type**: `string` **Default**: `""` ### [](#read_timeout)`read_timeout` Maximum duration for reading the entire request. **Type**: `string` **Default**: `10s` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for HTTP. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` Maximum duration for writing the response. **Type**: `string` **Default**: `10s` --- # Page 279: postgres_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/postgres_cdc.md --- # postgres_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: postgres_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/postgres_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/postgres_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/postgres_cdc.adoc page-git-created-date: "2024-12-05" page-git-modified-date: "2026-05-26" --- Streams data changes from a PostgreSQL database using logical replication. There is also a configuration option to [stream all existing data](#stream_snapshot) from the database. ```yml inputs: label: "" postgres_cdc: dsn: "" # No default (required) include_transaction_markers: false stream_snapshot: false snapshot_batch_size: 1000 schema: "" # No default (required) tables: [] # No default (required) checkpoint_limit: 1024 temporary_slot: false slot_name: "" # No default (required) pg_standby_timeout: 10s pg_wal_monitor_interval: 3s max_parallel_snapshot_tables: 1 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `postgres_cdc` input uses logical replication to capture changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. Redpanda Connect uses this replication method to allow you to choose which database tables in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from, and an [option to receive TOAST and deleted values](#receive-toast-and-deleted-values) in your data updates. ## [](#prerequisites)Prerequisites - PostgreSQL version 14 or later - Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see [Redpanda Cloud Networking](https://docs.redpanda.com/cloud-data-platform/networking/). - Logical replication enabled on your PostgreSQL cluster To check whether logical replication is already enabled, run the following query: ```SQL SHOW wal_level; ``` If the `wal_level` value is `logical`, you can start to use this connector. Otherwise, choose from the following sets of instructions to update your replication settings. ### Cloud platforms - [Amazon RDS for PostgreSQL DB](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication.html) - [Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical#prerequisites-for-logical-replication-and-logical-decoding) - [Google Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication), including creating a user with replication privileges - [Neon](https://neon.tech/docs/guides/logical-replication-guide) ### Self-Hosted PostgreSQL Use an account with sufficient permissions (superuser) to update your replication settings. 1. Open the `postgresql.conf` file. 2. Find the `wal_level` parameter. 3. Update the parameter value to `wal_level = logical`. If you already use replication slots, you may need to increase the limit on replication slots (`max_replication_slots`). The `max_wal_senders` parameter value must also be greater than or equal to `max_replication_slots`. 4. Restart the PostgreSQL server. For this input to make a successful connection to your database, also make sure that it allows replication connections. 1. Open the `pg_hba.conf` file. 2. Update this line. ```yaml host replication /32 md5 ``` Replace the following placeholders with your own values: - ``: The username from an account with superuser privileges. - ``: The IP address of the server where you are running Redpanda Connect. 3. Restart the PostgreSQL server. ## [](#choose-a-replication-mode)Choose a replication mode When you run a pipeline that uses the `postgres_cdc` input, Redpanda Connect connects to your PostgreSQL database and creates a replication slot. The replication slot uses a copy of the Write-Ahead Log (WAL) file to subscribe to changes in your database records as they are applied to the database. There are two replication modes you can choose from: snapshot mode and streaming mode. In snapshot mode, Redpanda Connect first takes a snapshot of the database and streams the contents before processing changes from the WAL. In streaming mode, Redpanda Connect directly processes changes from the WAL starting from the most recent changes without taking a snapshot first. For local testing, you can use the [example pipeline on this page](#example-pipeline), which runs in snapshot mode. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect: 1. Creates a snapshot of your database. 2. Streams the contents of the tables specified in the `postgres_cdc` input. 3. Starts processing changes in the WAL that occurred since the snapshot was taken, and streams them to Redpanda Connect. Once the initial replication process is complete, the snapshot is removed and the input keeps a connection open to the database so that it can receive data updates. If the pipeline restarts during the replication process, Redpanda Connect resumes processing data changes from where it left off. If there are other interruptions while the snapshot is taken, you may need to restart the snapshot process. For more information, see [Troubleshoot replication failures](#troubleshoot_replication_failures). ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect starts processing data changes from the end of the WAL. If the pipeline restarts, Redpanda Connect resumes processing data changes from the last acknowledged position in the WAL. ## [](#monitor-the-replication-process)Monitor the replication process You can monitor the initial replication of data using the following metrics: | Metric name | Description | | --- | --- | | replication_lag_bytes | Indicates how far the connector is lagging behind the source database when processing the transaction log. | | postgres_snapshot_progress | Shows the progress of snapshot processing for each table. | ## [](#troubleshoot-replication-failures)Troubleshoot replication failures If the database snapshot fails, the replication slot has only an incomplete record of the existing data in your database. To maintain data integrity, you must drop the replication slot manually in your source database and run the Redpanda Connect pipeline again. ```SQL SELECT pg_drop_replication_slot(SLOT_NAME); ``` ## [](#receive-toast-and-deleted-values)Receive TOAST and deleted values For full visibility of all data updates, you can also choose to stream [TOAST](https://www.postgresql.org/docs/current/storage-toast.html) and deleted values. To enable this option, run the following query on your source database: ```SQL ALTER TABLE large_data REPLICA IDENTITY FULL; ``` ## [](#data-mappings)Data mappings The following table shows how selected PostgreSQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | PostgreSQL data type | Bloblang value | | --- | --- | | TEXT, TIMESTAMP, UUID, VARCHAR | JSON strings, for example: this data | | BOOL | Boolean JSON fields, for example: true or false | | Numeric types (INT4) | JSON number types, for example: 1. | | JSONB | JSON objects, for example: { "message": "message text" } | | INTEGER[] | An array of integer values, for example: [1,2,3] | | TEXT[] | An array of string values, for example: ["value1", "value2", "value3"] | | INET | A string that contains an IP address, for example: "192.168.1.1" | | POINT | A string that represents a point in a two-dimensional plane, for example: (x, y) | | TSRANGE | A string that includes range bounds, for example: [2010-01-01 14:30, 2010-01-01 15:30) | | TSVECTOR | A string that includes vector data, for example: "'the':2 'question':3 'is':4" | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `table`: The name of the database table from which the message originated. - `operation`: The type of database operation that generated the message, such as `read`, `insert`, `update`, `delete`, `begin` and `commit`. A `read` operation occurs when a snapshot of the database is processed. The `begin` and `commit` operations are only included if the `include_transaction_markers` field is set to `true`. - `lsn`: The [Log Sequence Number](https://www.postgresql.org/docs/current/datatype-pg-lsn.html) of each data update from the source PostgreSQL database. The `lsn` values are strings that can be sorted to determine the order in which data updates were written to the WAL. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. This is useful for connecting to Amazon RDS or Aurora PostgreSQL instances with IAM database authentication enabled. The generated tokens are valid for 15 minutes and are automatically refreshed. For more information about AWS credentials configuration, see the [credentials for AWS](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/) guide. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for PostgreSQL. When enabled, an IAM authentication token is generated and used as the password. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The PostgreSQL endpoint hostname (e.g., mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The ID of credentials to use. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the PostgreSQL instance is located. If no region is specified then the environment default will be used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID for the role assumption. Only used with the `role` field. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles can be assumed in sequence, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. **Type**: `array` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#aws-token)`aws.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given log sequence number (LSN) is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the PostgreSQL database from which you want to stream updates. Use the format `postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​]`. For example, if you wanted to disable SSL in a secure environment, you would add `sslmode=disable` to the connection string. **Type**: `string` ```yaml # Examples: dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval between heartbeat messages, which Redpanda Connect writes to the WAL using the `pg_logical_emit_message` function. Heartbeat messages are useful when you subscribe to data changes from tables with low activity, while other tables in the database have higher-frequency updates. Heartbeat messages allow Redpanda Connect to periodically acknowledge new messages even when no data updates occur. Each acknowledgement advances the committed point in the WAL, which ensures that PostgreSQL can safely reclaim older log segments, preventing excessive disk space usage. Set `heartbeat_interval` to `0s` to disable heartbeats. **Type**: `string` **Default**: `1h` ```yaml # Examples: heartbeat_interval: 0s # --- heartbeat_interval: 24h ``` ### [](#include_transaction_markers)`include_transaction_markers` When set to `true`, creates empty messages for `BEGIN` and `COMMIT` operations which start and complete each transaction. Messages with the `operation` metadata field set to `BEGIN` or `COMMIT` have null message payloads. **Type**: `bool` **Default**: `false` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specify the maximum number of tables that are processed in parallel when the initial snapshot of the source database is taken. **Type**: `int` **Default**: `1` ### [](#pg_standby_timeout)`pg_standby_timeout` Specify the standby timeout after which an idle connection is refreshed to keep the connection alive. **Type**: `string` **Default**: `10s` ```yaml # Examples: pg_standby_timeout: 30s ``` ### [](#pg_wal_monitor_interval)`pg_wal_monitor_interval` How often to report changes to the replication lag and write them to Redpanda Connect metrics. **Type**: `string` **Default**: `3s` ```yaml # Examples: pg_wal_monitor_interval: 6s ``` ### [](#schema)`schema` The PostgreSQL schema from which to replicate data. **Type**: `string` ```yaml # Examples: schema: public # --- schema: "MyCaseSensitiveSchemaNeedingQuotes" ``` ### [](#slot_name)`slot_name` The name of the PostgreSQL logical replication slot to use. If not provided, a random name is generated unless you create a replication slot manually before starting replication. **Type**: `string` ```yaml # Examples: slot_name: my_test_slot ``` ### [](#snapshot_batch_size)`snapshot_batch_size` The number of table rows to fetch in each batch when querying the snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ```yaml # Examples: snapshot_batch_size: 10000 ``` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` ### [](#tables)`tables[]` A list of database table names to include in the snapshot and logical replication. Specify each table name as a separate item. **Type**: `array` ```yaml # Examples: tables: - my_table_1 - "MyCaseSensitiveTableNeedingQuotes" ``` ### [](#temporary_slot)`temporary_slot` If set to `true`, the input creates a temporary replication slot that is automatically dropped when the connection to your source database is closed. You might use this option to: - Avoid data accumulating in the replication slot when a pipeline is paused or stopped - Test the connector If the pipeline is restarted, another data snapshot is taken before data updates are streamed. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#unchanged_toast_value)`unchanged_toast_value` Specify the value to emit when unchanged [TOAST values](#receive-toast-and-deleted-values) appear in the message stream. Unchanged values occur for data updates and deletes when `REPLICA IDENTITY` is not set to `FULL`. **Type**: `unknown` **Default**: ```yaml null ``` ```yaml # Examples: unchanged_toast_value: __redpanda_connect_unchanged_toast_value__ ``` ## [](#example-pipeline)Example pipeline You can run the following pipeline locally to check that data updates are streamed from your source database to Redpanda Connect. All transactions are written to stdout. ```yml input: label: "postgres_cdc" postgres_cdc: dsn: postgres://user:password@host:port/dbname include_transaction_markers: false slot_name: test_slot_native_decoder snapshot_batch_size: 100000 stream_snapshot: true temporary_slot: true schema: schema_name tables: - table_name cache_resources: - label: data_caching file: directory: /tmp/cache output: label: main stdout: {} ``` --- # Page 280: read_until **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/read_until.md --- # read_until > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: read_until latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/read_until page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/read_until.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/read_until.adoc description: Reads messages from a child input until a consumed message passes a Bloblang query, at which point the input closes. It is also possible to configure a timeout after which the input is closed if no new messages arrive in that period. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Reads messages from a child input until a consumed message passes a [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/), at which point the input closes. It is also possible to configure a timeout after which the input is closed if no new messages arrive in that period. ```yml inputs: label: "" read_until: input: "" # No default (required) check: "" # No default (optional) idle_timeout: "" # No default (optional) restart_input: false ``` Messages are read continuously while the query check returns false, when the query returns true the message that triggered the check is sent out and the input is closed. Use this to define inputs where the stream should end once a certain message appears. If the idle timeout is configured, the input will be closed if no new messages arrive after that period of time. Use this field if you want to empty out and close an input that doesn’t have a logical end. Sometimes inputs close themselves. For example, when the `file` input type reaches the end of a file it will shut down. By default this type will also shut down. If you wish for the input type to be restarted every time it shuts down until the query check is met then set `restart_input` to `true`. ## [](#metadata)Metadata A metadata key `benthos_read_until` containing the value `final` is added to the first part of the message that triggers the input to stop. ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether the input should now be closed. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: count("messages") >= 100 ``` ### [](#idle_timeout)`idle_timeout` The maximum amount of time without receiving new messages after which the input is closed. **Type**: `string` ```yaml # Examples: idle_timeout: 5s ``` ### [](#input)`input` The child input to consume from. **Type**: `input` ### [](#restart_input)`restart_input` Whether the input should be reopened if it closes itself before the condition has resolved to true. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#consume-n-messages)Consume N Messages A common reason to use this input is to consume only N messages from an input and then stop. This can easily be done with the [`count` function](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#count): ```yaml # Only read 100 messages, and then exit. input: read_until: check: count("messages") >= 100 input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` ### [](#read-from-a-kafka-and-close-when-empty)Read from a kafka and close when empty A common reason to use this input is a job that consumes all messages and exits once its empty: ```yaml # Consumes all messages and exit when the last message was consumed 5s ago. input: read_until: idle_timeout: 5s input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` --- # Page 281: redis_list **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redis_list.md --- # redis_list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redis_list page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redis_list.adoc description: Pops messages from the beginning of a Redis list using the BLPop command. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Pops messages from the beginning of a Redis list using the BLPop command. #### Common ```yml inputs: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) auto_replay_nacks: true max_in_flight: 0 timeout: 5s command: blpop ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to pop elements from the Redis list **Type**: `string` **Default**: `blpop` **Options**: `blpop`, `brpop` ### [](#key)`key` The key of a list to read from. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` Optionally sets a limit on the number of messages that can be flowing through a Redpanda Connect stream pending acknowledgment from the input at any given time. Once a message has been either acknowledged or rejected (nacked) it is no longer considered pending. If the input produces logical batches then each batch is considered a single count against the maximum. **WARNING**: Batching policies at the output level will stall if this field limits the number of messages below the batching threshold. Zero (default) or lower implies no limit. **Type**: `int` **Default**: `0` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 282: redis_pubsub **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redis_pubsub.md --- # redis_pubsub > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_pubsub latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redis_pubsub page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redis_pubsub.adoc description: Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands. #### Common ```yml inputs: label: "" redis_pubsub: url: "" # No default (required) channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` In order to subscribe to channels using the `PSUBSCRIBE` command set the field `use_patterns` to `true`, then you can include glob-style patterns in your channel names. For example: - `h?llo` subscribes to hello, hallo and hxllo - `h*llo` subscribes to hllo and heeeello - `h[ae]llo` subscribes to hello and hallo, but not hillo Use `\` to escape special characters if you want to match them verbatim. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `redis_pubsub_channel` - `redis_pubsub_pattern` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#channels)`channels[]` A list of channels to consume from. **Type**: `array` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#use_patterns)`use_patterns` Whether to use the PSUBSCRIBE command, allowing for glob-style patterns within target channel names. **Type**: `bool` **Default**: `false` --- # Page 283: redis_scan **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redis_scan.md --- # redis_scan > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_scan latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redis_scan page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redis_scan.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redis_scan.adoc description: Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands. #### Common ```yml inputs: label: "" redis_scan: url: "" # No default (required) auto_replay_nacks: true match: "" ``` #### Advanced ```yml inputs: label: "" redis_scan: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true match: "" ``` Optionally, iterates only elements matching a blob-style pattern. For example: - `**foo**` iterates only keys which contain `foo` in it. - `foo*` iterates only keys starting with `foo`. This input generates a message for each key value pair in the following format: ```json {"key":"foo","value":"bar"} ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#match)`match` Iterates only elements matching the optional glob-style pattern. By default, it matches all elements. **Type**: `string` **Default**: `""` ```yaml # Examples: match: * # --- match: 1* # --- match: foo* # --- match: foo # --- match: *4* ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 284: redis_streams **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redis_streams.md --- # redis_streams > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_streams latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redis_streams page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redis_streams.adoc description: Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The client_id should be unique for each consumer of a group. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The `client_id` should be unique for each consumer of a group. #### Common ```yml inputs: label: "" redis_streams: url: "" # No default (required) body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" ``` #### Advanced ```yml inputs: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" create_streams: true start_from_oldest: true commit_period: 1s timeout: 1s ``` Redis stream entries are key/value pairs, as such it is necessary to specify the key that contains the body of the message. All other keys/value pairs are saved as metadata fields. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#body_key)`body_key` The field key to extract the raw message from. All other keys will be stored in the message as metadata. **Type**: `string` **Default**: `body` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current offset. Offsets are always committed during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the stream. **Type**: `string` **Default**: `""` ### [](#create_streams)`create_streams` Create subscribed streams if they do not exist (MKSTREAM option). **Type**: `bool` **Default**: `true` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#limit)`limit` The maximum number of messages to consume from a single request. **Type**: `int` **Default**: `10` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#start_from_oldest)`start_from_oldest` If an offset is not found for a stream, determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. **Type**: `bool` **Default**: `true` ### [](#streams)`streams[]` A list of streams to consume from. **Type**: `array` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `1s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 285: redpanda_common **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_common.md --- # redpanda_common > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda_common latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redpanda_common page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redpanda_common.adoc description: Consumes data from a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. Consumes data from a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) instead. #### Common ```yml inputs: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ### [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect uses at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda_common: topic: foo - retry: output: redpanda_common: topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#consumer_group)`consumer_group` An optional consumer group. When this value is specified: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 286: redpanda_migrator **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator.md --- # redpanda_migrator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda_migrator page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redpanda_migrator page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redpanda_migrator.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-10-02" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Unified Kafka consumer for migrating data between Kafka/Redpanda clusters. Use this input with the [`redpanda_migrator` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) to safely transfer topic data, ACLs, schemas, and consumer group offsets between clusters. This component is designed for migration scenarios. ### Common ```yml inputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} auto_replay_nacks: true ``` ### Advanced ```yml inputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} auto_replay_nacks: true ``` The `redpanda_migrator` input: - Reads a batch of messages from a broker. - Waits for the `redpanda_migrator` output to acknowledge the writes before updating the Kafka consumer group offset. - Provides the same delivery guarantees and ordering semantics as the [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/). Specify a consumer group to make this input consume one or more topics and automatically balance the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input requires a corresponding `redpanda_migrator` output in the same pipeline. Each pipeline must have both input and output components configured. For capabilities, guarantees, scheduling, and examples, see the output documentation. ## [](#requirements)Requirements - Must be paired with a `redpanda_migrator` output in the same pipeline. - Requires access to a source Kafka or Redpanda cluster. - Consumer group configuration is recommended for partition balancing. - When the source cluster enforces ACLs, a consumer ACL alone is not enough for the source principal: it needs at minimum topic `READ` **and** `DESCRIBE_CONFIGS`, plus consumer group and cluster permissions. A `READ` ACL grants `DESCRIBE` but not `DESCRIBE_CONFIGS`, so the migrator consumes messages but fails to create topics with `TOPIC_AUTHORIZATION_FAILED`. See [Required permissions](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator/#required-permissions). ## [](#multiple-migrator-pairs)Multiple migrator pairs When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair. ## [](#performance-tuning-for-high-throughput)Performance tuning for high throughput For workloads with high message rates or large messages, adjust the following settings to optimize throughput: On this input component: - `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size - `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded On the paired `redpanda_migrator` output component: - `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster) > 📝 **NOTE** > > Setting `max_yield_batch_bytes` over 1MB is counter-productive unless you change the broker settings to allow bigger messages or batches. The `partition_buffer_bytes` setting allows for partition readahead. ## [](#metrics)Metrics This input emits an `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic. This metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. Monitor this metric to track migration progress and detect bottlenecks. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - kafka\_key - kafka\_topic - kafka\_partition - kafka\_offset - kafka\_lag - kafka\_timestamp\_ms - kafka\_timestamp\_unix - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When specified, the partitions of specified topics are automatically distributed across consumers sharing a consumer group, and partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when explicit partitions are specified to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods of SASL authentication, which are tried in order. If the broker supports the first mechanism, all connections use that mechanism. If the first mechanism fails, the client picks the first supported mechanism. Connections fail if the broker does not support any client mechanisms. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ## [](#troubleshooting)Troubleshooting - Ensure the input and output `label` fields match exactly. - Both input and output must be present in the pipeline. - Verify consumer group configuration for partition balancing. - Monitor the lag metric for stalled migration. ## [](#suggested-reading)Suggested reading - [`redpanda_migrator` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) - [Migrating from legacy components](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/migrate-unified-redpanda-migrator/) --- # Page 287: redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda.md --- # redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/redpanda page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/redpanda.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2026-05-26" --- Consumes topic data from one or more Kafka brokers. #### Common ```yml inputs: label: "" redpanda: seed_brokers: [] # No default (optional) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB unordered_processing: enabled: false checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ## [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect leans towards at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda_common: topic: foo - retry: output: redpanda_common: topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When this value is specified: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. Optional when `seed_brokers` is configured in a top-level `redpanda` block. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ### [](#unordered_processing)`unordered_processing` Allows consumers to process messages of any given partition in parallel, which may result in unordered processing. This option enables asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the `checkpoint_limit` field. **Type**: `object` ### [](#unordered_processing-batching)`unordered_processing.batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/) that applies to individual topic partitions in order to batch messages together before flushing them for processing. Batching can be beneficial for performance and useful for windowed processing, and doing so preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#unordered_processing-batching-byte_size)`unordered_processing.batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-check)`unordered_processing.batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#unordered_processing-batching-count)`unordered_processing.batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-period)`unordered_processing.batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#unordered_processing-batching-processors)`unordered_processing.batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#unordered_processing-checkpoint_limit)`unordered_processing.checkpoint_limit` Determines how many messages of the same partition can be processed in parallel before applying back pressure. When a message of a given offset is delivered to the output the offset is only allowed to be committed when all messages of prior offsets have also been delivered, this ensures at-least-once delivery guarantees. However, this mechanism also increases the likelihood of duplicates in the event of crashes or server faults, reducing the checkpoint limit will mitigate this. **Type**: `int` **Default**: `1024` ### [](#unordered_processing-enabled)`unordered_processing.enabled` Whether to enable the unordered processing of messages from a given partition. **Type**: `bool` **Default**: `false` --- # Page 288: resource **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/resource.md --- # resource > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: resource latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/resource page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/resource.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/resource.adoc description: Resource is an input type that channels messages from a resource input, identified by its name. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Resource is an input type that channels messages from a resource input, identified by its name. ```yml inputs: label: "" resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - gcp_pubsub: project: bar subscription: baz ``` Could also be expressed as: ```yaml input: broker: inputs: - resource: foo - resource: bar input_resources: - label: foo kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - label: bar gcp_pubsub: project: bar subscription: baz ``` Resources also allow you to reference a single input in multiple places, such as multiple streams mode configs, or multiple entries in a broker input. However, when a resource is referenced more than once the messages it produces are distributed across those references, so each message will only be directed to a single reference, not all of them. --- # Page 289: salesforce_cdc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc.md --- # salesforce_cdc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: salesforce_cdc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/salesforce_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/salesforce_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/salesforce_cdc.adoc description: Subscribes to one or more Salesforce Pub/Sub topics in parallel and emits a message per event. page-git-created-date: "2026-05-26" page-git-modified-date: "2026-08-11" --- Subscribes to one or more Salesforce Pub/Sub topics in parallel and emits a message per event. Topics may be: - `/data/ChangeEvent` — per-sObject CDC channel. - `/data/ChangeEvents` — CDC firehose (every CDC-enabled sObject). - `/event/__e` — custom Platform Event. - `/event/` — standard Platform Event (e.g. `LoginEventStream`). - A bare sObject name (e.g. `Account`) is shorthand for `/data/AccountChangeEvent`. Optionally runs a REST snapshot for the CDC sObjects before opening the streaming subscriptions, so the pipeline sees the current state plus continuous changes. Per-topic replay state persists in a cache resource so each subscription resumes across restarts independently. ## [](#when-to-use-this-input)When to use this input Use `salesforce_cdc` for: - Continuous ingestion with both historical state (snapshot) and live changes (CDC). - Real-time custom or standard Platform Events. - Mixed CDC + Platform Event pipelines under a single component. Use a different Salesforce input instead if: - You only need a one-off extract or periodic SOQL query — use [`salesforce`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce/). - You need a GraphQL query (cross-object in one request) — use [`salesforce_graphql`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql/). ### Common ```yml inputs: label: "" salesforce_cdc: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 topics: [] # No default (required) stream_snapshot: true replay_preset: latest snapshot_max_batch_size: 2000 stream_batch_size: 100 max_parallel_snapshot_objects: 1 checkpoint_cache: "" # No default (required) checkpoint_cache_key: salesforce_cdc checkpoint_limit: 1024 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ### Advanced ```yml inputs: label: "" salesforce_cdc: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 topics: [] # No default (required) stream_snapshot: true replay_preset: latest snapshot_max_batch_size: 2000 stream_batch_size: 100 max_parallel_snapshot_objects: 1 checkpoint_cache: "" # No default (required) checkpoint_cache_key: salesforce_cdc checkpoint_limit: 1024 auto_replay_nacks: true grpc: reconnect_base_delay: 500ms reconnect_max_delay: 30s reconnect_max_attempts: 0 shutdown_timeout: 10s buffer_size: 1000 http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#metadata)Metadata Every emitted message has: - `topic`: The full Pub/Sub topic path (e.g. "/event/Order\_\_e"). - `replay_id`: The Pub/Sub replay ID in hex (streaming events only). CDC events also carry: - `operation`: "read" for snapshot rows; "create", "update", "delete", or "undelete" for CDC events. - `sobject`: The sObject API name (e.g. "Account"). - `record_ids`: Comma-separated record IDs affected by the event (when present). Platform Events also carry: - `event_uuid`: The Salesforce `EventUuid` extracted from the payload (the canonical dedup key), when present. ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org — check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` Name of the cache resource used to persist snapshot cursor and per-topic replay IDs across restarts. The cache must be declared under the top-level `cache_resources` block. Choose a durable cache (Redis, Postgres, DynamoDB) for production; in-memory caches lose checkpoints on restart. **Type**: `string` ```yaml # Examples: checkpoint_cache: persistent_cache ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` Key inside the checkpoint cache where this input’s state is stored. Change when running multiple `salesforce_cdc` inputs against the same cache resource to avoid collisions. **Type**: `string` **Default**: `salesforce_cdc` ### [](#checkpoint_limit)`checkpoint_limit` Maximum number of unacknowledged batches in flight (per topic) before that topic pauses reading. Prevents unbounded memory growth when downstream components stall. Higher values increase throughput in steady state; lower values bound memory under backpressure. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive — prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#grpc)`grpc` gRPC transport tuning for the Pub/Sub API connection. **Type**: `object` ### [](#grpc-buffer_size)`grpc.buffer_size` Size of the internal gRPC event receive buffer. **Type**: `int` **Default**: `1000` ### [](#grpc-reconnect_base_delay)`grpc.reconnect_base_delay` Base delay for gRPC reconnection backoff. **Type**: `string` **Default**: `500ms` ### [](#grpc-reconnect_max_attempts)`grpc.reconnect_max_attempts` Maximum number of gRPC reconnection attempts. 0 means unlimited. **Type**: `int` **Default**: `0` ### [](#grpc-reconnect_max_delay)`grpc.reconnect_max_delay` Maximum delay for gRPC reconnection backoff. **Type**: `string` **Default**: `30s` ### [](#grpc-shutdown_timeout)`grpc.shutdown_timeout` Timeout for graceful gRPC client shutdown. **Type**: `string` **Default**: `10s` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` TCP socket configuration. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#max_parallel_snapshot_objects)`max_parallel_snapshot_objects` Number of sObjects snapshotted concurrently during the REST snapshot phase. Each in-flight snapshot consumes one HTTP connection and Salesforce API call quota. Default 1 serializes the work — raise when snapshotting many sObjects and your API limits permit. **Type**: `int` **Default**: `1` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#replay_preset)`replay_preset` Initial replay position used per topic only on first run (when no checkpoint exists in the cache); ignored once a topic’s replay ID has been written. - `latest`: Start from new events only; any changes between prior run and Connect are skipped. - `earliest`: Replay from the retention start (24h standard, 72h with enhanced retention). Use to recover missed events after outages. **Type**: `string` **Default**: `latest` **Options**: `latest`, `earliest` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` Page size for the REST snapshot query — records per `/query` response. Must be between 200 and 2000 per Salesforce REST API limits. Larger pages reduce HTTP round trips; smaller pages reduce peak memory per fetch. **Type**: `int` **Default**: `2000` ```yaml # Examples: snapshot_max_batch_size: 2000 # --- snapshot_max_batch_size: 500 ``` ### [](#stream_batch_size)`stream_batch_size` Number of events requested per gRPC `Fetch` call, per topic. Higher values improve throughput at the cost of peak batch memory; lower values give steadier latency under load. **Type**: `int` **Default**: `100` ```yaml # Examples: stream_batch_size: 100 # --- stream_batch_size: 500 ``` ### [](#stream_snapshot)`stream_snapshot` When true (default), paginate a full REST snapshot of every CDC sObject in `topics` before opening any streaming subscription. When false, skip the snapshot and start streaming immediately. Platform Event topics (`/event/…​`) are always skipped — they have no REST equivalent. **Type**: `bool` **Default**: `true` ### [](#topics)`topics[]` Pub/Sub topics to subscribe to. Each entry is one of: a bare sObject name (`Account` → `/data/AccountChangeEvent`), an explicit CDC channel (`/data/AccountChangeEvent`), the CDC firehose (`/data/ChangeEvents`), or a Platform Event topic (`/event/Order__e`, `/event/LoginEventStream`). Each topic gets its own gRPC subscription with an independent replay cursor. **Type**: `array` ```yaml # Examples: topics: - Account - Contact # --- topics: - /data/ChangeEvents # --- topics: - Account - /event/Order__e # --- topics: - Opportunity - MyCustom__c - /event/Sync_Requested__e ``` --- # Page 290: salesforce_graphql **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql.md --- # salesforce_graphql > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: salesforce_graphql latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/salesforce_graphql page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/salesforce_graphql.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/salesforce_graphql.adoc description: Executes a GraphQL query against the Salesforce UIAPI and emits one message per record. page-git-created-date: "2026-05-26" page-git-modified-date: "2026-08-11" --- Executes a GraphQL query against the Salesforce UIAPI (`POST /services/data/{api_version}/graphql`), walks the response tree, and emits one message per record. When records are exhausted the input shuts down, letting the pipeline terminate gracefully (or the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to take over). ## [](#when-to-use-this-input)When to use this input Use `salesforce_graphql` for: - Cross-object queries in a single request (parent + children + grandchildren). - Response shapes that already match your downstream schema. - Queries benefiting from GraphQL’s field-level selection and nesting. - Adopting Salesforce’s UIAPI / future-forward query surface. Use a different Salesforce input instead if: - You only need single-object SELECTs — use [`salesforce`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce/) (simpler, no GraphQL schema knowledge needed). - You need continuous change events — use [`salesforce_cdc`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc/). When the query selects an `edges`/`pageInfo` connection, the input transparently paginates by injecting `after: ""` into the query string between requests. The query must include `pageInfo { hasNextPage endCursor }` for pagination to terminate cleanly. Responses without an `edges` array are emitted as a single message and the input completes. ### Common ```yml inputs: label: "" salesforce_graphql: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 query: "" # No default (required) variables: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml inputs: label: "" salesforce_graphql: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 query: "" # No default (required) variables: "" # No default (optional) auto_replay_nacks: true http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#metadata)Metadata This input adds no Salesforce-specific metadata. GraphQL response shapes vary by query, so record identity and context travel in the message body. ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org — check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive — prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` TCP socket configuration. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#query)`query` The GraphQL query document as a single string. Must target the Salesforce UIAPI schema (`uiapi.query.` **or `uiapi.mutation.`**). Typically follows the UIAPI convention of nested `edges { node { field { value } } }`. Variables are referenced with `$name` and supplied via the `variables` field. For automatic pagination, include `pageInfo { hasNextPage endCursor }` in the relevant connection. **Type**: `string` ```yaml # Examples: query: query Accounts { uiapi { query { Account { edges { node { Id { value } Name { value } } } pageInfo { hasNextPage endCursor } } } } } # --- query: query Accounts($first: Int) { uiapi { query { Account(first: $first) { edges { node { Id { value } Name { value } } } pageInfo { hasNextPage endCursor } } } } } ``` ### [](#variables)`variables` Optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) whose result must be an object whose keys are GraphQL variable names referenced by `query`. Values pass through as JSON in the request body’s `variables` field. The mapping is evaluated once at startup with no message context — use `env()`, `now()`, `cache()`, or static literals. **Type**: `string` ```yaml # Examples: variables: root = {"first": 100} # --- variables: root = {"since": now().ts_format("2006-01-02T15:04:05Z"), "limit": 500} ``` --- # Page 291: salesforce **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce.md --- # salesforce > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: salesforce latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/salesforce page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/salesforce.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/salesforce.adoc description: Runs a SOQL query against the Salesforce REST API, paginates through all result pages, and emits one message per record. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-11" --- Runs a SOQL query against the Salesforce REST API, paginates through all result pages, and emits one message per record. When results are exhausted the input shuts down, letting the pipeline terminate gracefully (or the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to take over). ## [](#when-to-use-this-input)When to use this input Use `salesforce` for: - One-shot extracts (e.g. dump all Accounts into a warehouse). - Periodic full-table refreshes via a scheduled pipeline or [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/). - Backfills and ad-hoc queries. - Warming up a downstream pipeline before switching to CDC. Use a different Salesforce input instead if: - You need continuous change events — use [`salesforce_cdc`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc/). - You need a GraphQL query (cross-object in one request) — use [`salesforce_graphql`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql/). ### Common ```yml inputs: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 object: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` ### Advanced ```yml inputs: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 object: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) auto_replay_nacks: true http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sobject`: The sObject API name (e.g. "Account"). ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org — check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#args_mapping)`args_mapping` Optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) whose result must be an array of values matching the count of `?` placeholders in `where`. Values are SOQL-escaped: strings become quoted literals, timestamps become ISO-8601, booleans and numbers pass through. The mapping is evaluated once at startup with no message context — use `now()`, `env()`, or `cache()`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ (now() - "1h").ts_format("2006-01-02T15:04:05Z") ] # --- args_mapping: root = [ "Active", (now() - "24h").ts_format("2006-01-02T15:04:05Z") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` Consumer Key of the Salesforce Connected App authorized for the OAuth Client Credentials flow. Create the Connected App under Setup → App Manager → New Connected App, enable OAuth settings, enable the Client Credentials Flow under `Flow Enablement`, then copy the Consumer Key from `Manage Consumer Details`. **Type**: `string` ### [](#client_secret)`client_secret` Consumer Secret of the Salesforce Connected App, paired with `client_id`. Sensitive — prefer environment variable interpolation (`${SALESFORCE_CLIENT_SECRET}`) over inlining. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#columns)`columns[]` Ordered list of field API names to retrieve. SOQL does not accept `*` — every field must be listed explicitly. Standard fields use their documented names; custom fields end with `__c`. Relationship fields traverse parents via dot notation (`Account.Name`, `Owner.Manager.Email`) up to 5 levels deep. Requesting a non-existent or non-queryable field fails at Connect time with a SOQL compile error. **Type**: `array` ```yaml # Examples: columns: - Id - Name - LastModifiedDate # --- columns: - Id - Account.Name - Owner.Email # --- columns: - Id - MyCustom__c ``` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` TCP socket configuration. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#object)`object` The sObject API name to SELECT from. Case-sensitive; uses the API name, not the display label. Standard objects use the noun (`Account`, `Opportunity`); custom objects end with `_c_`_; Big Objects end with_ `b`; External Objects end with `__x`. Confirm the exact API name in Setup → Object Manager. **Type**: `string` ```yaml # Examples: object: Account # --- object: Contact # --- object: MyCustom__c ``` ### [](#org_url)`org_url` Salesforce instance base URL for your org, protocol included and no trailing slash. Used as the base for both the OAuth token endpoint and REST queries. Production orgs use `[https://{my-domain}.my.salesforce.com](https://{my-domain}.my.salesforce.com)`; sandboxes use `[https://{my-domain}.sandbox.my.salesforce.com](https://{my-domain}.sandbox.my.salesforce.com)`. Legacy instance URLs (`[https://na123.salesforce.com](https://na123.salesforce.com)`) still work but My Domain URLs are strongly recommended by Salesforce. **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#prefix)`prefix` Optional SOQL fragment inserted before the SELECT keyword. Rarely needed — provided for forward compatibility with future SOQL extensions or Bulk API framing. **Type**: `string` ### [](#suffix)`suffix` Optional SOQL fragment appended after the WHERE clause. Typical uses: `ORDER BY` for deterministic pagination, `LIMIT` to cap result size, `FOR REFERENCE` / `FOR VIEW` to mark records for Chatter tracking. **Type**: `string` ```yaml # Examples: suffix: ORDER BY LastModifiedDate DESC # --- suffix: ORDER BY Id LIMIT 1000 # --- suffix: ORDER BY CreatedDate DESC LIMIT 10000 ``` ### [](#where)`where` Optional SOQL WHERE body, without the `WHERE` keyword. `?` placeholders are substituted client-side from `args_mapping` with SOQL literal escaping (quoted strings, ISO-8601 datetimes). Supports the full WHERE grammar: `AND`/`OR`/`NOT`, `LIKE`, `IN`, date literals (`TODAY`, `LAST_N_DAYS:7`), subqueries. Date/datetime comparisons require ISO-8601 with explicit timezone. **Type**: `string` ```yaml # Examples: where: LastModifiedDate > ? # --- where: Status__c = ? AND CreatedDate > ? # --- where: OwnerId IN (?, ?) ``` --- # Page 292: schema_registry **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/schema_registry.md --- # schema_registry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: schema_registry latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/schema_registry page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/schema_registry.adoc description: Reads schemas from SchemaRegistry. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Reads schemas from a schema registry. You can use this connector to extract and back up schemas during a data migration. This input uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml inputs: label: "" schema_registry: url: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" schema_registry: url: "" # No default (required) include_deleted: false subject_filter: "" fetch_in_order: true tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `schema_registry_subject` - `schema_registry_subject_compatibility_level` - `schema_registry_version` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#example)Example This example reads all schemas from a schema registry that are associated with subjects matching the `^foo.*` filter, including deleted schemas. ```yaml input: schema_registry: url: http://localhost:8081 include_deleted: true subject_filter: ^foo.* ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#fetch_in_order)`fetch_in_order` Indicate whether to fetch all schemas from the schema registry service and sort them by ID. Set this value to `true` if you use schemas that refer to other schemas (schema references). **Type**: `bool` **Default**: `true` ### [](#include_deleted)`include_deleted` Include deleted entities. **Type**: `bool` **Default**: `false` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication for secure data transmission from your schema registry to this component. This feature is in beta and may change in future releases. **Type**: `object` ### [](#jwt-claims)`jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A PEM-encoded file containing a private key that is formatted using either PKCS1 or PKCS8 standards. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The method used to sign the token, such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#subject_filter)`subject_filter` Include only subjects which match the regular expression filter, or leave this field value blank to select all subjects. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 293: sequence **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence.md --- # sequence > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sequence latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/sequence page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/sequence.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/sequence.adoc description: Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and so on. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and so on. #### Common ```yml inputs: label: "" sequence: inputs: [] # No default (required) ``` #### Advanced ```yml inputs: label: "" sequence: sharded_join: type: none id_path: "" iterations: 1 merge_strategy: array inputs: [] # No default (required) ``` This input is useful for consuming from inputs that have an explicit end but must not be consumed in parallel. ## [](#examples)Examples ### [](#end-of-stream-message)End of Stream Message A common use case for sequence might be to generate a message at the end of our main input. With the following config once the records within `./dataset.csv` are exhausted our final payload `{"status":"finished"}` will be routed through the pipeline. ```yaml input: sequence: inputs: - file: paths: [ ./dataset.csv ] scanner: csv: {} - generate: count: 1 mapping: 'root = {"status":"finished"}' ``` ### [](#joining-data-simple)Joining Data (Simple) Redpanda Connect can be used to join unordered data from fragmented datasets in memory by specifying a common identifier field and a number of sharded iterations. For example, given two CSV files, the first called "main.csv", which contains rows of user data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second called "hobbies.csv" that, for each user, contains zero or more rows of hobbies: ```csv uuid,hobby CCC,pokemon go AAA,rowing AAA,golf ``` We can parse and join this data into a single dataset: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid merge_strategy: array inputs: - file: paths: - ./hobbies.csv - ./main.csv scanner: csv: {} ``` ### [](#joining-data-advanced)Joining Data (Advanced) In this example we are able to join unordered and fragmented data from a combination of CSV files and newline-delimited JSON documents by specifying multiple sequence inputs with their own processors for extracting the structured data. The first file "main.csv" contains straight forward CSV data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second file called "hobbies.ndjson" contains JSON documents, one per line, that associate an identifier with an array of hobbies. However, these data objects are in a nested format: ```json {"document":{"uuid":"CCC","hobbies":[{"type":"pokemon go"}]}} {"document":{"uuid":"AAA","hobbies":[{"type":"rowing"},{"type":"golf"}]}} ``` And so we will want to map these into a flattened structure before the join, and then we will end up with a single dataset that looks like this: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid iterations: 10 merge_strategy: array inputs: - file: paths: [ ./main.csv ] scanner: csv: {} - file: paths: [ ./hobbies.ndjson ] scanner: lines: {} processors: - mapping: | root.uuid = this.document.uuid root.hobbies = this.document.hobbies.map_each(this.type) ``` ## [](#fields)Fields ### [](#inputs)`inputs[]` An array of inputs to read from sequentially. **Type**: `array` ### [](#sharded_join)`sharded_join` EXPERIMENTAL: Provides a way to perform outer joins of arbitrarily structured and unordered data resulting from the input sequence, even when the overall size of the data surpasses the memory available on the machine. When configured the sequence of inputs will be consumed one or more times according to the number of iterations, and when more than one iteration is specified each iteration will process an entirely different set of messages by sharding them by the ID field. Increasing the number of iterations reduces the memory consumption at the cost of needing to fully parse the data each time. Each message must be structured (JSON or otherwise processed into a structured form) and the fields will be aggregated with those of other messages sharing the ID. At the end of each iteration the joined messages are flushed downstream before the next iteration begins, hence keeping memory usage limited. **Type**: `object` ### [](#sharded_join-id_path)`sharded_join.id_path` A [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) that points to a common field within messages of each fragmented data set and can be used to join them. Messages that are not structured or are missing this field will be dropped. This field must be set in order to enable joins. **Type**: `string` **Default**: `""` ### [](#sharded_join-iterations)`sharded_join.iterations` The total number of iterations (shards), increasing this number will increase the overall time taken to process the data, but reduces the memory used in the process. The real memory usage required is significantly higher than the real size of the data and therefore the number of iterations should be at least an order of magnitude higher than the available memory divided by the overall size of the dataset. **Type**: `int` **Default**: `1` ### [](#sharded_join-merge_strategy)`sharded_join.merge_strategy` The chosen strategy to use when a data join would otherwise result in a collision of field values. The strategy `array` means non-array colliding values are placed into an array and colliding arrays are merged. The strategy `replace` replaces old values with new values. The strategy `keep` keeps the old value. **Type**: `string` **Default**: `array` **Options**: `array`, `replace`, `keep` ### [](#sharded_join-type)`sharded_join.type` The type of join to perform. A `full-outer` ensures that all identifiers seen in any of the input sequences are sent, and is performed by consuming all input sequences before flushing the joined results. An `outer` join consumes all input sequences but only writes data joined from the last input in the sequence, similar to a left or right outer join. With an `outer` join if an identifier appears multiple times within the final sequence input it will be flushed each time it appears. `full-outter` and `outter` have been deprecated in favour of `full-outer` and `outer`. **Type**: `string` **Default**: `none` **Options**: `none`, `full-outer`, `outer`, `full-outter`, `outter` --- # Page 294: sftp **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sftp.md --- # sftp > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sftp latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/sftp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/sftp.adoc description: Consumes files from an SFTP server. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consumes files from an SFTP server. #### Common ```yml inputs: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` #### Advanced ```yml inputs: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" max_sftp_sessions: 10 paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} delete_on_finish: false watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `sftp_path` - `sftp_mod_time` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The private key used to authenticate with the SFTP server. This field provides an alternative to the [`private_key_file`](#credentials-private_key_file). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the [`private_key`](#credentials-private_key) field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` A passphrase for the private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#delete_on_finish)`delete_on_finish` Whether to delete files from the server once they are processed. **Type**: `bool` **Default**: `false` ### [](#max_sftp_sessions)`max_sftp_sessions` The maximum number of SFTP sessions. **Type**: `int` **Default**: `10` ### [](#paths)`paths[]` A list of paths to consume sequentially. Glob patterns are supported. **Type**: `array` ### [](#scanner)`scanner` The [scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#watcher)`watcher` An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files. **Type**: `object` ### [](#watcher-cache)`watcher.cache` A [cache resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) for storing the paths of files already consumed. **Type**: `string` **Default**: `""` ### [](#watcher-enabled)`watcher.enabled` Whether file watching is enabled. **Type**: `bool` **Default**: `false` ### [](#watcher-minimum_age)`watcher.minimum_age` The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to. **Type**: `string` **Default**: `1s` ```yaml # Examples: minimum_age: 10s # --- minimum_age: 1m # --- minimum_age: 10m ``` ### [](#watcher-poll_interval)`watcher.poll_interval` The interval between each attempt to scan the target paths for new files. **Type**: `string` **Default**: `1s` ```yaml # Examples: poll_interval: 100ms # --- poll_interval: 1s ``` --- # Page 295: slack_users **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/slack_users.md --- # slack_users > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: slack_users latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/slack_users page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/slack_users.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/slack_users.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Returns [the full profile](https://api.slack.com/methods/users.list#examples) of all users in your Slack organization using the API method [users.list](https://api.slack.com/methods/users.list). Optionally, you can filter the list of returned users by team ID. This input is useful when you need to: - Join user information to Slack posts. - Ingest user information into a data lakehouse to create joins with other fields. ```yml inputs: label: "" slack_users: bot_token: "" # No default (required) team_id: "" auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` Your [Slack bot user’s OAuth token](https://api.slack.com/concepts/token-types), which must have the [`users.read` scope](https://api.slack.com/scopes/users:read) to access your Slack organization. **Type**: `string` ### [](#team_id)`team_id` The encoded ID of a Slack team by which to filter the list of returned users, which you can get from the [`team.info` Slack API method](https://api.slack.com/methods/team.info). If `team_id` is left empty, users from all teams within the organization are returned. **Type**: `string` **Default**: `""` --- # Page 296: slack **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/slack.md --- # slack > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: slack latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/slack page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/slack.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/slack.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Connects to Slack using [Socket Mode](https://api.slack.com/apis/socket-mode), and can receive events, interactions (automated and user-initiated), and slash commands. This input is useful for: - Building bots that can query or write data. - Sending events to data warehouses. You could also try pairing this input with Redpanda Connect’s AI processors, which use the prefixes `cohere`, `openai`, and `ollama`. ```yml inputs: label: "" slack: app_token: "" # No default (required) bot_token: "" # No default (required) auto_replay_nacks: true ``` See also: [Examples](#examples) ## [](#metadata)Metadata Each message emitted from this input has an `@type` metadata flag to indicate the event type, either `"events_api"`, `"interactions"`, or `"slash_commands"`. ## [](#fields)Fields ### [](#app_token)`app_token` The app-level token to use to authenticate and connect to Slack. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the [`connections.write` scope](https://api.slack.com/scopes/connections:write) to access your Slack app’s [Socket Mode WebSocket URL](https://api.slack.com/methods/apps.connections.open). **Type**: `string` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 297: spicedb_watch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/spicedb_watch.md --- # spicedb_watch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: spicedb_watch page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/spicedb_watch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/spicedb_watch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/spicedb_watch.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-11-19" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Consumes messages from the [Watch API](https://buf.build/authzed/api/docs/main:authzed.api.v1#authzed.api.v1.WatchService.Watch) of a [SpiceDB](https://authzed.com/docs/spicedb/getting-started/discovering-spicedb) instance. This input is useful if you have downstream applications that need to react to real-time changes in data managed by SpiceDB. #### Common ```yml inputs: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" cache: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" max_receive_message_bytes: 4MB cache: "" # No default (required) cache_key: authzed.com/spicedb/watch/last_zed_token tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` ## [](#authentication)Authentication For this input to authenticate with your SpiceDB instance, you must provide: - The [`endpoint`](#endpoint) of the SpiceDB instance - A [bearer token](#bearer_token) ## [](#configure-a-cache)Configure a cache You must use a cache resource to store the [ZedToken](https://authzed.com/docs/spicedb/concepts/consistency#zedtokens) (ID) of the latest message consumed and acknowledged by this input. Ideally, the cache should persist across restarts. This means that every time the input is initialized, it starts reading from the newest data updates. The following example uses a [`redis` cache](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/redis/). ```yml # Example input: label: "" spicedb_watch: endpoint: grpc.authzed.com:443 bearer_token: "" cache: "spicedb_cache" cache_resources: - label: "spicedb_cache" redis: url: redis://:6379 ``` To learn more about cache configuration, see the [Caches section](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/), which includes a range of cache components. ## [](#fields)Fields ### [](#bearer_token)`bearer_token` The SpiceDB bearer token to use to authenticate with your SpiceDB instance. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: bearer_token: t_your_token_here_1234567deadbeef ``` ### [](#cache)`cache` The [cache resource](#configure-a-cache) that you must configure to store the ZedToken (ID) of the last message processed. The ZedToken is stored in the cache within the `ACK` function of the message. This means that a ZedToken is only stored when a message is successfully routed through all processors and outputs in the data pipeline. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier to use when storing the ZedToken (ID) of the last message received. **Type**: `string` **Default**: `authzed.com/spicedb/watch/last_zed_token` ### [](#endpoint)`endpoint` The endpoint of your SpiceDB instance. **Type**: `string` ```yaml # Examples: endpoint: grpc.authzed.com:443 ``` ### [](#max_receive_message_bytes)`max_receive_message_bytes` The maximum message size (in bytes) this input can receive. If a message exceeds this limit, an `rpc error` is written to the Redpanda Connect logs. **Type**: `string` **Default**: `4MB` ```yaml # Examples: max_receive_message_bytes: 100MB # --- max_receive_message_bytes: 50mib ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 298: splunk **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/splunk.md --- # splunk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: splunk latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/splunk page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/splunk.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/splunk.adoc description: Consumes messages from Splunk. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consumes messages from Splunk. #### Common ```yml inputs: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#password)`password` Splunk account password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#query)`query` Splunk search query. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` Full HTTP Search API endpoint URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/search/v2/jobs/export ``` ### [](#user)`user` Splunk account user. **Type**: `string` --- # Page 299: sql_raw **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sql_raw.md --- # sql_raw > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_raw latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/sql_raw page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/sql_raw.adoc description: Executes a select query and creates a message for each row received. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a select query and creates a message for each row received. #### Common ```yml inputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` When the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate or for the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to execute. ## [](#examples)Examples ### [](#consumes-an-sql-table-using-a-query-as-an-input)Consumes an SQL table using a query as an input. Here we perform an aggregate over a list of names in a table that are less than 3600 seconds old. ```yaml input: sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT name, count(*) FROM person WHERE last_updated < $1 GROUP BY name;" args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#query)`query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: SELECT * FROM footable WHERE user_id = $1; ``` --- # Page 300: sql_select **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sql_select.md --- # sql_select > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_select latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/sql_select page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/sql_select.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/sql_select.adoc description: Executes a select query and creates a message for each row received. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a select query and creates a message for each row received. #### Common ```yml inputs: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` Once the rows from the query are exhausted this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/sequence/) to execute). ## [](#examples)Examples ### [](#consume-a-table-postgresql)Consume a Table (PostgreSQL) Here we define a pipeline that will consume all rows from a table created within the last hour by comparing the unix timestamp stored in the row column "created\_at": ```yaml input: sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: created_at >= ? args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to select. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` The table to select from. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks, and will automatically be converted to dollar syntax when the postgres or clickhouse drivers are used. **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 301: timeplus **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/timeplus.md --- # timeplus > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: timeplus latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/inputs/timeplus page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/inputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/inputs/timeplus.adoc description: Executes a query on Timeplus Enterprise and creates a message from each row received. page-git-created-date: "2024-11-19" page-git-modified-date: "2026-05-26" --- Executes a streaming or table query on [Timeplus Enterprise (Cloud or Self-Hosted)](https://docs.timeplus.com/) or the `timeplusd` component, and creates a structured message for each table row received. If you execute a streaming query, this input runs until the query terminates. For table queries, it shuts down after all rows returned by the query are exhausted. ```yml inputs: label: "" timeplus: query: "" # No default (required) url: tcp://localhost:8463 workspace: "" # No default (optional) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) ``` ## [](#examples)Examples ### [](#from-timeplus-enterprise-cloud-via-http)From Timeplus Enterprise Cloud via HTTP You will need to create API Key on Timeplus Enterprise Cloud Web console first and then set the `apikey` field. ```yaml input: timeplus: url: https://us-west-2.timeplus.cloud workspace: my_workspace_id query: select * from iot apikey: ``` ### [](#from-timeplus-enterprise-self-hosted-via-http)From Timeplus Enterprise (self-hosted) via HTTP For self-hosted Timeplus Enterprise, you will need to specify the username and password as well as the URL of the App server ```yaml input: timeplus: url: http://localhost:8000 workspace: my_workspace_id query: select * from iot username: username password: pw ``` ### [](#from-timeplus-enterprise-self-hosted-via-tcp)From Timeplus Enterprise (self-hosted) via TCP Make sure the the schema of url is tcp ```yaml input: timeplus: url: tcp://localhost:8463 query: select * from iot username: timeplus password: timeplus ``` ## [](#fields)Fields ### [](#apikey)`apikey` The API key for the Timeplus Enterprise REST API. You need to generate the key in the web console of Timeplus Enterprise (Cloud). This field is required if you are reading messages from Timeplus Enterprise (Cloud). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#password)`password` The password for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#query)`query` The query to execute on Timeplus Enterprise (Cloud or Self-Hosted) or `timeplusd`. **Type**: `string` ```yaml # Examples: query: select * from iot # --- query: select count(*) from table(iot) ``` ### [](#url)`url` The URL of your Timeplus instance, which should always include the schema and host. **Type**: `string` **Default**: `tcp://localhost:8463` ### [](#username)`username` The username for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. **Type**: `string` ### [](#workspace)`workspace` The ID of the workspace you want to read messages from. This field is required if you are connecting to Timeplus Enterprise (Cloud or Self-Hosted) using HTTP. **Type**: `string` --- # Page 302: Logger **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/logger/about.md --- # Logger > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Logger latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/logger/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/logger/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/logger/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect logging prints to stdout (or stderr if your output is stdout) and is formatted as [logfmt](https://brandur.org/logfmt) by default. Use these configuration options to change both the logging formats as well as the destination of logs. #### Common ```yaml # Common config fields, showing default values logger: level: INFO format: logfmt add_timestamp: false static_fields: '@service': redpanda-connect ``` #### Advanced ```yaml # All config fields, showing default values logger: level: INFO format: logfmt add_timestamp: false level_name: level timestamp_name: time message_name: msg static_fields: '@service': redpanda-connect file: path: "" rotate: false rotate_max_age_days: 0 ``` ## [](#fields)Fields The schema of the `logger` section is as follows: ### [](#level)`level` Set the minimum severity level for emitting logs. **Type**: `string` **Default**: `"INFO"` Options: `OFF` , `FATAL` , `ERROR` , `WARN` , `INFO` , `DEBUG` , `TRACE` , `ALL` , `NONE` ### [](#format)`format` Set the format of emitted logs. **Type**: `string` **Default**: `"logfmt"` Options: `json` , `logfmt` ### [](#add_timestamp)`add_timestamp` Whether to include timestamps in logs. **Type**: `bool` **Default**: `false` ### [](#level_name)`level_name` The name of the level field added to logs when the `format` is `json`. **Type**: `string` **Default**: `"level"` ### [](#timestamp_name)`timestamp_name` The name of the timestamp field added to logs when `add_timestamp` is set to `true` and the `format` is `json`. **Type**: `string` **Default**: `"time"` ### [](#message_name)`message_name` The name of the message field added to logs when the `format` is `json`. **Type**: `string` **Default**: `"msg"` ### [](#static_fields)`static_fields` A map of key/value pairs to add to each structured log. **Type**: `object` **Default**: `{"@service":"redpanda-connect"}` ### [](#file)`file` Experimental: Specify fields for optionally writing logs to a file. **Type**: `object` ### [](#file-path)`file.path` The file path to write logs to, if the file does not exist it will be created. Leave this field empty or unset to disable file based logging. **Type**: `string` **Default**: `""` ### [](#file-rotate)`file.rotate` Whether to rotate log files automatically. **Type**: `bool` **Default**: `false` ### [](#file-rotate_max_age_days)`file.rotate_max_age_days` The maximum number of days to retain old log files based on the timestamp encoded in their filename, after which they are deleted. Setting to zero disables this mechanism. **Type**: `int` **Default**: `0` --- # Page 303: Metrics **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about.md --- # Metrics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Metrics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/metrics/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/metrics/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/metrics/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect emits lots of metrics in order to expose how components configured within your pipeline are behaving. You can configure exactly where these metrics end up with the config field `metrics`, which describes a metrics format and destination. For example, if you wished to push them via the StatsD protocol you could use this configuration: ```yaml metrics: statsd: address: localhost:8125 flush_period: 100ms ``` Redpanda Connect automatically [exports detailed metrics](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/monitor-connect/) for each component of your data pipeline to a Prometheus endpoint. ## [](#timings)Timings It’s worth noting that timing metrics within Redpanda Connect are measured in nanoseconds and are therefore named with a `_ns` suffix. However, some exporters do not support this level of precision and are downgraded, or have the unit converted for convenience. In these cases the exporter documentation outlines the conversion and why it is made. ## [](#metric-names)Metric names Each major Redpanda Connect component type emits one or more metrics with the name prefixed by the type. These metrics are intended to provide an overview of behavior, performance and health. Some specific component implementations may provide their own unique metrics on top of these standardized ones, these extra metrics can be found listed on their respective documentation pages. ## [](#inputs)Inputs - `input_received`: A count of the number of messages received by the input. - `input_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read up to the moment the message has either been acknowledged by an output, has been stored within a buffer, or has been rejected (nacked). - `batch_created`: A count of each time an input-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `input_connection_up`: For continuous stream based inputs represents a count of the number of the times the input has successfully established a connection to the target source. For poll based inputs that do not retain an active connection this value will increment once. - `input_connection_failed`: For continuous stream based inputs represents a count of the number of times the input has failed to establish a connection to the target source. - `input_connection_lost`: For continuous stream based inputs represents a count of the number of times the input has lost a previously established connection to the target source. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on input type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#buffers)Buffers - `buffer_received`: A count of the number of messages written to the buffer. - `buffer_batch_received`: A count of the number of message batches written to the buffer. - `buffer_sent`: A count of the number of messages read from the buffer. - `buffer_batch_sent`: A count of the number of message batches read from the buffer. - `buffer_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read from the buffer up to the moment it has been acknowledged by the output. - `batch_created`: A count of each time a buffer-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. ### [](#processors)Processors - `processor_received`: A count of the number of messages the processor has been executed upon. - `processor_batch_received`: A count of the number of message batches the processor has been executed upon. - `processor_sent`: A count of the number of messages the processor has returned. - `processor_batch_sent`: A count of the number of message batches the processor has returned. - `processor_error`: A count of the number of times the processor has errored. In cases where an error is batch-wide the count is incremented by one, and therefore would not match the number of messages. - `processor_latency_ns`: Latency of message processing in nanoseconds. When a processor acts upon a batch of messages this latency measures the time taken to process all messages of the batch. ### [](#outputs)Outputs - `output_sent`: A count of the number of messages sent by the output. - `output_batch_sent`: A count of the number of message batches sent by the output. - `output_error`: A count of the number of send attempts that have failed. On failed batched sends this count is incremented once only. - `output_latency_ns`: Latency of writes in nanoseconds. This metric may not be populated by outputs that are pull-based such as the `http_server`. - `batch_created`: A count of each time an output-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `output_connection_up`: For continuous stream based outputs represents a count of the number of the times the output has successfully established a connection to the target sink. For poll based outputs that do not retain an active connection this value will increment once. - `output_connection_failed`: For continuous stream based outputs represents a count of the number of times the output has failed to establish a connection to the target sink. - `output_connection_lost`: For continuous stream based outputs represents a count of the number of times the output has lost a previously established connection to the target sink. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on output type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#caches)Caches All cache metrics have a label `operation` denoting the operation that triggered the metric series, one of; `add`, `get`, `set` or `delete`. - `cache_success`: A count of the number of successful cache operations. - `cache_error`: A count of the number of cache operations that resulted in an error. - `cache_latency_ns`: Latency of operations in nanoseconds. - `cache_not_found`: A count of the number of get operations that yielded no value due to the item not being found. This count is separate from `cache_error`. - `cache_duplicate`: A count of the number of add operations that were aborted due to the key already existing. This count is separate from `cache_error`. ### [](#rate-limits)Rate limits - `rate_limit_checked`: A count of the number of times the rate limit has been probed. - `rate_limit_triggered`: A count of the number of times the rate limit has been triggered by a probe. - `rate_limit_error`: A count of the number of times the rate limit has errored when probed. ## [](#metric-labels)Metric labels The standard metric names are unique to the component type, but a benthos config may consist of any number of component instantiations. In order to provide a metrics series that is unique for each instantiation Redpanda Connect adds labels (or tags) that uniquely identify the instantiation. These labels are as follows: ### [](#path)`path` The `path` label contains a string representation of the position of a component instantiation within a config in a format that would locate it within a Bloblang mapping, beginning at `root`. This path is a best attempt and may not exactly represent the source component position in all cases and is intended to be used for assisting observability only. This is the highest cardinality label since paths will change as configs are updated and expanded. It is therefore worth removing this label with a [mapping](#metric-mapping) in cases where you wish to restrict the number of unique metric series. ### [](#label)`label` The `label` label contains the unique label configured for a component emitting the metric series, or is empty for components that do not have a configured label. This is the most useful label for uniquely identifying a series for a component. ### [](#stream)`stream` The `stream` label is present in a metric series emitted from a stream config executed when Redpanda Connect is running in streams mode, and is populated with the stream name. ## [](#example)Example The following Redpanda Connect configuration: ```yaml input: label: foo http_server: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() output: label: bar stdout: {} metrics: prometheus: {} ``` Would produce the following metrics series: ```text input_latency_ns{label="foo",path="root.input"} input_received{endpoint="post",label="foo",path="root.input"} input_received{endpoint="websocket",label="foo",path="root.input"} processor_batch_received{label="",path="root.pipeline.processors.0"} processor_batch_sent{label="",path="root.pipeline.processors.0"} processor_error{label="",path="root.pipeline.processors.0"} processor_latency_ns{label="",path="root.pipeline.processors.0"} processor_received{label="",path="root.pipeline.processors.0"} processor_sent{label="",path="root.pipeline.processors.0"} output_batch_sent{label="bar",path="root.output"} output_connection_failed{label="bar",path="root.output"} output_connection_lost{label="bar",path="root.output"} output_connection_up{label="bar",path="root.output"} output_error{label="bar",path="root.output"} output_latency_ns{label="bar",path="root.output"} output_sent{label="bar",path="root.output"} ``` ## [](#metric-mapping)Metric mapping Since Redpanda Connect emits a large variety of metrics it is often useful to restrict or modify the metrics that are emitted. This can be done using the [Bloblang mapping language](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) in the field `metrics.mapping`. This is a mapping executed for each metric that is registered within the Redpanda Connect service and allows you to delete an entire series, modify the series name and delete or modify individual labels. Within the mapping the input document (referenced by the keyword `this`) is a string value containing the metric name, and the resulting document (referenced by the keyword `root`) must be a string value containing the resulting name. As is standard in Bloblang mappings, if the value of `root` is not assigned within the mapping then the metric name remains unchanged. If the value of `root` is `deleted()` then the metric series is dropped. Labels can be referenced as metadata values with the function `meta`, where if the label does not exist in the series being mapped the value `null` is returned. Labels can be changed by using meta assignments, and can be assigned `deleted()` in order to remove them. For example, the following mapping removes all but the `label` label entirely, which reduces the cardinality of each series. It also renames the `label` (for some reason) so that labels containing meows now contain woofs. Finally, the mapping restricts the metrics emitted to only three series; one for the input count, one for processor errors, and one for the output count, it does this by looking up metric names in a static array of allowed names, and if not present the `root` is assigned `deleted()`: ```yaml metrics: mapping: | # Delete all pre-existing labels meta = deleted() # Re-add the `label` label with meows replaced with woofs meta label = meta("label").replace("meow", "woof") # Delete all metric series that aren't in our list root = if ![ "input_received", "processor_error", "output_sent", ].contains(this) { deleted() } prometheus: use_histogram_timing: false ``` --- # Page 304: none **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/none.md --- # none > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: none latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/metrics/none page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/metrics/none.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/metrics/none.adoc description: Disable metrics entirely. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Disable metrics entirely. ```yml # Config fields, showing default values metrics: none: {} mapping: "" ``` --- # Page 305: open_telemetry_collector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/open_telemetry_collector.md --- # open_telemetry_collector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: open_telemetry_collector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/metrics/open_telemetry_collector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/metrics/open_telemetry_collector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/metrics/open_telemetry_collector.adoc description: Send metrics to an Open Telemetry collector. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-11" --- Send metrics to an [Open Telemetry collector](https://opentelemetry.io/docs/collector/). Exports Redpanda Connect metrics to one or more OpenTelemetry Collector endpoints over HTTP or gRPC for aggregation and onward export. Metrics are encoded using the OpenTelemetry Metrics protocol and can be sent to any collector endpoint that supports OTLP. You can configure multiple collector endpoints (both HTTP and gRPC simultaneously). All configured endpoints will receive the same metrics data. This is useful for redundancy or sending metrics to multiple observability platforms. ### Common ```yml metrics: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) ``` ### Advanced ```yml metrics: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) tags: {} ``` ## [](#fields)Fields ### [](#grpc)`grpc[]` A list of grpc collectors. **Type**: `array` ### [](#grpc-address)`grpc[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4317 ``` ### [](#grpc-secure)`grpc[].secure` Connect to the collector with client transport security **Type**: `bool` **Default**: `false` ### [](#http)`http[]` A list of http collectors. **Type**: `array` ### [](#http-address)`http[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4318 ``` ### [](#http-secure)`http[].secure` Connect to the collector over HTTPS **Type**: `bool` **Default**: `false` ### [](#service)`service` The name of the service in metrics. **Type**: `string` **Default**: `benthos` ### [](#tags)`tags` A map of tags to add to all exported spans and metrics. **Type**: `object` **Default**: `{}` ## [](#usage)Usage The most common setup uses a local OpenTelemetry Collector running as a sidecar or daemon, which then forwards metrics to your observability backend: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 ``` For production deployments with remote collectors, enable TLS: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: otel-collector.example.com:4317 secure: true ``` Use the `tags` field to add labels to all exported metrics for filtering and grouping in your observability platform: ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 tags: environment: production cluster: kafka-01 ``` --- # Page 306: prometheus **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/prometheus.md --- # prometheus > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: prometheus latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/metrics/prometheus page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/metrics/prometheus.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/metrics/prometheus.adoc description: Host endpoints (/metrics and /stats) for Prometheus scraping. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Host endpoints (`/metrics` and `/stats`) for Prometheus scraping. ### Common ```yml metrics: prometheus: {} ``` ### Advanced ```yml metrics: prometheus: use_histogram_timing: false histogram_buckets: [] summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 add_process_metrics: false add_go_metrics: false push_url: "" # No default (optional) push_interval: "" # No default (optional) push_job_name: benthos_push push_basic_auth: username: "" password: "" file_output_path: "" ``` ## [](#fields)Fields ### [](#add_go_metrics)`add_go_metrics` Whether to export Go runtime metrics such as GC pauses in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#add_process_metrics)`add_process_metrics` Whether to export process metrics such as CPU and memory usage in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#file_output_path)`file_output_path` An optional file path to write all prometheus metrics on service shutdown. **Type**: `string` **Default**: `""` ### [](#histogram_buckets)`histogram_buckets[]` Timing metrics histogram buckets (in seconds). If left empty defaults to DefBuckets ([https://pkg.go.dev/github.com/prometheus/client\_golang/prometheus#pkg-variables](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#pkg-variables)). Applicable when `use_histogram_timing` is set to `true`. **Type**: `array` **Default**: `[]` ### [](#push_basic_auth)`push_basic_auth` The Basic Authentication credentials. **Type**: `object` ### [](#push_basic_auth-password)`push_basic_auth.password` The Basic Authentication password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#push_basic_auth-username)`push_basic_auth.username` The Basic Authentication username. **Type**: `string` **Default**: `""` ### [](#push_interval)`push_interval` The period of time between each push when sending metrics to a Push Gateway. **Type**: `string` ### [](#push_job_name)`push_job_name` An identifier for push jobs. **Type**: `string` **Default**: `benthos_push` ### [](#push_url)`push_url` An optional [Push Gateway URL](#push-gateway) to push metrics to. **Type**: `string` ### [](#summary_quantiles_objectives)`summary_quantiles_objectives[]` A list of timing metrics summary buckets (as quantiles). Applicable when `use_histogram_timing` is set to `false`. **Type**: `array` **Default**: ```yaml - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ```yaml # Examples: summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ### [](#summary_quantiles_objectives-error)`summary_quantiles_objectives[].error` Permissible margin of error for quantile calculations. Precise calculations in a streaming context (without prior knowledge of the full dataset) can be resource-intensive. To balance accuracy with computational efficiency, an error margin is introduced. For instance, if the 90th quantile (`0.9`) is determined to be `100ms` with a 1% error margin (`0.01`), the true value will fall within the `[99ms, 101ms]` range.) **Type**: `float` **Default**: `0` ### [](#summary_quantiles_objectives-quantile)`summary_quantiles_objectives[].quantile` Quantile value. **Type**: `float` **Default**: `0` ### [](#use_histogram_timing)`use_histogram_timing` Whether to export timing metrics as a histogram, if `false` a summary is used instead. When exporting histogram timings the delta values are converted from nanoseconds into seconds in order to better fit within bucket definitions. For more information on histograms and summaries refer to: [https://prometheus.io/docs/practices/histograms/](https://prometheus.io/docs/practices/histograms/). **Type**: `bool` **Default**: `false` ## [](#push-gateway)Push gateway The field `push_url` is optional and when set will trigger a push of metrics to a [Prometheus Push Gateway](https://prometheus.io/docs/instrumenting/pushing/) once Redpanda Connect shuts down. It is also possible to specify a `push_interval` which results in periodic pushes. The Push Gateway is useful for when Redpanda Connect instances are short lived. Do not include the "/metrics/jobs/…​" path in the push URL. If the Push Gateway requires HTTP Basic Authentication it can be configured with `push_basic_auth`. --- # Page 307: Outputs **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about.md --- # Outputs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Outputs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- An output config section looks like this: ```yaml output: label: my_s3_output aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' # Optional list of processing steps processors: - mapping: '{"message":this,"meta":{"link_count":this.links.length()}}' ``` ## [](#back-pressure)Back pressure Redpanda Connect outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Redpanda Connect will gracefully stop consuming until the issue is resolved. ## [](#retries)Retries When a Redpanda Connect output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka). It’s possible to instead have Redpanda Connect indefinitely retry an output until success with a [`retry`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/retry/) output. Some other outputs, such as the [`broker`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/broker/), might also retry indefinitely depending on their configuration. ## [](#dead-letter-queues)Dead letter queues It’s possible to create fallback outputs for when an output target fails using a [`fallback`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) output: ```yaml output: fallback: - aws_sqs: url: https://sqs.us-west-2.amazonaws.com/TODO/TODO max_in_flight: 20 - http_client: url: http://backup:1234/dlq verb: POST ``` ## [](#multiplexing-outputs)Multiplexing outputs There are a few different ways of multiplexing in Redpanda Connect, here’s a quick run through: ### [](#interpolation-multiplexing)Interpolation multiplexing Some output fields support [field interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/), which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service. For example, multiplexing against Kafka topics is a common pattern: ```yaml output: kafka: addresses: [ TODO:6379 ] topic: ${! meta("target_topic") } ``` Refer to the field documentation for a given output to see if it support interpolation. ### [](#switch-multiplexing)Switch multiplexing A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the [`switch` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/switch/): ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type.not_null() | "unknown" ``` ## [](#labels)Labels Outputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple outputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/). --- # Page 308: amqp_0_9 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/amqp_0_9.md --- # amqp_0_9 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: amqp_0_9 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/amqp_0_9 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/amqp_0_9.adoc description: Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ.Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml outputs: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) key: "" type: "" metadata: exclude_prefixes: [] max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) exchange_declare: enabled: false type: direct durable: true arguments: "" # No default (optional) key: "" type: "" content_type: application/octet-stream content_encoding: "" correlation_id: "" reply_to: "" expiration: "" message_id: "" user_id: "" app_id: "" metadata: exclude_prefixes: [] priority: "" max_in_flight: 64 persistent: false mandatory: false immediate: false timeout: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` The metadata fields from each message are delivered as headers. TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. You can use [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) to dynamically set values for the following fields: `key`, `exchange`, and `type`. ## [](#fields)Fields ### [](#app_id)`app_id` Set an application ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` The content encoding attribute of each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The MIME type of each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#correlation_id)`correlation_id` Set a unique correlation ID for each message using a dynamic interpolated expression to help match messages to responses. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#exchange)`exchange` The AMQP exchange to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#exchange_declare)`exchange_declare` Passively declares the [target exchange](#exchange) to check whether an exchange with the specified name exists and is configured correctly. If the exchange exists, then the passive declaration verifies that fields specified in this object match its properties. If the target exchange does not exist, this output creates it. **Type**: `object` ### [](#exchange_declare-arguments)`exchange_declare.arguments` Arguments for server-specific implementations of the exchange (optional). You can use arguments to configure additional parameters for exchange types that require them. **Type**: `object` ```yaml # Examples: arguments: alternate-exchange: my-ae ``` ### [](#exchange_declare-durable)`exchange_declare.durable` Whether the declared exchange is durable. **Type**: `bool` **Default**: `true` ### [](#exchange_declare-enabled)`exchange_declare.enabled` Whether to enable exchange declaration. **Type**: `bool` **Default**: `false` ### [](#exchange_declare-type)`exchange_declare.type` The type of the exchange, which determines how messages are routed to queues. > 📝 **NOTE** > > Dots (`.`) in message keys are only enforced in routing keys and message types for `topic` exchanges. **Type**: `string` **Default**: `direct` **Options**: `direct`, `fanout`, `topic`, `headers`, `x-custom` ### [](#expiration)`expiration` Set the TTL of each message in milliseconds. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#immediate)`immediate` Whether to set the immediate flag on published messages. When set to `true`, if there are no active consumers for a queue, the message is dropped instead of waiting. **Type**: `bool` **Default**: `false` ### [](#key)`key` The binding key to set for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#mandatory)`mandatory` Whether to set the mandatory flag on published messages. When set to `true`, a published message that cannot be routed to any queues is returned to the sender. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_id)`message_id` Set a message ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#persistent)`persistent` Whether to store delivered messages on disk. By default, message delivery is transient. **Type**: `bool` **Default**: `false` ### [](#priority)`priority` Set the priority of each message using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: priority: 0 # --- priority: ${! meta("amqp_priority") } # --- priority: ${! json("doc.priority") } ``` ### [](#reply_to)`reply_to` Set the name of the queue to which responses are sent using a dynamic interpolated expression. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#timeout)`timeout` The maximum period to wait for a message acknowledgment before abandoning it and attempting a resend. If this value is not set, the system waits indefinitely. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#type)`type` A custom message type to set for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#urls)`urls[]` A list of URLs to connect to. This input attempts to connect to each URL in the list, in order, until a successful connection is established. It then continues to use that URL until the connection is closed. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` ### [](#user_id)`user_id` Set the user ID to the name of the publisher. If this property is set by a publisher, its value must match the name of the user that opened the connection. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 309: arc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/arc.md --- # arc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: arc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/arc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/arc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/arc.adoc description: Writes data to an Arc database via the msgpack ingestion endpoint. page-git-created-date: "2026-04-20" page-git-modified-date: "2026-08-11" --- Writes data to an Arc database via the msgpack ingestion endpoint. This output sends data to an [Arc](https://github.com/Basekick-Labs/arc) columnar analytical database using its high-performance MessagePack ingestion endpoint. Arc supports two payload formats: - **columnar** (default): Transposes batched messages into column arrays. This is the recommended format, offering significantly faster ingestion. - **row**: Sends each message as an individual row record with fields and optional tags. Data is encoded as MessagePack and optionally compressed with zstd (recommended) or gzip before being sent to the Arc endpoint. > 📝 **NOTE** > > In columnar mode, all messages within a single batch must have the same set of fields. Arc validates that all column arrays have equal length and rejects batches with mismatched columns. Schema evolution across separate batches is fully supported. Use row format if messages within a batch have varying schemas. ### Common ```yml outputs: label: "" arc: base_url: "" # No default (required) timeout: 5s token: "" # No default (optional) database: default measurement: "" # No default (required) format: columnar tags_mapping: "" # No default (optional) compression: zstd max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ### Advanced ```yml outputs: label: "" arc: base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 token: "" # No default (optional) database: default measurement: "" # No default (required) format: columnar timestamp_field: "" timestamp_unit: auto tags_mapping: "" # No default (optional) compression: zstd max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` Base URL of the target service (e.g., [https://api.example.com](https://api.example.com)). TLS is enabled automatically for https URLs. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#compression)`compression` Compression algorithm for the request body. `zstd` is recommended for best decompression performance in Arc. **Type**: `string` **Default**: `zstd` **Options**: `zstd`, `gzip`, `none` ### [](#database)`database` The target database name. **Type**: `string` **Default**: `default` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#format)`format` The payload format. `columnar` transposes batch messages into column arrays for best performance. `row` sends each message as an individual record. **Type**: `string` **Default**: `columnar` **Options**: `columnar`, `row` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#measurement)`measurement` The measurement (table) name. Supports interpolation functions. **Type**: `string` ```yaml # Examples: measurement: cpu_metrics # --- measurement: ${!metadata("measurement")} # --- measurement: ${!json("type")} ``` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#tags_mapping)`tags_mapping` An optional Bloblang mapping to extract tags from each message. Only used in `row` format. The result must be a `map[string]string`. **Type**: `string` ```yaml # Examples: tags_mapping: root = {"host": this.hostname, "region": this.region} ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#timestamp_field)`timestamp_field` The field name within each message containing the timestamp. If empty, the current time is used. Supports Unix timestamps and RFC3339 strings. **Type**: `string` **Default**: `""` ### [](#timestamp_unit)`timestamp_unit` The unit of a numeric timestamp field. `auto` detects the unit based on magnitude. Ignored when `timestamp_field` is empty. **Type**: `string` **Default**: `auto` **Options**: `us`, `ms`, `s`, `ns`, `auto` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#token)`token` Bearer token for authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). --- # Page 310: aws_dynamodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_dynamodb.md --- # aws_dynamodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_dynamodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_dynamodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_dynamodb.adoc description: Inserts items into a DynamoDB table. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Inserts items into a DynamoDB table. #### Common ```yml outputs: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} ttl: "" ttl_key: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 3 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` The field `string_columns` is a map of column names to string values, where the values are [function interpolated](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) per message of a batch. This allows you to populate string columns of an item by extracting fields within the document payload or metadata like follows: ```yml string_columns: id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} full_content: ${!content()} ``` The field `json_map_columns` is a map of column names to json paths, where the [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) is extracted from each document and converted into a map value. Both an empty path and the path `.` are interpreted as the root of the document. This allows you to populate map columns of an item like follows: ```yml json_map_columns: user: path.to.user whole_document: . ``` A column name can be empty: ```yml json_map_columns: "": . ``` In which case the top level document fields will be written at the root of the item, potentially overwriting previously defined column values. If a path is not found within a document the column will not be populated. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#json_map_columns)`json_map_columns` A map of column keys to [field paths](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) pointing to value data within messages. **Type**: `object` **Default**: `{}` ```yaml # Examples: json_map_columns: user: path.to.user whole_document: . # --- json_map_columns: "": . ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `3` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#string_columns)`string_columns` A map of column keys to string values to store. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: string_columns: full_content: ${!content()} id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} ``` ### [](#table)`table` The table to store messages in. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl)`ttl` An optional TTL to set for items, calculated from the moment the message is sent. **Type**: `string` **Default**: `""` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. **Type**: `string` **Default**: `""` --- # Page 311: aws_kinesis_firehose **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_kinesis_firehose.md --- # aws_kinesis_firehose > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_kinesis_firehose latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_kinesis_firehose page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_kinesis_firehose.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_kinesis_firehose.adoc description: Sends messages to a Kinesis Firehose delivery stream. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to a Kinesis Firehose delivery stream. #### Common ```yml outputs: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 312: aws_kinesis **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_kinesis.md --- # aws_kinesis > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_kinesis latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_kinesis page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_kinesis.adoc description: Sends messages to a Kinesis stream. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to a Kinesis stream. #### Common ```yml outputs: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) hash_key: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` Both the `partition_key`(required) and `hash_key` (optional) fields can be dynamically set using function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). When sending batched messages the interpolations are performed per message part. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#hash_key)`hash_key` A optional hash key for partitioning messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#partition_key)`partition_key` A required key for partitioning messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. Streams can either be specified by their name or full ARN. **Type**: `string` ```yaml # Examples: stream: foo # --- stream: arn:aws:kinesis:*:111122223333:stream/my-stream ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 313: aws_s3 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_s3.md --- # aws_s3 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_s3 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_s3 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_s3.adoc description: Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded with the path specified with the path field. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Uploads messages to an Amazon S3 bucket as objects, using the path specified in the `path` field. #### Common ```yml outputs: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream content_encoding: "" cache_control: "" content_disposition: "" content_language: "" website_redirect_location: "" metadata: exclude_prefixes: [] storage_class: STANDARD kms_key_id: "" checksum_algorithm: "" server_side_encryption: "" force_path_style_urls: false max_in_flight: 64 timeout: 5s object_canned_acl: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` To use a different path for each object, use [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which is evaluated for each message in a batch. ## [](#metadata)Metadata Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/metadata/). ## [](#tags)Tags The `tags` field accepts key/value pairs to attach to objects as tags, and the values support [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries): ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#batching)Batching It’s common to want to upload messages to S3 as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) or [`compress`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/compress/) processor. For example, the following configuration uploads messages as a `.tar.gz` archive of documents: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` This configuration uploads JSON documents as a single large document containing an array of objects: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#bucket-name-format)Bucket name format The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml output: aws_s3: bucket: r2-bucket path: ${!uuid_v4()}.json endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#batching-2)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The bucket to upload messages to. **Type**: `string` ### [](#cache_control)`cache_control` The cache control to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#checksum_algorithm)`checksum_algorithm` The algorithm used to validate each object during its upload to the Amazon S3 bucket. **Type**: `string` **Default**: `""` **Options**: `CRC32`, `CRC32C`, `SHA1`, `SHA256` ### [](#content_disposition)`content_disposition` The content disposition to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` An optional content encoding to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_language)`content_language` The content language to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The content type to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#kms_key_id)`kms_key_id` An optional server-side encryption key. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata-2)`metadata` Specify criteria for which metadata values are attached to objects as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#object_canned_acl)`object_canned_acl` The object canned ACL value. Leave empty to omit the ACL from upload requests, which is required for buckets that have ACLs disabled (the AWS default since 2023). **Type**: `string` **Default**: `""` **Options**: `` `, `private ``, `public-read`, `public-read-write`, `authenticated-read`, `aws-exec-read`, `bucket-owner-read`, `bucket-owner-full-control` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#server_side_encryption)`server_side_encryption` An optional server-side encryption algorithm. **Type**: `string` **Default**: `""` ### [](#storage_class)`storage_class` The storage class to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `STANDARD` **Options**: `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `DEEP_ARCHIVE` ### [](#tags-2)`tags` Key/value pairs to store with the object as tags. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#website_redirect_location)`website_redirect_location` The website redirect location to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 314: aws_sns **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_sns.md --- # aws_sns > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_sns latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_sns page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_sns.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_sns.adoc description: Sends messages to an AWS SNS topic. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to an AWS SNS topic. #### Common ```yml outputs: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] ``` #### Advanced ```yml outputs: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] timeout: 5s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_group_id)`message_group_id` An optional group ID to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#subject)`subject` An optional subject to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#topic_arn)`topic_arn` The topic to publish to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 315: aws_sqs **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/aws_sqs.md --- # aws_sqs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_sqs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/aws_sqs page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/aws_sqs.adoc description: Sends messages to an SQS queue. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to an SQS queue. #### Common ```yml outputs: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_records_per_request: 10 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` Metadata values are sent along with the payload as attributes with the data type String. If the number of metadata values in a message exceeds the message attribute limit (10) then the top ten keys ordered alphabetically will be selected. The fields `message_group_id`, `message_deduplication_id` and `delay_seconds` can be set dynamically using [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which are resolved individually for each message of a batch. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delay_seconds)`delay_seconds` An optional delay time in seconds for message. Value between 0 and 900 This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_records_per_request)`max_records_per_request` The maximum number of records delivered in a single SQS request. Enter only values from `0` to `10`. **Type**: `int` **Default**: `10` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_group_id)`message_group_id` An optional group ID to set for messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The URL of the target SQS queue. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 316: azure_blob_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/azure_blob_storage.md --- # azure_blob_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_blob_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/azure_blob_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/azure_blob_storage.adoc description: Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the container field. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` field. #### Common ```yml outputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt blob_type: BLOCK public_access_level: PRIVATE max_in_flight: 64 ``` In order to have a different path for each object you should use function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#blob_type)`blob_type` Block and Append blobs are comprized of blocks, and each blob can support up to 50,000 blocks. The default value is ``"`BLOCK`"``.\` This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `BLOCK` **Options**: `BLOCK`, `APPEND` ### [](#container)`container` The container for uploading the messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: container: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#public_access_level)`public_access_level` The container’s public access level. The default value is `PRIVATE`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `PRIVATE` **Options**: `PRIVATE`, `BLOB`, `CONTAINER` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` --- # Page 317: azure_cosmosdb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/azure_cosmosdb.md --- # azure_cosmosdb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_cosmosdb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/azure_cosmosdb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/azure_cosmosdb.adoc description: Creates or updates messages as JSON documents in Azure CosmosDB. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). ### Common ```yml outputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` ### Advanced ```yml outputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#examples)Examples ### [](#create-documents)Create documents Create new documents in the `blobfish` container with partition key `/habitat`. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = json("habitat") operation: Create ``` ### [](#patch-documents)Patch documents Execute the Patch operation on documents from the `blobfish` container. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! json("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `array` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 318: azure_data_lake_gen2 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/azure_data_lake_gen2.md --- # azure_data_lake_gen2 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_data_lake_gen2 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/azure_data_lake_gen2 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/azure_data_lake_gen2.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/azure_data_lake_gen2.adoc description: Sends message parts as files to an Azure Data Lake Gen2 filesystem. Each file is uploaded with the filename specified with the path field. page-git-created-date: "2024-11-05" page-git-modified-date: "2026-05-26" --- Sends message parts as files to an [Azure Data Lake Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) file system. Each file is uploaded with the file name specified in the `path` field. ```yml outputs: label: "" azure_data_lake_gen2: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" filesystem: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` To specify a different [`path` value](#path) (file name) for each file, use [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). Function interpolations are calculated for each message in a batch. ## [](#authentication-methods)Authentication methods This output supports multiple authentication methods. You must configure at least one method from the following list: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access using [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If you configure multiple authentication methods, the `storage_connection_string` takes precedence. ## [](#performance)Performance Sends multiple messages in flight in parallel for improved performance. You can tune the number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#filesystem)`filesystem` The name of the data lake storage file system you want to upload messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: filesystem: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `64` ### [](#path)`path` The path (file name) of each message to upload to the data lake storage file system. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. You must enter a value for this field if no other authentication method is specified. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` --- # Page 319: azure_queue_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/azure_queue_storage.md --- # azure_queue_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_queue_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/azure_queue_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/azure_queue_storage.adoc description: Sends messages to an Azure Storage Queue. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to an Azure Storage Queue. #### Common ```yml outputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) ttl: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `queue_name` you can use function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#queue_name)`queue_name` The name of the target Queue Storage queue. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#ttl)`ttl` The TTL of each individual message as a duration string. Defaults to 0, meaning no retention period is set This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 320: azure_table_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/azure_table_storage.md --- # azure_table_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_table_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/azure_table_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/azure_table_storage.adoc description: Stores messages in an Azure Table Storage table. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores messages in an Azure Table Storage table. #### Common ```yml outputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} transaction_type: INSERT max_in_flight: 64 timeout: 5s batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `table_name`, `partition_key` and `row_key` you can use function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. If the `properties` are not set in the config, all the `json` fields are marshalled and stored in the table, which will be created if it does not exist. The `object` and `array` fields are marshaled as strings. e.g.: The JSON message: ```json { "foo": 55, "bar": { "baz": "a", "bez": "b" }, "diz": ["a", "b"] } ``` Will store in the table the following properties: ```yml foo: '55' bar: '{ "baz": "a", "bez": "b" }' diz: '["a", "b"]' ``` It’s also possible to use function interpolations to get or transform the properties values, e.g.: ```yml properties: device: '${! json("device") }' timestamp: '${! json("timestamp") }' ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#partition_key)`partition_key` The partition key. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: partition_key: ${! json("date") } ``` ### [](#properties)`properties` A map of properties to store into the table. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ### [](#row_key)`row_key` The row key. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: row_key: ${! json("device")}-${!uuid_v4() } ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to store messages into. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table_name: ${! meta("kafka_topic") } # --- table_name: ${! json("table") } ``` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#transaction_type)`transaction_type` Type of transaction operation. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `INSERT` **Options**: `INSERT`, `INSERT_MERGE`, `INSERT_REPLACE`, `UPDATE_MERGE`, `UPDATE_REPLACE`, `DELETE` ```yaml # Examples: transaction_type: ${! json("operation") } # --- transaction_type: ${! meta("operation") } # --- transaction_type: INSERT ``` --- # Page 321: broker **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/broker.md --- # broker > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: broker latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/broker page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/broker.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/broker.adoc description: Allows you to route messages to multiple child outputs using a range of brokering patterns. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A meta-output that routes messages to child outputs using a range of brokering [patterns](#patterns). Unlike regular outputs, `broker` doesn’t send messages anywhere by itself. Instead, it wraps other outputs and controls how messages are delivered across them. Use `broker` to fan out the same message to multiple destinations (for example, publishing events to Kafka while also writing them to a database), or to distribute messages across a pool of outputs for load balancing or throughput scaling. The delivery pattern determines whether each message is written to all outputs or routed to a single output, and whether writes happen in parallel or in sequence. > 📝 **NOTE** > > The name `broker` refers to the brokering delivery pattern, not a Redpanda broker (cluster node). #### Common ```yml outputs: label: "" broker: pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" broker: copies: 1 pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` [Processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) can be listed to apply across individual outputs or all outputs: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar # Processors only applied to messages sent to bar. processors: - resource: bar_processor # Processors applied to messages sent to all brokered outputs. processors: - resource: general_processor ``` ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` The number of copies of each configured output to spawn. **Type**: `int` **Default**: `1` ### [](#outputs)`outputs[]` A list of child outputs to broker. **Type**: `array` ### [](#pattern)`pattern` The brokering pattern to use. **Type**: `string` **Default**: `fan_out` **Options**: `fan_out`, `fan_out_fail_fast`, `fan_out_sequential`, `fan_out_sequential_fail_fast`, `round_robin`, `greedy` ## [](#patterns)Patterns The broker pattern determines how messages are distributed across outputs. Use `fan_out` (the default) when every output should receive every message. Use `round_robin` or `greedy` when you want to distribute messages across outputs for load balancing rather than duplication. The available patterns are: ### [](#fan_out)`fan_out` With the fan out pattern all outputs will be sent every message that passes through Redpanda Connect in parallel. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. Sometimes it is useful to disable the back pressure or retries of certain fan out outputs and instead drop messages that have failed or were blocked. In this case you can wrap outputs with a [`drop_on` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/drop_on/). ### [](#fan_out_fail_fast)`fan_out_fail_fast` The same as the `fan_out` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#fan_out_sequential)`fan_out_sequential` Similar to the fan out pattern except outputs are written to sequentially, meaning an output is only written to once the preceding output has confirmed receipt of the same message. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. ### [](#fan_out_sequential_fail_fast)`fan_out_sequential_fail_fast` The same as the `fan_out_sequential` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#round_robin)`round_robin` With the round robin pattern each message will be assigned a single output following their order. If an output applies back pressure it will block all subsequent messages. If an output fails to send a message then the message will be re-attempted with the next input, and so on. ### [](#greedy)`greedy` The greedy pattern results in higher output throughput at the cost of potentially disproportionate message allocations to those outputs. Each message is sent to a single output, which is determined by allowing outputs to claim messages as soon as they are able to process them. This results in certain faster outputs potentially processing more messages at the cost of slower outputs. --- # Page 322: cache **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/cache.md --- # cache > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cache latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/cache page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/cache.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/cache.adoc description: Stores each message in a cache. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Stores each message in a [cache](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/). #### Common ```yml outputs: label: "" cache: target: "" # No default (required) key: ${!count("items")}-${!timestamp_unix_nano()} max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" cache: target: "" # No default (required) key: ${!count("items")}-${!timestamp_unix_nano()} ttl: "" # No default (optional) max_in_flight: 64 ``` Caches are configured as [resources](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/), where there’s a wide variety to choose from. The `target` field must reference a configured cache resource label like follows: ```yaml output: cache: target: foo key: ${!json("document.id")} cache_resources: - label: foo memcached: addresses: - localhost:11211 default_ttl: 60s ``` In order to create a unique `key` value per item you should use function interpolations described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#key)`key` The key to store messages by, function interpolation should be used in order to derive a unique key for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!count("items")}-${!timestamp_unix_nano()}` ```yaml # Examples: key: ${!count("items")}-${!timestamp_unix_nano()} # --- key: ${!json("doc.id")} # --- key: ${!meta("kafka_key")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#target)`target` The target cache to store messages in. **Type**: `string` ### [](#ttl)`ttl` The TTL of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 323: cyborgdb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/cyborgdb.md --- # cyborgdb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cyborgdb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/cyborgdb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/cyborgdb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/cyborgdb.adoc description: Inserts items into a CyborgDB encrypted vector index. page-git-created-date: "2025-10-09" page-git-modified-date: "2026-08-11" --- Inserts items into a CyborgDB encrypted vector index. #### Common ```yaml outputs: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` #### Advanced ```yaml outputs: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) create_if_missing: false operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure. ## [](#fields)Fields ### [](#api_key)`api_key` The API key for authenticating with the CyborgDB service. This key identifies your account and provides access to your CyborgDB indexes. Keep this key secure and avoid exposing it in logs or version control. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_if_missing)`create_if_missing` Whether to create the index if it doesn’t exist. When enabled, CyborgDB automatically detects the vector dimensions from your data and optimizes the index configuration for performance. This is useful for development and testing environments. **Type**: `bool` **Default**: `false` ### [](#host)`host` The host URL for the CyborgDB instance. This should include the protocol (https://) and port number if required. For example: `[https://api.cyborgdb.com](https://api.cyborgdb.com)` or `[https://localhost:8080](https://localhost:8080)`. **Type**: `string` ```yaml # Examples: host: api.cyborg.com # --- host: localhost:8000 ``` ### [](#id)`id` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that determines the unique identifier for each vector entry. This ID is used to update existing vectors during upsert operations or to specify which vectors to delete. If not provided, CyborgDB will generate unique IDs automatically. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#index_key)`index_key` The base64-encoded encryption key for the CyborgDB index. This key must be exactly 32 bytes when decoded from base64. All vector data is encrypted client-side using this key before transmission, ensuring complete data privacy. Store this key securely as it cannot be recovered if lost. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: index_key: your-base64-encoded-32-byte-key ``` ### [](#index_name)`index_name` The name of the CyborgDB index to write vectors to. If the index doesn’t exist and `create_if_missing` is enabled, CyborgDB will create it automatically with optimized settings based on your data. **Type**: `string` **Default**: `redpanda-vectors` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that extracts metadata to associate with the vector entry. The metadata can contain any JSON-serializable data that helps identify or categorize the vector. This data is stored encrypted alongside the vector. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: root = {"summary": this.summary, "category": this.category} ``` ### [](#operation)`operation` The operation to perform against the CyborgDB index. Supported operations: - `upsert`: Insert new vectors or update existing ones (requires `vector_mapping`) - `delete`: Remove vectors from the index (requires `id`) - `query`: Search for similar vectors (requires `vector_mapping`) **Type**: `string` **Default**: `upsert` **Options**: `upsert`, `delete` ### [](#vector_mapping)`vector_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that extracts the vector from the message. The result must be an array of floating-point numbers representing the vector embeddings. This field is required for `upsert` and `query` operations. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 324: drop_on **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/drop_on.md --- # drop_on > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: drop_on latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/drop_on page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/drop_on.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/drop_on.adoc description: Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead of being reattempted (or nacked). page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead of being reattempted (or nacked). ```yml outputs: label: "" drop_on: error: false error_patterns: [] # No default (optional) back_pressure: "" # No default (optional) output: "" # No default (required) ``` Regular Redpanda Connect outputs will apply back pressure when downstream services aren’t accessible, and Redpanda Connect retries (or nacks) all messages that fail to be delivered. However, in some circumstances, or for certain output types, we instead might want to relax these mechanisms, which is when this output becomes useful. ## [](#fields)Fields ### [](#back_pressure)`back_pressure` An optional duration string that determines the maximum length of time to wait for a given message to be accepted by the child output before the message should be dropped instead. The most common reason for an output to block is when waiting for a lost connection to be re-established. Once a message has been dropped due to back pressure all subsequent messages are dropped immediately until the output is ready to process them again. Note that if `error` is set to `false` and this field is specified then messages dropped due to back pressure will return an error response (are nacked or reattempted). **Type**: `string` ```yaml # Examples: back_pressure: 30s # --- back_pressure: 1m ``` ### [](#error)`error` Whether messages should be dropped when the child output returns an error of any type. For example, this could be when an `http_client` output gets a 4XX response code. In order to instead drop only on specific error patterns use the `error_matches` field instead. **Type**: `bool` **Default**: `false` ### [](#error_patterns)`error_patterns[]` A list of regular expressions (re2) where if the child output returns an error that matches any part of any of these patterns the message will be dropped. **Type**: `array` ```yaml # Examples: error_patterns: - "and that was really bad$" # --- error_patterns: - "roughly [0-9]+ issues occurred" ``` ### [](#output)`output` A child output to wrap with this drop mechanism. **Type**: `output` nclude::connect:components:partial$examples/outputs/drop\_on.adoc\[\] --- # Page 325: drop **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/drop.md --- # drop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: drop latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/drop page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/drop.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/drop.adoc description: Drop output reference for silently discarding messages in Redpanda Connect pipelines. page-topic-type: reference personas: streaming_developer, app_developer learning-objective-1: Look up drop output syntax and configuration learning-objective-2: Find examples of drop in conditional routing learning-objective-3: Identify use cases for drop output in pipelines page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Silently discards all messages without error or side effects. The `drop` output is a utility component that drops messages from the [pipeline](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#pipeline). Unlike filtering or conditional processors that modify or route messages, `drop` consumes messages and does nothing with them. This is useful for: - **Testing and debugging**: Measure input throughput without output bottlenecks - **Conditional workflows**: Discard messages that don’t meet certain criteria - **Dead letter queue patterns**: Provide a final fallback when all other outputs fail - **Development**: Temporarily disable output while testing pipeline logic Use this reference to: - Look up drop output syntax and configuration - Find examples of drop in conditional routing - Identify use cases for drop output in pipelines ```yaml outputs: label: "" drop: {} ``` ## [](#performance)Performance The `drop` output has minimal overhead and immediately acknowledges messages. This makes it ideal for performance testing, as it removes output processing time from measurements. ## [](#examples)Examples ### [](#conditional-filtering)Conditional filtering Use `drop` with the [`switch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/switch/) output to conditionally discard messages: ```yaml output: switch: cases: - check: this.type == "error" output: label: error_sink kafka: addresses: ["kafka:9092"] topic: errors - check: this.type == "debug" output: label: drop_debug drop: {} # Don't process debug messages in production - output: label: main_sink kafka: addresses: ["kafka:9092"] topic: events ``` ### [](#dead-letter-queue-pattern)Dead letter queue pattern Use `drop` as a last resort in a [`fallback`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) chain: ```yaml output: fallback: - kafka: addresses: ["kafka:9092"] topic: primary_topic max_in_flight: 1 - kafka: addresses: ["kafka:9092"] topic: dlq_topic - drop: {} # Last resort: drop if both outputs fail ``` ### [](#testing-input-throughput)Testing input throughput Measure how fast your input can consume data without output bottlenecks: ```yaml input: kafka: addresses: ["kafka:9092"] topics: ["test"] output: drop: {} # Measure input consumption speed without output overhead ``` For more patterns on message routing, filtering, and when to use `drop` vs. other approaches, see the [Message Routing Patterns](https://docs.redpanda.com/connect/cookbooks/message_routing/) cookbook. --- # Page 326: elasticsearch_v8 **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/elasticsearch_v8.md --- # elasticsearch_v8 > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: elasticsearch_v8 latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/elasticsearch_v8 page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/elasticsearch_v8.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/elasticsearch_v8.adoc page-git-created-date: "2025-03-12" page-git-modified-date: "2026-05-26" --- Publishes messages into an [Elasticsearch index](https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html). If the index does not exist, this output creates it using dynamic mapping. > 📝 **NOTE** > > The `elasticsearch_v8` output is based on the the [go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch?tab=readme-ov-file) library. For full information about breaking changes from previous versions, see [Elastic’s Migrating to 8.0 guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking_80_rest_api_changes). To help configure your own `elasticsearch_v8` output, this page includes [example pipeline configurations](#example-pipelines). ### Common ```yml outputs: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 api_key: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ### Advanced ```yml outputs: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" retry_on_conflict: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 api_key: "" basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#set-values-dynamically)Set values dynamically You can use [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) to dynamically set values for the [`id`](#id) and [`index`](#index) fields, as well as other fields where [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) are supported. When message batches are sent, interpolations are performed per message. ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#action)`action` The action to perform on each document. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). For more information on how the `update` action works, see [Example pipelines](#example-pipelines). **Type**: `string` ### [](#api_key)`api_key` An API key to authenticate with. If set, it supersedes basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to Elasticsearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` Define the ID for indexed messages. Use [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The Elasticsearch index where messages are published. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` Specify the ID of a pipeline to preprocess incoming documents before they are published (optional). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#retry_on_conflict)`retry_on_conflict` The number of times to retry an update operation when a version conflict occurs. **Type**: `int` **Default**: `0` ### [](#routing)`routing` The routing key to use for the document. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. This output attempts to connect to each URL in the list, in order, until a successful connection is established. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` ## [](#example-pipelines)Example pipelines ### Update documents To update documents in the target index, the top level of the request body must include at least one of the following fields: - `doc`: Performs partial updates on a document. - `upsert`: Updates an existing document or inserts a document if it doesn’t exist. - `script`: Performs an update using a scripting language, such as [Elasticsearch’s Painless scripting language](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html). The following examples show how to configure mapping processors with this output to achieve different types of updates. Example 1: Partial document update ```yaml output: processors: # Sets the metadata ID field to the message ID then # performs a partial update on the document. - mapping: | meta id = this.id root.doc = this elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 2: Scripted update ```yaml output: processors: # Sets the metadata ID field to the message ID then # increments the counter field by `1` using a script. - mapping: | meta id = this.id root.script.source = "ctx._source.counter += 1" elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 3: Upsert ```yaml output: processors: # Sets the metadata ID field to the message ID. # If the product with the specified ID exists, update its product_price to 100. # If the document does not exist, insert a new document with the ID set to 1 # and the `product_price` set to 50. - mapping: | meta id = this.id root.doc.product_price = 100 root.upsert.product_price = 50 elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` For more information on the structures and behaviors of `doc`, `upsert`, and `script` fields, see the [Elasticsearch Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html). ### Index documents from Redpanda Reads messages from a Redpanda cluster and writes them to an Elasticsearch index using a field from the message as the document ID. ```yaml # Reads messages from a Redpanda cluster. input: redpanda: seed_brokers: [localhost:19092] # The address of the Redpanda broker. topics: ["product_code"] # The topic to consume messages from. consumer_group: "rpcn3" # The consumer group ID. processors: # Sets the metadata ID field to the message ID and # sets the root of the message to the message content. - mapping: | meta id = this.id root = this # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "product_code" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("id") } # Sets the document ID to the value of the metadata ID field. ``` ### Index documents from AWS S3 Reads messages from a AWS S3 bucket and writes them to an Elasticsearch index using the S3 key as the ID for the Elasticsearch document. ```yaml # Reads messages from an AWS S3 bucket. input: aws_s3: bucket: "my_bucket" # The name of the S3 bucket. prefix: "prod_inventory/" # A prefix to filter objects in the bucket. scanner: to_the_end: {} # Scans the bucket to the end. # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "current_prod_inventory" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("s3_key") } # Sets the document ID to the S3 key. ``` --- # Page 327: fallback **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback.md --- # fallback > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: fallback latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/fallback page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/fallback.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/fallback.adoc description: Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the list is attempted, and so on. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the list is attempted, and so on. ```yml outputs: label: "" fallback: - label: "" stdout: codec: lines - label: "" file: path: /tmp/fallback.txt codec: lines ``` This pattern is useful for triggering events in the case where certain output targets have broken. For example, if you had an output type `http_client` but wished to reroute messages whenever the endpoint becomes unreachable you could use this pattern: ```yaml output: fallback: - http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s processors: - mapping: 'root = "failed to send this message to foo: " + content()' - file: path: /usr/local/benthos/everything_failed.jsonl ``` ## [](#metadata)Metadata When a given output fails the message routed to the following output will have a metadata value named `fallback_error` containing a string error message outlining the cause of the failure. The content of this string will depend on the particular output and can be used to enrich the message or provide information used to broker the data to an appropriate output using something like a `switch` output. ## [](#batching)Batching When an output within a fallback sequence uses batching, like so: ```yaml output: fallback: - aws_dynamodb: table: foo string_columns: id: ${!json("id")} content: ${!content()} batching: count: 10 period: 1s - file: path: /usr/local/benthos/failed_stuff.jsonl ``` Redpanda Connect makes a best attempt at inferring which specific messages of the batch failed, and only propagates those individual messages to the next fallback tier. However, depending on the output and the error returned it is sometimes not possible to determine the individual messages that failed, in which case the whole batch is passed to the next tier in order to preserve at-least-once delivery guarantees. --- # Page 328: gcp_bigquery_write_api **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_bigquery_write_api.md --- # gcp_bigquery_write_api > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_bigquery_write_api latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/gcp_bigquery_write_api page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/gcp_bigquery_write_api.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/gcp_bigquery_write_api.adoc description: Streams data into BigQuery using the Storage Write API. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-11" --- Streams data into BigQuery using the Storage Write API. Writes messages to a BigQuery table using the Storage Write API. This provides higher throughput and lower latency than the legacy streaming API or load jobs. Messages can be formatted as JSON (default) or raw protobuf bytes. When using JSON format the component automatically fetches the table schema and converts each message to the corresponding proto representation. > ⚠️ **WARNING** > > The proto3 JSON mapping encodes int64 and uint64 values as strings. JSON messages with integer fields must use string values (e.g. `"age": "30"` not `"age": 30`). Otherwise the write will fail with an unmarshalling error. When batching is enabled the table name is resolved from the first message in each batch. All messages in the same batch are written to that table. #### Common ```yml outputs: label: "" gcp_bigquery_write_api: project: "" dataset: "" # No default (required) table: "" # No default (required) message_format: json change_type: "" # No default (optional) change_sequence_number: "" # No default (optional) primary_keys: [] # No default (optional) max_in_flight: 4 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) credentials_json: "" ``` #### Advanced ```yml outputs: label: "" gcp_bigquery_write_api: project: "" dataset: "" # No default (required) table: "" # No default (required) message_format: json write_mode: default_stream change_type: "" # No default (optional) change_sequence_number: "" # No default (optional) primary_keys: [] # No default (optional) auto_create_table: false schema: [] time_partitioning: type: "" # No default (optional) field: "" expiration: 0s require_filter: false clustering: [] max_in_flight: 4 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) credentials_json: "" target_principal: "" delegates: [] stream_idle_timeout: 5m stream_sweep_interval: 1m max_cached_streams: 1024 schema_resolve_timeout: 15s schema_evolution_timeout: 30s endpoint: http: "" grpc: "" ``` ## [](#fields)Fields ### [](#auto_create_table)`auto_create_table` If true and the target table does not exist, the output creates it using the configured `schema`, `time_partitioning`, and `clustering`. AlreadyExists errors from concurrent creators are treated as success. When the table name is interpolated, every auto-created table receives the same schema and partition/clustering settings. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#change_sequence_number)`change_sequence_number` Optional Bloblang expression resolving to the `_CHANGE_SEQUENCE_NUMBER` pseudo-column value. Format: 1 to 4 sections of 1 to 16 hexadecimal characters each, separated by `/`. Example: `${! metadata("scn") }` or `${! "0/0/0/0" }`. When unset, BigQuery resolves ordering by arrival time. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#change_type)`change_type` Bloblang expression resolving to the `_CHANGE_TYPE` pseudo-column value for each row. Must resolve to `UPSERT` or `DELETE` (case-insensitive). Required when `write_mode` is `upsert` or `upsert_delete`. Example: `${! metadata("operation") }`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#clustering)`clustering[]` Optional clustering columns (up to 4) applied during `auto_create_table`. All names must appear in `schema`. **Type**: `array` **Default**: `[]` ### [](#credentials_json)`credentials_json` An optional JSON string containing GCP credentials. If empty, credentials are loaded from the environment. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#dataset)`dataset` The BigQuery dataset ID. **Type**: `string` ### [](#delegates)`delegates[]` Optional delegation chain for chained service account impersonation. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next in the chain. **Type**: `array` **Default**: `[]` ### [](#endpoint)`endpoint` Optional endpoint overrides for the BigQuery and Storage Write API clients. **Type**: `object` ### [](#endpoint-grpc)`endpoint.grpc` Override the BigQuery Storage gRPC endpoint. Useful for local emulators. **Type**: `string` **Default**: `""` ### [](#endpoint-http)`endpoint.http` Override the BigQuery HTTP endpoint. Useful for local emulators. **Type**: `string` **Default**: `""` ### [](#max_cached_streams)`max_cached_streams` Soft cap on the number of cached streams. When the cache exceeds this size, the least-recently-used stream is evicted. Set to 0 for unlimited (rely on idle-timeout sweeping only). Relevant when the table field uses interpolation to route to many tables. **Type**: `int` **Default**: `1024` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `4` ### [](#message_format)`message_format` The format of input messages. Use 'json' to have the component convert JSON to proto automatically. Use 'protobuf' to supply raw proto-encoded bytes. **Type**: `string` **Default**: `json` **Options**: `json`, `protobuf` ### [](#primary_keys)`primary_keys[]` Optional list of primary-key column names. Required when `auto_create_table` is true and `write_mode` is `upsert` or `upsert_delete`. A pre-existing table must already declare its PRIMARY KEY — this field cannot add one; when both are set they must match exactly (same columns, same order). Up to 16 columns; composite keys are supported in the same order they are listed. **Type**: `array` ### [](#project)`project` The GCP project ID. If empty, the project is auto-detected from the environment. **Type**: `string` **Default**: `""` ### [](#schema)`schema[]` Column definitions used by `auto_create_table`. Required when `auto_create_table` is true. **Type**: `array` **Default**: `[]` ### [](#schema-fields)`schema[].fields[]` For RECORD columns, the list of nested fields. Same shape as the top-level schema list. **Type**: `array` ### [](#schema-mode)`schema[].mode` Column mode: NULLABLE (default), REQUIRED, or REPEATED. **Type**: `string` **Default**: `NULLABLE` ### [](#schema-name)`schema[].name` Column name. **Type**: `string` ### [](#schema-type)`schema[].type` BigQuery column type (STRING, BYTES, INTEGER/INT64, FLOAT/FLOAT64, NUMERIC, BIGNUMERIC, BOOLEAN/BOOL, TIMESTAMP, DATE, TIME, DATETIME, GEOGRAPHY, JSON, RECORD). **Type**: `string` ### [](#schema_evolution_timeout)`schema_evolution_timeout` Total time budget for a single schema evolution attempt (Metadata + Update across all CAS retries on HTTP 412). Bounds how long the WriteBatch retry loop can be starved by a wedged backend. **Type**: `string` **Default**: `30s` ### [](#schema_resolve_timeout)`schema_resolve_timeout` How long a single BigQuery table-metadata fetch can run before being aborted. Coalesced concurrent resolves share one fetch, so this bounds the time a wedged backend can stall every batch routing to the same table. On the auto\_create\_table path the budget covers Metadata→Create→Metadata, so it needs to absorb transient backend slowness on top of the metadata fetch itself. **Type**: `string` **Default**: `15s` ### [](#stream_idle_timeout)`stream_idle_timeout` How long a cached stream can remain unused before being closed. Relevant when the table field uses interpolation to route to many tables. **Type**: `string` **Default**: `5m` ### [](#stream_sweep_interval)`stream_sweep_interval` How often to check for idle streams to close. **Type**: `string` **Default**: `1m` ### [](#table)`table` The BigQuery table ID. Supports interpolation functions. When batching, resolved from the first message in each batch. **Type**: `string` ### [](#target_principal)`target_principal` Service account email to impersonate. When set, the output obtains tokens acting as this service account. Requires the caller to have roles/iam.serviceAccountTokenCreator on the target. **Type**: `string` **Default**: `""` ### [](#time_partitioning)`time_partitioning` Optional time-partitioning settings applied during `auto_create_table`. Setting `type` is the trigger — when omitted, the block is treated as absent. **Type**: `object` ### [](#time_partitioning-expiration)`time_partitioning.expiration` Optional partition expiration. Zero means no expiration. **Type**: `string` **Default**: `0s` ### [](#time_partitioning-field)`time_partitioning.field` Column to partition on. Must be of type DATE, TIMESTAMP, or DATETIME. If empty, the table uses ingestion-time partitioning (`_PARTITIONTIME`). **Type**: `string` **Default**: `""` ### [](#time_partitioning-require_filter)`time_partitioning.require_filter` If true, queries against the table must filter on the partition column. **Type**: `bool` **Default**: `false` ### [](#time_partitioning-type)`time_partitioning.type` Partitioning granularity. **Type**: `string` **Options**: `DAY`, `HOUR`, `MONTH`, `YEAR` ### [](#write_mode)`write_mode` How the output writes to BigQuery. `default_stream` uses the multiplexed default stream (at-least-once, lowest latency). `pending_stream` allocates a per-batch pending stream that commits atomically, providing exactly-once semantics within a single committed batch. `upsert` writes UPSERT-only rows to a BigQuery CDC-enabled table; the target table must have a PRIMARY KEY. `upsert_delete` allows both UPSERT and DELETE rows. Both CDC modes use the default stream as required by BigQuery. **Type**: `string` **Default**: `default_stream` **Options**: `default_stream`, `pending_stream`, `upsert`, `upsert_delete` --- # Page 329: gcp_bigquery **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_bigquery.md --- # gcp_bigquery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_bigquery latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/gcp_bigquery page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/gcp_bigquery.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/gcp_bigquery.adoc description: Sends messages as new rows to a Google Cloud BigQuery table. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Inserts message data as new rows in a Google Cloud BigQuery table. #### Common ```yml outputs: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 job_labels: {} credentials_json: "" csv: header: [] field_delimiter: , allow_jagged_rows: false allow_quoted_newlines: false encoding: UTF-8 skip_leading_rows: 1 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 write_disposition: WRITE_APPEND create_disposition: CREATE_IF_NEEDED ignore_unknown_values: false max_bad_records: 0 auto_detect: false job_labels: {} credentials_json: "" csv: header: [] field_delimiter: , allow_jagged_rows: false allow_quoted_newlines: false encoding: UTF-8 skip_leading_rows: 1 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a [shared credentials file](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/gcp/) when connecting to GCP services. ## [](#format)Format The `gcp_bigquery` output currently supports only `NEWLINE_DELIMITED_JSON`, `CSV` and `PARQUET` formats. To learn more about how to use BigQuery with these formats, see the following documentation: - [`NEWLINE_DELIMITED_JSON`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json) - [`CSV`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv) - [`PARQUET`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet) ### [](#newline-delimited-json)Newline-delimited JSON Each JSON message may contain multiple elements separated by newlines. For example, a single message containing: ```json {"key": "1"} {"key": "2"} ``` Is equivalent to two separate messages: ```json {"key": "1"} ``` And: ```json {"key": "2"} ``` The same is true for the CSV format. ### [](#csv)CSV When the field `csv.header` is specified for the `CSV` format, a header row is inserted as the first line of each message batch. If this field is not provided, then the first message of each message batch must include a header line. ### [](#parquet)Parquet Each message sent to this output must be a Parquet file. You can use the [`parquet_encode` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/parquet_encode/) to convert message data into the correct format. For example: ```yaml input: generate: mapping: | root = { "foo": random_int(), "bar": uuid_v4(), "time": now(), } interval: 0 count: 1000 batch_size: 1000 pipeline: processors: - parquet_encode: schema: - name: foo type: INT64 - name: bar type: UTF8 - name: time type: UTF8 default_compression: zstd output: gcp_bigquery: project: "${PROJECT}" dataset: "my_bq_dataset" table: "redpanda_connect_ingest" format: PARQUET ``` ## [](#performance)Performance The `gcp_bigquery` output benefits from sending multiple messages in parallel for improved performance. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. This output also sends messages as a batch for improved performance. Redpanda Connect can form batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#auto_detect)`auto_detect` Whether this component automatically infers the options and schema for `CSV` and `NEWLINE_DELIMITED_JSON` sources. If this value is set to `false` and the destination table doesn’t exist, the output throws an insertion error as it is unable to insert data. > ⚠️ **CAUTION** > > This field delegates schema detection to the GCP BigQuery service. For the `CSV` format, values like `no` may be treated as booleans. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_disposition)`create_disposition` Specifies the circumstances under which a destination table is created. - Use `CREATE_IF_NEEDED` to create the destination table if it does not already exist. Tables are created atomically on successful completion of a job. - Use `CREATE_NEVER` if the destination table must already exist. **Type**: `string` **Default**: `CREATE_IF_NEEDED` **Options**: `CREATE_IF_NEEDED`, `CREATE_NEVER` ### [](#credentials_json)`credentials_json` Sets the [Google Service Account Credentials JSON](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account) (optional). > ⚠️ **WARNING** > > When using [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) to populate this field, wrap the function in single quotes, not double quotes. For example, use `'${secrets.GCP_CREDENTIALS_JSON}'` instead of `"${secrets.GCP_CREDENTIALS_JSON}"`. Double quotes cause JSON parsing errors because the credentials already contain JSON content. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#csv-2)`csv` Specify how CSV data is interpreted. **Type**: `object` ### [](#csv-allow_jagged_rows)`csv.allow_jagged_rows` Set to `true` to treat optional missing trailing columns as nulls in CSV data. **Type**: `bool` **Default**: `false` ### [](#csv-allow_quoted_newlines)`csv.allow_quoted_newlines` Whether quoted data sections containing new lines are allowed when reading CSV data. **Type**: `bool` **Default**: `false` ### [](#csv-encoding)`csv.encoding` The character encoding of CSV data. **Type**: `string` **Default**: `UTF-8` **Options**: `UTF-8`, `ISO-8859-1` ### [](#csv-field_delimiter)`csv.field_delimiter` The separator for fields in a CSV file. The output uses this value when reading or exporting data. **Type**: `string` **Default**: `,` ### [](#csv-header)`csv.header[]` A list of values to use as the header for each batch of messages. If not specified, the first line of each message is used as the header. **Type**: `array` **Default**: `[]` ### [](#csv-skip_leading_rows)`csv.skip_leading_rows` The number of rows at the top of a CSV file that BigQuery will skip when reading data. The default value is `1`, which allows Redpanda Connect to add the specified header in the first line of each batch sent to BigQuery. **Type**: `int` **Default**: `1` ### [](#dataset)`dataset` The BigQuery Dataset ID. **Type**: `string` ### [](#format-2)`format` The format of each incoming message. **Type**: `string` **Default**: `NEWLINE_DELIMITED_JSON` **Options**: `NEWLINE_DELIMITED_JSON`, `CSV`, `PARQUET` ### [](#ignore_unknown_values)`ignore_unknown_values` Set this value to `true` to ignore values that do not match the schema: - For the `CSV` format, extra values at the end of a line are ignored. - For the `NEWLINE_DELIMITED_JSON` format, values that do not match any column name are ignored. By default, this value is set to `false`, and records containing unknown values are treated as bad records. Use the `max_bad_records` field to customize how bad records are handled. **Type**: `bool` **Default**: `false` ### [](#job_labels)`job_labels` A list of labels to add to the load job. **Type**: `object` **Default**: `{}` ### [](#job_project)`job_project` Specify the project ID in which jobs are executed. If not set, the `project` value is used. **Type**: `string` **Default**: `""` ### [](#max_bad_records)`max_bad_records` The maximum number of bad records to ignore when reading data and [`ignore_unknown_values`](#ignore_unknown_values) is set to `true`. **Type**: `int` **Default**: `0` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#project)`project` Specify the project ID of the dataset to insert data into. If not set, the project ID is inferred from the project linked to the service account or read from the `GOOGLE_CLOUD_PROJECT` environment variable. **Type**: `string` **Default**: `""` ### [](#table)`table` The table to insert messages into. **Type**: `string` ### [](#write_disposition)`write_disposition` Specifies how existing data in a destination table is treated. **Type**: `string` **Default**: `WRITE_APPEND` **Options**: `WRITE_APPEND`, `WRITE_EMPTY`, `WRITE_TRUNCATE` --- # Page 330: gcp_cloud_storage **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_cloud_storage.md --- # gcp_cloud_storage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_cloud_storage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/gcp_cloud_storage page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/gcp_cloud_storage.adoc description: Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the path field. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field. #### Common ```yml outputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream collision_mode: overwrite timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream content_encoding: "" collision_mode: overwrite chunk_size: 16777216 timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` In order to have a different path for each object you should use function interpolations described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#metadata)Metadata Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/metadata/). ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in [Google Cloud Platform](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/gcp/). ## [](#batching)Batching It’s common to want to upload messages to Google Cloud Storage as batched archives, the easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) and/or [`compress`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/compress/) processor. For example, if we wished to upload messages as a .tar.gz archive of documents we could achieve that with the following config: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` Alternatively, if we wished to upload JSON documents as a single large document containing an array of objects we can do that with: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching-2)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The bucket to upload messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#chunk_size)`chunk_size` An optional chunk size which controls the maximum number of bytes of the object that the Writer will attempt to send to the server in a single request. If ChunkSize is set to zero, chunking will be disabled. **Type**: `int` **Default**: `16777216` ### [](#collision_mode)`collision_mode` Determines how file path collisions should be dealt with. Options are "overwrite", which replaces the existing file with the new one, "append", which appends the message bytes to the original file, "error-if-exists", which returns an error and rejects the message if the file exists, and "ignore", does not modify the original file and drops the message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `overwrite` **Options**: `overwrite`, `append`, `error-if-exists`, `ignore` ### [](#content_encoding)`content_encoding` An optional content encoding to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The content type to set for each object. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `3s` ```yaml # Examples: timeout: 1s # --- timeout: 500ms ``` --- # Page 331: gcp_pubsub **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_pubsub.md --- # gcp_pubsub > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_pubsub latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/gcp_pubsub page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/gcp_pubsub.adoc description: Sends messages to a GCP Cloud Pub/Sub topic. Metadata from messages are sent as attributes. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends messages to a GCP Cloud Pub/Sub topic. [Metadata](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/metadata/) from messages are sent as attributes. #### Common ```yml outputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" ordering_key: "" # No default (optional) max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 publish_timeout: 1m0s validate_topic: true metadata: exclude_prefixes: [] flow_control: max_outstanding_bytes: -1 max_outstanding_messages: 1000 limit_exceeded_behavior: block batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` For information on how to set up credentials, see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#troubleshooting)Troubleshooting If you’re consistently seeing `Failed to send message to gcp_pubsub: context deadline exceeded` error logs without any further information it is possible that you are encountering [https://github.com/benthosdev/benthos/issues/1042](https://github.com/benthosdev/benthos/issues/1042), which occurs when metadata values contain characters that are not valid utf-8. This can frequently occur when consuming from Kafka as the key metadata field may be populated with an arbitrary binary value, but this issue is not exclusive to Kafka. If you are blocked by this issue then a work around is to delete either the specific problematic keys: ```yaml pipeline: processors: - mapping: | meta kafka_key = deleted() ``` Or delete all keys with: ```yaml pipeline: processors: - mapping: meta = deleted() ``` ## [](#fields)Fields ### [](#batching)`batching` Configures a batching policy on this output. While the PubSub client maintains its own internal buffering mechanism, preparing larger batches of messages can further trade-off some latency for throughput. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#byte_threshold)`byte_threshold` Publish a batch when its size in bytes reaches this value. **Type**: `int` **Default**: `1000000` ### [](#count_threshold)`count_threshold` Publish a pubsub buffer when it has this many messages **Type**: `int` **Default**: `100` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#delay_threshold)`delay_threshold` Publish a non-empty pubsub buffer after this delay has passed. **Type**: `string` **Default**: `10ms` ### [](#endpoint)`endpoint` An optional endpoint to override the default of `pubsub.googleapis.com:443`. This can be used to connect to a region specific pubsub endpoint. For a list of valid values, see [this document](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#flow_control)`flow_control` For a given topic, configures the PubSub client’s internal buffer for messages to be published. **Type**: `object` ### [](#flow_control-limit_exceeded_behavior)`flow_control.limit_exceeded_behavior` Configures the behavior when trying to publish additional messages while the flow controller is full. The available options are block (default), ignore (disable), and signal\_error (publish results will return an error). **Type**: `string` **Default**: `block` **Options**: `ignore`, `block`, `signal_error` ### [](#flow_control-max_outstanding_bytes)`flow_control.max_outstanding_bytes` Maximum size of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `-1` ### [](#flow_control-max_outstanding_messages)`flow_control.max_outstanding_messages` Maximum number of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `1000` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increasing this may improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as attributes, all are sent by default. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#ordering_key)`ordering_key` The ordering key to use for publishing messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#project)`project` The project ID of the topic to publish to. **Type**: `string` ### [](#publish_timeout)`publish_timeout` The maximum length of time to wait before abandoning a publish attempt for a message. **Type**: `string` **Default**: `1m0s` ```yaml # Examples: publish_timeout: 10s # --- publish_timeout: 5m # --- publish_timeout: 60m ``` ### [](#topic)`topic` The topic to publish to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#validate_topic)`validate_topic` Whether to validate the existence of the topic before publishing. If set to false and the topic does not exist, messages will be lost. **Type**: `bool` **Default**: `true` --- # Page 332: http_client **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/http_client.md --- # http_client > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: http_client latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/http_client page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/http_client.adoc page-git-created-date: "2025-03-04" page-git-modified-date: "2026-05-26" --- Sends messages to a HTTP server. #### Common ```yml outputs: label: "" http_client: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml # All configuration fields, showing default values output: label: "" http_client: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" # Optional oauth: enabled: false consumer_key: "" # Optional consumer_secret: "" # Optional access_token: "" # Optional access_token_secret: "" # Optional oauth2: enabled: false client_key: "" # Optional client_secret: "" # Optional token_url: "" # Optional scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" # Optional password: "" # Optional jwt: enabled: false private_key_file: "" # Optional signing_method: "" # Optional claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) batch_as_multipart: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" # Optional check: "" # Optional processors: [] # No default (optional) multipart: [] ``` ## [](#message-sends)Message sends The body of the request sent to the HTTP server is the raw contents of the message payload. If the message has multiple parts (is a batch), the request is sent according to [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). To disable this behavior, set the [`batch_as_multipart`](#batch_as_multipart) field to `false`. When message retries are exhausted, this output rejects a message. Typically, a pipeline then continues attempts to send the message until it succeeds, whilst applying back pressure. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all message in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#follow_redirects)`follow_redirects` Whether or not to transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys to add to the HTTP request as headers (optional). **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#multipart)`multipart[]` EXPERIMENTAL: Create explicit multipart HTTP requests by specifying an array of parts to add to a request. Each part consists of content headers and a data field, which can be populated dynamically. If populated, this field overrides the [default request creation behavior](#message-sends). **Type**: `array` **Default**: `[]` ### [](#multipart-body)`multipart[].body` The body of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: body: ${! this.data.part1 } ``` ### [](#multipart-content_disposition)`multipart[].content_disposition` The content disposition of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_disposition: form-data; name="bin"; filename='${! @AttachmentName } ``` ### [](#multipart-content_type)`multipart[].content_type` The content type of the individual message part. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_type: application/bin ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 333: iceberg **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/iceberg.md --- # iceberg > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: iceberg latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/iceberg page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/iceberg.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/iceberg.adoc description: Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. page-git-created-date: "2026-03-05" page-git-modified-date: "2026-08-11" --- Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. This output is well suited for migrating fanout pipelines from Kafka Connect to Redpanda Connect, and supports: - Multiple storage backends (S3, GCS, Azure) - Automatic table creation with schema detection - Partition transforms (year, month, day, hour, bucket, truncate) - Schema evolution (automatic column addition) - Transaction retry logic for concurrent writes ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ### Common ```yml outputs: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) service: "" # No default (optional) headers: "" # No default (optional) tls_skip_verify: false namespace: "" # No default (required) table: "" # No default (required) storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) force_path_style_urls: false credentials: id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) endpoint: "" # No default (optional) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) endpoint: "" # No default (optional) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ### Advanced ```yml outputs: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) service: "" # No default (optional) headers: "" # No default (optional) tls_skip_verify: false namespace: "" # No default (required) table: "" # No default (required) case_sensitive_columns: true row_operation: insert identifier_fields: [] storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) force_path_style_urls: false credentials: id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) endpoint: "" # No default (optional) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) endpoint: "" # No default (optional) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) schema_evolution: enabled: false partition_spec: () table_location: "" # No default (optional) schema_metadata: "" new_column_type_mapping: "" # No default (optional) require_schema_metadata: false commit: manifest_merge_enabled: true max_snapshot_age: 24h max_retries: 3 parquet: string_encoding: delta_length_byte_array batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ## [](#catalog-integration)Catalog integration This output works with REST catalog implementations including Apache Polaris, AWS Glue Data Catalog, and Databricks Unity Catalog. ### [](#apache-polaris)Apache Polaris To use with [Apache Polaris](https://polaris.apache.org): - Set `catalog.url` to the Polaris REST endpoint (e.g., `[http://localhost:8181/api/catalog](http://localhost:8181/api/catalog)`). - Set `catalog.warehouse` to the catalog name configured in Polaris. - Configure `catalog.auth.oauth2` with client credentials granted access to the catalog. ### [](#aws-glue-data-catalog)AWS Glue Data Catalog To use with AWS Glue Data Catalog: - Set `catalog.url` to `[https://glue..amazonaws.com/iceberg](https://glue.\.amazonaws.com/iceberg)` (the REST client appends the API version automatically). - Set `catalog.warehouse` to your AWS account ID (the Glue catalog identifier). - Set `schema_evolution.table_location` to an S3 prefix (e.g., `s3://my-bucket/`) since Glue does not automatically assign table locations. - Configure `catalog.auth.aws_sigv4` with the appropriate region and set `service` to `glue`. - Configure `storage.aws_s3` with the same bucket and region. ### [](#azure-blob-storage-adls-gen2)Azure Blob Storage (ADLS Gen2) To use with Azure Data Lake Storage Gen2: - Configure `storage.azure_blob_storage` with your storage account name and container. - Authenticate using one of: `storage_access_key` (shared key), `storage_sas_token`, or `storage_connection_string`. - The storage account must have hierarchical namespace (HNS) enabled for ADLS Gen2 compatibility. ## [](#type-mapping)Type mapping | Bloblang type | Iceberg type | | --- | --- | | string | string | | bytes | binary | | bool | boolean | | number | double | | timestamp | timestamp (with timezone) | | object | struct | | array | list | ## [](#row-level-operations)Row-level operations By default, this output is append-only: every message becomes a new row (`row_operation: insert`), and existing configurations are unaffected. Set `row_operation` to apply a per-message operation, with `identifier_fields` defining the row identity: - `insert`: Append the row. This is an unconditional append, and is not keyed or deduplicated. - `upsert`: Replace any existing rows matching `identifier_fields`, then append this row. This is equivalent to Iceberg’s Flink UPSERT mode. - `delete`: Remove rows matching `identifier_fields`. `row_operation` supports interpolation, so the operation can be driven by the data itself, for example by mapping a change-data-capture stream’s operation field. No CDC-specific format is assumed. The field is named `row_operation` to distinguish it from Iceberg’s snapshot-level operation. `upsert` and `delete` require `identifier_fields` and use Iceberg merge-on-read equality deletes, which require table format version 2. A version-1 table is automatically upgraded to version 2 on the first `upsert` or `delete`. This upgrade is irreversible. ### [](#identifier-fields)Identifier fields `identifier_fields` must reference existing table columns of a primitive, non-floating-point type. A static `upsert` or `delete` is validated at startup. An interpolated `row_operation` is validated for each message at write time, so an empty `identifier_fields` is not caught until the first `upsert` or `delete` message arrives. Identifier columns of a temporal type (`timestamp`, `timestamptz`, `date`, `time`) must arrive as time values, not bare numbers. A numeric epoch is ambiguous as a delete key and is rejected at write time, so convert it to a timestamp upstream. If the table is partitioned, every partition source column must be one of the `identifier_fields`, because equality deletes are partition-scoped. When this output auto-creates a table through `schema_evolution`, the `identifier_fields` columns are created as required and registered as the table’s Iceberg identifier-field-ids, so downstream engines and other writers see the primary key. As a result, a null or missing value in an identifier column is rejected on write, even for `insert`. Identifier columns must therefore be present at creation, either in the first message or declared through `schema_metadata`. Pre-existing tables are never modified. ### [](#batching-and-ordering)Batching and ordering Within a single batch, the last `upsert` or `delete` for each `identifier_fields` key wins. Each batch containing an `upsert` or `delete` is committed as its own snapshot. These commits are never coalesced, which is required for correctness, so a high-throughput mutation workload produces one snapshot per batch. Size batches accordingly, and run regular table maintenance (snapshot expiry and compaction) to keep metadata manageable. Pure `insert`\-only batches keep the original append fast path, which does coalesce commits. Ordering only holds within a batch. With more than one batch in flight, concurrent batches can commit out of order, so a stale `upsert` might overwrite a newer one for the same key. Set `max_in_flight: 1` for keyed (change-data-capture) workloads to preserve per-key order. This is enforced by config linting whenever `row_operation` is anything other than a static `insert`. > ⚠️ **CAUTION** > > `insert` is an unconditional append, and is not keyed or deduplicated. For keyed data, including change-data-capture, map create and read events to `upsert`, never `insert`. Mixing `insert` with `upsert` or `delete` on the same key in one batch produces duplicate rows. ### [](#change-data-capture-example)Change-data-capture example Materialize a change-data-capture stream into an Iceberg table. The mapping derives the row operation from the source’s operation field (here Debezium’s `op`, where `c`, `r`, and `u` map to `upsert`, and `d` maps to `delete`) and selects the row image, while `identifier_fields` is the primary key: ```yaml input: redpanda: seed_brokers: [ localhost:9092 ] topics: [ dbserver.inventory.customers ] consumer_group: iceberg_sink pipeline: processors: - mapping: | meta op = match this.op { "d" => "delete", _ => "upsert", } # Debezium puts the row image in 'after', or 'before' for deletes. root = this.after | this.before output: iceberg: catalog: url: http://localhost:8181/api/catalog namespace: inventory table: customers row_operation: ${! metadata("op") } identifier_fields: [ id ] # Keyed writes must stay ordered. A single batch in flight prevents # concurrent batches from committing a stale update over a newer one. max_in_flight: 1 storage: aws_s3: bucket: my-iceberg-data region: us-east-1 ``` ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#case_sensitive_columns)`case_sensitive_columns` Controls how message field names are matched against table column names, and how column references in the partition spec are resolved. When `true` (the default), names must match exactly. When `false`, matching is case-insensitive — set this when your downstream catalog or query engine treats column names as case-insensitive (the iceberg specification’s recommended convention) so that, for example, a message keyed `"COLUMN"` lands in an existing `column` rather than triggering schema evolution. Ambiguous case-only duplicates in the input are rejected. **Type**: `bool` **Default**: `true` ### [](#catalog)`catalog` REST catalog configuration. **Type**: `object` ### [](#catalog-auth)`catalog.auth` Authentication configuration for the REST catalog. Only one authentication method can be active at a time. **Type**: `object` ### [](#catalog-auth-aws_sigv4)`catalog.auth.aws_sigv4` AWS SigV4 authentication (for AWS Glue Data Catalog or API Gateway). **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials)`catalog.auth.aws_sigv4.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials-from_ec2_role)`catalog.auth.aws_sigv4.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#catalog-auth-aws_sigv4-credentials-id)`catalog.auth.aws_sigv4.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-profile)`catalog.auth.aws_sigv4.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role)`catalog.auth.aws_sigv4.credentials.role` A role ARN to assume. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role_external_id)`catalog.auth.aws_sigv4.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-secret)`catalog.auth.aws_sigv4.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-token)`catalog.auth.aws_sigv4.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#catalog-auth-aws_sigv4-endpoint)`catalog.auth.aws_sigv4.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#catalog-auth-aws_sigv4-region)`catalog.auth.aws_sigv4.region` The AWS region to target. **Type**: `string` ### [](#catalog-auth-aws_sigv4-service)`catalog.auth.aws_sigv4.service` AWS service name for SigV4 signing. **Type**: `string` ### [](#catalog-auth-aws_sigv4-tcp)`catalog.auth.aws_sigv4.tcp` TCP socket configuration. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-connect_timeout)`catalog.auth.aws_sigv4.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive)`catalog.auth.aws_sigv4.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-count)`catalog.auth.aws_sigv4.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-idle)`catalog.auth.aws_sigv4.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-interval)`catalog.auth.aws_sigv4.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-tcp_user_timeout)`catalog.auth.aws_sigv4.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-bearer)`catalog.auth.bearer` Static bearer token for authentication. For testing only, not recommended for production. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#catalog-auth-oauth2)`catalog.auth.oauth2` OAuth2 authentication configuration. **Type**: `object` ### [](#catalog-auth-oauth2-client_id)`catalog.auth.oauth2.client_id` OAuth2 client identifier. **Type**: `string` ### [](#catalog-auth-oauth2-client_secret)`catalog.auth.oauth2.client_secret` OAuth2 client secret. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#catalog-auth-oauth2-scope)`catalog.auth.oauth2.scope` OAuth2 scope to request. **Type**: `string` ### [](#catalog-auth-oauth2-server_uri)`catalog.auth.oauth2.server_uri` OAuth2 token endpoint URI. **Type**: `string` **Default**: `/v1/oauth/tokens` ### [](#catalog-headers)`catalog.headers` Custom HTTP headers to include in all requests to the catalog. **Type**: `object` ```yaml # Examples: headers: X-Api-Key: your-api-key ``` ### [](#catalog-tls_skip_verify)`catalog.tls_skip_verify` Skip TLS certificate verification. Not recommended for production. **Type**: `bool` **Default**: `false` ### [](#catalog-url)`catalog.url` The REST catalog endpoint URL. **Type**: `string` ```yaml # Examples: url: http://localhost:8181/api/catalog # --- url: https://polaris.example.com/api/catalog # --- url: https://glue.us-east-1.amazonaws.com/iceberg ``` ### [](#catalog-warehouse)`catalog.warehouse` The REST catalog warehouse. **Type**: `string` ```yaml # Examples: warehouse: redpanda-catalog ``` ### [](#commit)`commit` Commit behavior configuration. **Type**: `object` ### [](#commit-manifest_merge_enabled)`commit.manifest_merge_enabled` Merge small manifest files during commits to reduce metadata overhead. **Type**: `bool` **Default**: `true` ### [](#commit-max_retries)`commit.max_retries` Maximum number of times to retry a failed transaction commit. **Type**: `int` **Default**: `3` ### [](#commit-max_snapshot_age)`commit.max_snapshot_age` Maximum age of snapshots to retain for time-travel queries. Set to zero to disable removing old snapshots. **Type**: `string` **Default**: `24h` ### [](#identifier_fields)`identifier_fields[]` The columns forming the row identity (the Iceberg identifier fields, or equality-delete key) used by `upsert` and `delete`. Required when `row_operation` can evaluate to `upsert` or `delete`, and must reference existing table columns of a primitive, non-floating-point type. See the [Row-level operations](#row-level-operations) section above for the full constraints, including the temporal-type and partitioning rules and when the requirement is enforced. **Type**: `array` **Default**: `[]` ```yaml # Examples: identifier_fields: - id # --- identifier_fields: - tenant_id - user_id ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `4` ### [](#namespace)`namespace` The Iceberg namespace for the table, dot delimiters are split as nested namespaces. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: namespace: analytics.events # --- namespace: production ``` ### [](#parquet)`parquet` Parquet writer configuration. **Type**: `object` ### [](#parquet-string_encoding)`parquet.string_encoding` The encoding to use for string and binary columns. Use `plain` for compatibility with readers that do not support `DELTA_LENGTH_BYTE_ARRAY` encoding, such as AWS Redshift Spectrum. **Type**: `string` **Default**: `delta_length_byte_array` **Options**: `plain`, `delta_length_byte_array` ### [](#row_operation)`row_operation` The row-level operation to apply for each message: `insert` (append), `upsert` (replace rows matching `identifier_fields`, then append), or `delete` (remove rows matching `identifier_fields`). Supports interpolation so the operation can be driven by the data, such as a change-data-capture stream’s operation field. Defaults to `insert`, preserving the original append-only behavior. See the [Row-level operations](#row-level-operations) section above for the full semantics, the format-version-2 upgrade, batching behavior, and important caveats. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `insert` ```yaml # Examples: row_operation: insert # --- row_operation: ${! metadata("op") } # --- row_operation: ${! this.op == "d" ? "delete" : "upsert" } ``` ### [](#schema_evolution)`schema_evolution` Schema evolution configuration. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Enable automatic schema evolution. When enabled, new columns will be automatically added to the table. **Type**: `bool` **Default**: `false` ### [](#schema_evolution-new_column_type_mapping)`schema_evolution.new_column_type_mapping` An optional Bloblang mapping to customize column types during schema evolution. This mapping is executed for each new column and can override the inferred or schema-metadata-derived type. The mapping receives an object with fields `name` (column name), `path` (dot-separated path), `value` (sample value), `inferred_type` (the type that would be used without this mapping), `message` (the full message body), `namespace`, and `table`. It must return a string with a valid Iceberg type name: `boolean`, `int`, `long`, `float`, `double`, `string`, `binary`, `date`, `time`, `timestamp`, `timestamptz`, `uuid`, `decimal(p,s)`, or `fixed[n]`. **Type**: `string` ### [](#schema_evolution-partition_spec)`schema_evolution.partition_spec` A Bloblang expression to evaluate when a new table is created to determine the table’s partition spec. The result of the mapping should be an Iceberg partition spec in the same string format as the Redpanda Streaming Topic Property (see Redpanda Core documentation for Iceberg topics). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `()` ```yaml # Examples: partition_spec: (col1) # --- partition_spec: (nested.col) # --- partition_spec: (year(my_ts_col)) # --- partition_spec: (year(my_ts_col), col2) # --- partition_spec: (hour(my_ts_col), truncate(42, col2)) # --- partition_spec: (day(my_ts_col), bucket(4, nested.col)) # --- partition_spec: (day(my_ts_col), void(`non.nested column.with.dots`), identity(nested.column)) ``` ### [](#schema_evolution-require_schema_metadata)`schema_evolution.require_schema_metadata` When `true`, writing a numeric value into a `timestamp`, `timestamptz`, `date`, or `time` column without `schema_metadata` registered for that column is a hard error. The default `false` permits a fallback path that interprets bare numeric timestamps as Unix seconds and bare numeric times as already-microseconds — convenient, but silently wrong if upstream produced milliseconds. Enable this when you cannot guarantee the upstream attaches schema metadata and want to fail loudly rather than corrupt dates by ~50,000 years. No effect on time-typed columns receiving `time.Time`/`time.Duration` Go values, which carry their own unit unambiguously, and no effect on non-time columns. Requires `schema_metadata` to be set. **Type**: `bool` **Default**: `false` ### [](#schema_evolution-schema_metadata)`schema_evolution.schema_metadata` The name of a message metadata field containing a schema definition. When set, the schema is used to determine column types during schema evolution and table creation instead of inferring types from values. The schema must be in the standard common schema format (the same format used by the `parquet_encode` processor’s `schema_metadata` field). For batches of messages, the first message’s schema is used. Record presence drives schema shape: fields declared in the schema metadata that are absent from the record are not added to the table, while the metadata controls column ordering, naming, and types for fields that are present. In case-insensitive mode, top-level column names use the metadata’s casing — record keys are matched by case-folding and the metadata’s name is what lands in the table. **Type**: `string` **Default**: `""` ### [](#schema_evolution-table_location)`schema_evolution.table_location` A prefix used as the location for new tables when the catalog does not automatically assign one. For example, AWS Glue requires explicit table locations. When set, table locations are derived as `{prefix}{namespace}/{table}`. **Type**: `string` ```yaml # Examples: table_location: s3://my-iceberg-bucket/ ``` ### [](#storage)`storage` Storage backend configuration for data files. Exactly one of `aws_s3`, `gcp_cloud_storage`, or `azure_blob_storage` must be specified. **Type**: `object` ### [](#storage-aws_s3)`storage.aws_s3` S3 storage configuration. **Type**: `object` ### [](#storage-aws_s3-bucket)`storage.aws_s3.bucket` The S3 bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-aws_s3-credentials)`storage.aws_s3.credentials` Static AWS credentials for S3 access. When not specified, credentials are loaded from the default AWS credential chain. **Type**: `object` ### [](#storage-aws_s3-credentials-id)`storage.aws_s3.credentials.id` The AWS access key ID. **Type**: `string` ### [](#storage-aws_s3-credentials-secret)`storage.aws_s3.credentials.secret` The AWS secret access key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#storage-aws_s3-credentials-token)`storage.aws_s3.credentials.token` The AWS session token, required when using short term credentials. **Type**: `string` ### [](#storage-aws_s3-endpoint)`storage.aws_s3.endpoint` Custom endpoint for S3-compatible storage (e.g., MinIO). **Type**: `string` ```yaml # Examples: endpoint: http://localhost:9000 ``` ### [](#storage-aws_s3-force_path_style_urls)`storage.aws_s3.force_path_style_urls` Forces the client API to use path style URLs, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#storage-aws_s3-region)`storage.aws_s3.region` The AWS region. **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#storage-azure_blob_storage)`storage.azure_blob_storage` Azure Blob Storage (ADLS Gen2) configuration. **Type**: `object` ### [](#storage-azure_blob_storage-container)`storage.azure_blob_storage.container` The Azure blob container name. **Type**: `string` ```yaml # Examples: container: iceberg-data ``` ### [](#storage-azure_blob_storage-endpoint)`storage.azure_blob_storage.endpoint` Custom endpoint for Azure-compatible storage. **Type**: `string` ### [](#storage-azure_blob_storage-storage_access_key)`storage.azure_blob_storage.storage_access_key` Azure storage access key for shared key authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#storage-azure_blob_storage-storage_account)`storage.azure_blob_storage.storage_account` The Azure storage account name. **Type**: `string` ```yaml # Examples: storage_account: mystorageaccount ``` ### [](#storage-azure_blob_storage-storage_connection_string)`storage.azure_blob_storage.storage_connection_string` Azure storage connection string. Use this or other auth methods, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#storage-azure_blob_storage-storage_sas_token)`storage.azure_blob_storage.storage_sas_token` SAS token for authentication. Prefix with the container name followed by a dot if container-specific. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#storage-gcp_cloud_storage)`storage.gcp_cloud_storage` Google Cloud Storage configuration. **Type**: `object` ### [](#storage-gcp_cloud_storage-bucket)`storage.gcp_cloud_storage.bucket` The GCS bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-gcp_cloud_storage-credentials_file)`storage.gcp_cloud_storage.credentials_file` Path to a GCP credentials JSON file. **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_json)`storage.gcp_cloud_storage.credentials_json` GCP credentials JSON content. Use this or `credentials_file`, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_type)`storage.gcp_cloud_storage.credentials_type` The type of credentials to use. Valid values: `service_account`, `authorized_user`, `impersonated_service_account`, `external_account`. **Type**: `string` ```yaml # Examples: credentials_type: service_account ``` ### [](#storage-gcp_cloud_storage-endpoint)`storage.gcp_cloud_storage.endpoint` Custom endpoint for GCS-compatible storage. **Type**: `string` ### [](#table)`table` The destination Iceberg table name. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table: user_events # --- table: events_${!meta("topic")} ``` --- # Page 334: inproc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/inproc.md --- # inproc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: inproc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/inproc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/inproc.adoc description: Sends data directly to Redpanda Connect inputs in the same process by connecting to a unique ID, to link isolated streams when running in streams mode. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- ```yml outputs: label: "" inproc: "" ``` Sends data directly to Redpanda Connect inputs by connecting to a unique ID. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 335: kafka_franz **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/kafka_franz.md --- # kafka_franz > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: kafka_franz latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/kafka_franz page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/kafka_franz.adoc description: A Kafka output using the Franz Kafka client library. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. The `kafka_franz` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgments back to the input. This output often outperforms the traditional `kafka` output, as well as providing more useful logs and error messages. This output uses the [Franz Kafka client library](https://github.com/twmb/franz-go). #### Common ```yml outputs: label: "" kafka_franz: seed_brokers: [] # No default (required) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true acks: all compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: 0 max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enables the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable. **Type**: `bool` **Default**: `true` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. **Type**: `int` **Default**: `10` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. When `idempotent_write` is enabled, this is capped at 5 by the Kafka protocol (and at 1 for Kafka < v1.0.0). When `idempotent_write` is disabled, higher values improve throughput by pipelining requests but may cause out-of-order delivery. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum space (in bytes) that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 336: kafka **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/kafka.md --- # kafka > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: kafka latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/kafka page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/kafka.adoc description: The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. The `kafka` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. #### Common ```yml outputs: label: "" kafka: addresses: [] # No default (required) topic: "" # No default (required) target_version: "" # No default (optional) key: "" partitioner: fnv1a_hash compression: none static_headers: "" # No default (optional) metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" kafka: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" topic: "" # No default (required) client_id: benthos target_version: "" # No default (optional) rack_id: "" key: "" partitioner: fnv1a_hash partition: "" custom_topic_creation: enabled: false partitions: -1 replication_factor: -1 compression: none static_headers: "" # No default (optional) metadata: exclude_prefixes: [] inject_tracing_map: "" # No default (optional) max_in_flight: 64 idempotent_write: false ack_replicas: false max_msg_bytes: 1000000 timeout: 5s retry_as_batch: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_retries: 0 backoff: initial_interval: 3s max_interval: 10s max_elapsed_time: 30s timestamp_ms: "" # No default (optional) ``` The configuration field `ack_replicas` determines whether Redpanda Connect waits for acknowledgement from all replicas or just a single broker. Both the `key` and `topic` fields can be dynamically set using function interpolations described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). [Metadata](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/metadata/) will be added to each message sent as headers (version 0.11+), but can be restricted using the field [`metadata`](#metadata). ## [](#strict-ordering-and-retries)Strict ordering and retries When strict ordering is required for messages written to topic partitions it is important to ensure that both the field `max_in_flight` is set to `1` and that the field `retry_as_batch` is set to `true`. You must also ensure that failed batches are never rerouted back to the same output. This can be done by setting the field `max_retries` to `0` and `backoff.max_elapsed_time` to empty, which will apply back pressure indefinitely until the batch is sent successfully. However, this also means that manual intervention will eventually be required in cases where the batch cannot be sent due to configuration problems such as an incorrect `max_msg_bytes` estimate. A less strict but automated alternative would be to route failed batches to a dead letter queue using a [`fallback` broker](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/), but this would allow subsequent batches to be delivered in the meantime whilst those failed batches are dealt with. ## [](#troubleshooting)Troubleshooting If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer [`kafka_franz` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/kafka_franz/). - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#ack_replicas)`ack_replicas` Ensure that messages have been copied across all replicas before acknowledging receipt. **Type**: `bool` **Default**: `false` ### [](#addresses)`addresses[]` A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `3s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#compression)`compression` The compression algorithm to use. **Type**: `string` **Default**: `none` **Options**: `none`, `snappy`, `lz4`, `gzip`, `zstd` ### [](#custom_topic_creation)`custom_topic_creation` If enabled, topics will be created with the specified number of partitions and replication factor if they do not already exist. **Type**: `object` ### [](#custom_topic_creation-enabled)`custom_topic_creation.enabled` Whether to enable custom topic creation. **Type**: `bool` **Default**: `false` ### [](#custom_topic_creation-partitions)`custom_topic_creation.partitions` The number of partitions to create for new topics. Leave at -1 to use the broker configured default. Must be >= 1. **Type**: `int` **Default**: `-1` ### [](#custom_topic_creation-replication_factor)`custom_topic_creation.replication_factor` The replication factor to use for new topics. Leave at -1 to use the broker configured default. Must be an odd number, and less then or equal to the number of brokers. **Type**: `int` **Default**: `-1` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER` and can be disabled if this permission is not available. **Type**: `bool` **Default**: `false` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_msg_bytes)`max_msg_bytes` The maximum size in bytes of messages sent to the target topic. **Type**: `int` **Default**: `1000000` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent with messages as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#partition)`partition` The manually-specified partition to publish messages to, relevant only when the field `partitioner` is set to `manual`. Must be able to parse as a 32-bit integer. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#partitioner)`partitioner` The partitioning algorithm to use. **Type**: `string` **Default**: `fnv1a_hash` **Options**: `fnv1a_hash`, `murmur2_hash`, `random`, `round_robin`, `manual` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#retry_as_batch)`retry_as_batch` When enabled forces an entire batch of messages to be retried if any individual message fails on a send, otherwise only the individual messages that failed are retried. Disabling this helps to reduce message duplicates during intermittent errors, but also makes it impossible to guarantee strict ordering of messages. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#static_headers)`static_headers` An optional map of static headers that should be added to messages in addition to metadata. **Type**: `object` ```yaml # Examples: static_headers: first-static-header: value-1 second-static-header: value-2 ``` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `5s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 337: mongodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/mongodb.md --- # mongodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mongodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/mongodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/mongodb.adoc description: Inserts items into a MongoDB collection. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- Inserts items into a MongoDB collection. #### Common ```yml outputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight, in parallel, for improved performance. You can tune the maximum number of in flight messages (or message batches) using the `max_in_flight` field. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection)`collection` The name of the target collection. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, and `update-one`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. This output uses `filter_map` to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `update-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username required to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` requests acknowledgement from MongoDB, which is created when write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` requests acknowledgement, which write operations propagate to the specified number of MongoDB instances. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 338: mqtt **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/mqtt.md --- # mqtt > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mqtt latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/mqtt page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/mqtt.adoc description: Pushes messages to an MQTT broker. page-git-created-date: "2024-11-07" page-git-modified-date: "2026-05-26" --- Pushes messages to an MQTT broker. #### Common ```yml outputs: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false retained_interpolated: "" # No default (optional) max_in_flight: 64 ``` The `topic` field can be dynamically set using function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#qos)`qos` The QoS value to set for each message. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#retained)`retained` Set message as retained on the topic. **Type**: `bool` **Default**: `false` ### [](#retained_interpolated)`retained_interpolated` Override the value of `retained` with an interpolable value, this allows it to be dynamically set based on message contents. The value must resolve to either `true` or `false`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` The maximum amount of time to wait to write data before the attempt is abandoned. **Type**: `string` **Default**: `3s` ```yaml # Examples: write_timeout: 1s # --- write_timeout: 500ms ``` --- # Page 339: nats_jetstream **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/nats_jetstream.md --- # nats_jetstream > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_jetstream latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/nats_jetstream page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/nats_jetstream.adoc description: Write messages to a NATS JetStream subject. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Write messages to a NATS JetStream subject. #### Common ```yml outputs: label: "" nats_jetstream: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 ``` #### Advanced ```yml outputs: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 340: nats_kv **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/nats_kv.md --- # nats_kv > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_kv latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/nats_kv page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/nats_kv.adoc description: Put messages in a NATS key-value bucket. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Put messages into a NATS key-value bucket. #### Common ```yml outputs: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 ``` #### Advanced ```yml outputs: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` The `key` field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), which lets you create a unique key for each message. ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.${! json("meta.type") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 341: nats **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/nats.md --- # nats > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/nats page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/nats.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/nats.adoc description: Publish to an NATS subject. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Publish to an NATS subject. #### Common ```yml outputs: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" nats: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` This output interpolates functions within the subject field. For a full list of functions, see [configuration:interpolation.adoc#bloblang-queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` The subject to publish to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 342: opensearch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/opensearch.md --- # opensearch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: opensearch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/opensearch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/opensearch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/opensearch.adoc description: Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. #### Common ```yml outputs: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Both the `id` and `index` fields can be dynamically set using function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#examples)Examples ### [](#updating-documents)Updating Documents When [updating documents](https://opensearch.org/docs/latest/api-reference/document-apis/update-document/) the request body should contain a combination of a `doc`, `upsert`, and/or `script` fields at the top level, this should be done via mapping processors. ```yaml output: processors: - mapping: | meta id = this.id root.doc = this opensearch: urls: [ TODO ] index: foo id: ${! @id } action: update ``` ## [](#fields)Fields ### [](#action)`action` The action to take on the document. This field must resolve to one of the following action types: `index`, `update` or `delete`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#aws)`aws` Enables and customises connectivity to Amazon Elastic Service. **Type**: `object` ### [](#aws-credentials)`aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#aws-credentials-from_ec2_role)`aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#aws-credentials-id)`aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#aws-credentials-profile)`aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#aws-credentials-role)`aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#aws-credentials-role_external_id)`aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#aws-credentials-secret)`aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#aws-credentials-token)`aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#aws-enabled)`aws.enabled` Whether to connect to Amazon Elastic Service. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region to target. **Type**: `string` ### [](#aws-tcp)`aws.tcp` TCP socket configuration. **Type**: `object` ### [](#aws-tcp-connect_timeout)`aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#aws-tcp-keep_alive)`aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#aws-tcp-keep_alive-count)`aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#aws-tcp-keep_alive-idle)`aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-keep_alive-interval)`aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-tcp_user_timeout)`aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` The ID for indexed messages. Interpolation should be used in order to create a unique ID for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The index to place messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` An optional pipeline id to preprocess incoming documents. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#routing)`routing` The routing key to use for the document. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` --- # Page 343: otlp_grpc **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/otlp_grpc.md --- # otlp_grpc > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: otlp_grpc latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/otlp_grpc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/otlp_grpc.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-08-11" --- Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Sends OpenTelemetry telemetry data to a remote collector via OTLP/gRPC protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml outputs: label: "" otlp_grpc: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" otlp_grpc: endpoint: "" # No default (required) headers: {} timeout: 30s compression: gzip tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via gRPC. ## [](#authentication)Authentication Supports multiple authentication methods: - Bearer token authentication (via `auth_token` field) - OAuth v2 (via `oauth2` configuration block) > 📝 **NOTE** > > OAuth2 requires TLS to be enabled. ## [](#fields)Fields ### [](#compression)`compression` Compression type for gRPC requests. Options: 'gzip' or 'none'. **Type**: `string` **Default**: `gzip` **Options**: `gzip`, `none` ### [](#endpoint)`endpoint` The gRPC endpoint of the remote OTLP collector. **Type**: `string` ### [](#headers)`headers` A map of headers to add to the gRPC request metadata. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth2)`oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for gRPC requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for gRPC client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 344: otlp_http **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/otlp_http.md --- # otlp_http > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: otlp_http latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/otlp_http page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/otlp_http.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-08-11" --- Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Sends OpenTelemetry telemetry data to a remote collector via OTLP/HTTP protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml outputs: label: "" otlp_http: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" otlp_http: endpoint: "" # No default (required) content_type: protobuf headers: {} timeout: 30s proxy_url: "" follow_redirects: false disable_http2: false tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via HTTP POST. ## [](#endpoints)Endpoints The output automatically appends the signal type path to the base endpoint: - Traces: `{endpoint}/v1/traces` - Logs: `{endpoint}/v1/logs` - Metrics: `{endpoint}/v1/metrics` ## [](#content-types)Content types Supports two content types: - `protobuf` (default): `application/x-protobuf` - `json`: `application/json` ## [](#authentication)Authentication Supports multiple authentication methods: - Basic authentication - OAuth v1 - OAuth v2 - JWT ## [](#fields)Fields ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` Content type for HTTP requests. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `protobuf` **Options**: `protobuf`, `json` ### [](#disable_http2)`disable_http2` Whether or not to disable HTTP/2. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` The HTTP endpoint of the remote OTLP collector (without the signal path). **Type**: `string` ### [](#follow_redirects)`follow_redirects` Transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `false` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth)`oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for HTTP requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for HTTP client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 345: pinecone **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/pinecone.md --- # pinecone > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: pinecone latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/pinecone page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/pinecone.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/pinecone.adoc description: Inserts items into a Pinecone index. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Inserts items into a Pinecone index. #### Common ```yml outputs: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` #### Advanced ```yml outputs: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors namespace: "" id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#api_key)`api_key` The Pinecone API key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#host)`host` The host for the Pinecone index. **Type**: `string` ### [](#id)`id` The ID for the index entry in Pinecone. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional mapping of message to metadata in the Pinecone index entry. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: root = {"summary": this.summary, "foo": this.other_field} ``` ### [](#namespace)`namespace` The namespace to write to - writes to the default namespace by default. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#operation)`operation` The operation to perform against the Pinecone index. **Type**: `string` **Default**: `upsert-vectors` **Options**: `update-vector`, `upsert-vectors`, `delete-vectors` ### [](#vector_mapping)`vector_mapping` The mapping to extract out the vector from the document. The result must be a floating point array. Required if not a delete operation. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 346: qdrant **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/qdrant.md --- # qdrant > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: qdrant latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/qdrant page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/qdrant.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/qdrant.adoc description: Adds items to a Qdrant collection. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Adds items to a [Qdrant](https://qdrant.tech/) collection #### Common ```yml outputs: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` #### Advanced ```yml outputs: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token for authentication. Defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection_name)`collection_name` The name of the collection in Qdrant. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#id)`id` The ID of the point to insert. Can be a UUID string or positive integer. **Type**: `string` ```yaml # Examples: id: root = "dc88c126-679f-49f5-ab85-04b77e8c2791" # --- id: root = 832 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#payload_mapping)`payload_mapping` An optional mapping of message to payload associated with the point. **Type**: `string` **Default**: `root = {}` ```yaml # Examples: payload_mapping: root = {"field": this.value, "field_2": 987} # --- payload_mapping: root = metadata() ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` The mapping to extract the vector from the document. **Type**: `string` ```yaml # Examples: vector_mapping: root = {"dense_vector": [0.352,0.532,0.754],"sparse_vector": {"indices": [23,325,532],"values": [0.352,0.532,0.532]}, "multi_vector": [[0.352,0.532],[0.352,0.532]]} # --- vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}} # --- vector_mapping: root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]} # --- vector_mapping: root = {"some_dense": [0.352,0.532,0.532,0.234]} ``` --- # Page 347: questdb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/questdb.md --- # questdb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: questdb page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/questdb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/questdb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/questdb.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-11-07" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Pushes messages to a [QuestDB](https://questdb.io/docs/) table. #### Common ```yml outputs: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` #### Advanced ```yml outputs: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) retry_timeout: "" # No default (optional) request_timeout: "" # No default (optional) request_min_throughput: "" # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` > ❗ **IMPORTANT** > > Redpanda Data recommends enabling the dedupe feature on the QuestDB server. For more information about deploying, configuring, and using QuestDB, see the [QuestDB documentation](https://questdb.io/docs/). ## [](#performance)Performance For improved performance, this output sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches), using the `max_in_flight` field. You can configure batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#address)`address` The host and port of the QuestDB server. **Type**: `string` ```yaml # Examples: address: localhost:9000 ``` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#designated_timestamp_field)`designated_timestamp_field` The name of the designated timestamp field in QuestDB. **Type**: `string` ### [](#designated_timestamp_unit)`designated_timestamp_unit` Units used for the designated timestamp field in QuestDB. **Type**: `string` **Default**: `auto` ### [](#doubles)`doubles[]` Columns that must be the `double` type, with `int` as the default. **Type**: `array` ### [](#error_on_empty_messages)`error_on_empty_messages` Mark a message as an error if it is empty after field validation. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` The password to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#request_min_throughput)`request_min_throughput` The minimum expected throughput in bytes per second for HTTP requests. If the throughput is lower than this value, the connection times out. The `quest_db` output uses this value to calculate an additional timeout on top of the `request_timeout`. This setting is useful for large requests. Set it to `0` to disable this logic. **Type**: `int` ### [](#request_timeout)`request_timeout` The period of time to wait for a response from the QuestDB server in addition to any connection timeout calculated for the `request_min_throughput` field. **Type**: `string` ### [](#retry_timeout)`retry_timeout` The period of time to continue retrying after a failed HTTP request. The interval between retries is an exponential backoff starting at 10 ms, and doubling after each failed attempt up to a maximum of 1 second. **Type**: `string` ### [](#symbols)`symbols[]` Columns that must be the `symbol` type. String values default to `string` types. **Type**: `array` ### [](#table)`table` The destination table in QuestDB. **Type**: `string` ```yaml # Examples: table: trades ``` ### [](#timestamp_string_fields)`timestamp_string_fields[]` String fields with textual timestamps. **Type**: `array` ### [](#timestamp_string_format)`timestamp_string_format` The timestamp format, which is used when parsing timestamp string fields and uses Golang’s time formatting. **Type**: `string` **Default**: `Jan _2 15:04:05.000000Z0700` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#token)`token` The bearer token to use for authentication, which takes precedence over the basic authentication username and password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#username)`username` The username to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` --- # Page 348: redis_hash **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redis_hash.md --- # redis_hash > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_hash latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redis_hash page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redis_hash.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redis_hash.adoc description: Sets Redis hash objects using the HSET command. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sets Redis hash objects using the HMSET command. #### Common ```yml outputs: label: "" redis_hash: url: "" # No default (required) key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" redis_hash: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` The field `key` supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. The field `fields` allows you to specify an explicit map of field names to interpolated values, also evaluated per message of a batch: ```yaml output: redis_hash: url: tcp://localhost:6379 key: ${!json("id")} fields: topic: ${!meta("kafka_topic")} partition: ${!meta("kafka_partition")} content: ${!json("document.text")} ``` If the field `walk_metadata` is set to `true` then Redpanda Connect will walk all metadata fields of messages and add them to the list of hash fields to set. If the field `walk_json_object` is set to `true` then Redpanda Connect will walk each message as a JSON object, extracting keys and the string representation of their value and adds them to the list of hash fields to set. The order of hash field extraction is as follows: 1. Metadata (if enabled) 2. JSON object (if enabled) 3. Explicit fields Where latter stages will overwrite matching field names of a former stage. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#fields-2)`fields` A map of key/value pairs to set as hash fields. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ### [](#key)`key` The key for each message, function interpolations should be used to create a unique key per message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#walk_json_object)`walk_json_object` Whether to walk each message as a JSON object and add each key/value pair to the list of hash fields to set. **Type**: `bool` **Default**: `false` ### [](#walk_metadata)`walk_metadata` Whether all metadata fields of messages should be walked and added to the list of hash fields to set. **Type**: `bool` **Default**: `false` --- # Page 349: redis_list **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redis_list.md --- # redis_list > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_list latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redis_list page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redis_list.adoc description: Pushes messages onto the end of a Redis list (which is created if it doesn't already exist) using the RPUSH command. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Pushes messages onto the end of a Redis list (which is created if it doesn’t already exist) using the RPUSH command. #### Common ```yml outputs: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) command: rpush ``` The field `key` supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to push elements to the Redis list **Type**: `string` **Default**: `rpush` **Options**: `rpush`, `lpush` ### [](#key)`key` The key for each message, function interpolations can be optionally used to create a unique key per message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: some_list # --- key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 350: redis_pubsub **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redis_pubsub.md --- # redis_pubsub > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_pubsub latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redis_pubsub page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redis_pubsub.adoc description: Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received. #### Common ```yml outputs: label: "" redis_pubsub: url: "" # No default (required) channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This output will interpolate functions within the channel field, you can find a list of functions [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#channel)`channel` The channel to publish messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 351: redis_streams **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redis_streams.md --- # redis_streams > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_streams latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redis_streams page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redis_streams.adoc description: Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn't already exist) using the XADD command. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn’t already exist) using the XADD command. #### Common ```yml outputs: label: "" redis_streams: url: "" # No default (required) stream: "" # No default (required) id: * body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] stream: "" # No default (required) id: * body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` It’s possible to specify a maximum length of the target stream by setting it to a value greater than 0, in which case this cap is applied only when Redis is able to remove a whole macro node, for efficiency. Redis stream entries are key/value pairs, as such it is necessary to specify the key to be set to the body of the message. All metadata fields of the message will also be set as key/value pairs, if there is a key collision between a metadata item and the body then the body takes precedence. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#body_key)`body_key` A key to set the raw body of the message to. **Type**: `string` **Default**: `body` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#id)`id` The entry ID for the stream message. Allows function interpolations. When set to `*` (the default), Redis auto-generates a unique ID based on the current time. Set a custom ID to control message ordering, for example to replay messages in upstream order. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `*` ```yaml # Examples: id: * # --- id: ${! @redis_stream } # --- id: ${! this.id } # --- id: ${! counter() }-0 ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_length)`max_length` When greater than zero enforces a rough cap on the length of the target stream. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify criteria for which metadata values are included in the message body. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#stream)`stream` The stream to add messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 352: redpanda_common **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_common.md --- # redpanda_common > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda_common latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redpanda_common page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redpanda_common.adoc description: Sends data to a Redpanda (Kafka) broker, using credentials defined in a common top-level redpanda config block. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [`redpanda` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) components. Sends data to a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda/) and [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) instead. #### Common ```yml outputs: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#key)`key` A key to populate for each message (optional). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `10` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 353: redpanda_migrator **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator.md --- # redpanda_migrator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda_migrator latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redpanda_migrator page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redpanda_migrator.adoc description: A specialised Kafka producer for comprehensive data migration between Apache Kafka and Redpanda clusters. page-git-created-date: "2024-10-02" page-git-modified-date: "2026-05-26" --- Migrates topics, schemas, and consumer groups between Kafka and Redpanda clusters. > ❗ **IMPORTANT** > > Pair this output with a [`redpanda_migrator` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator/) in the same pipeline. The following shows all available configuration fields and their defaults. #### Common ```yml outputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: "" # No default (optional) sync_topic_acls: false headers: "" # No default (optional) max_in_flight: 10 ``` #### Advanced ```yml outputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true acks: all compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: 0 max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: "" # No default (optional) sync_topic_interval: 5m sync_topic_acls: false serverless: false headers: "" # No default (optional) provenance_header: redpanda-migrator-provenance offset_header: redpanda-migrator-offset max_in_flight: 10 ``` ## [](#requirements)Requirements When the destination cluster enforces ACLs, the destination principal needs permission to create topics and add partitions, not only to produce records. Grant these ACLs at minimum: - Topic `CREATE`, `WRITE`, `ALTER`, and `DESCRIBE_CONFIGS`. Cluster `CREATE` also authorizes topic creation. - When consumer group migration is enabled, consumer group `READ`. - When `sync_topic_acls` is enabled, cluster `ALTER`. For the source-principal ACLs and full details, see [Required permissions](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator/#required-permissions). ## [](#multiple-migrator-pairs)Multiple migrator pairs Each migrator pair requires a unique `label`. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating. ## [](#performance-tuning)Performance tuning For high-throughput workloads, adjust the following settings: On this output: - `max_in_flight`: Set to the total number of partitions being migrated. Higher values provide no benefit beyond the partition count. On the paired [`redpanda_migrator` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator/#performance-tuning): - `partition_buffer_bytes`: Set to `2MB` to increase the per-partition buffer. - `max_yield_batch_bytes`: Set to `1MB` to yield larger batches. ## [](#synchronization-details)Synchronization details ### [](#topics)Topics - Topic names carry over from the source by default. Use the `topic` field with interpolation to rename topics at the destination. - The migrator creates each topic at the destination with the same partition count as the source. - Replication factor defaults to the source value. Set `topic_replication_factor` to override it. - The migrator copies a serverless-aware subset of topic configuration keys. - ACL replication is optional (`sync_topic_acls`): - The migrator does not copy `ALLOW WRITE` ACL entries. - The migrator downgrades `ALLOW ALL` ACL entries to `ALLOW READ`. - Resource pattern type and host filters carry over. ### [](#schema-registry)Schema Registry - Syncs once at startup, then periodically based on `schema_registry.interval` (default: every 5 minutes). Set `schema_registry.interval: 0s` for a one-time sync only. - Use include and exclude regex patterns to filter which subjects are migrated. - Use the `subject` field with interpolation to rename subjects at the destination. - Schema versions: `latest` (most recent version only) or `all` (full history). Defaults to `all`. - Include soft-deleted subjects with `schema_registry.include_deleted`. - The migrator can translate schema IDs to new destination IDs or preserve them as-is. See `schema_registry.translate_ids`. - Schema normalization is optional (`schema_registry.normalize`). - Compatibility settings carry over per-subject. Schema metadata and rules do not carry over in Serverless mode. ### [](#consumer-groups)Consumer groups - Sync periodically based on `consumer_groups.interval` (default: every 1 minute). - Use include and exclude regex patterns to filter which groups are migrated. - By default, all groups except those in `Dead` state migrate. Set `consumer_groups.only_empty: true` to migrate only `Empty` state groups. - The migrator translates consumer offsets using timestamps. This is approximate and may be imprecise when multiple records share the same timestamp. - Offsets only move forward and never rewind at the destination. - Source and destination must have matching partition counts for each migrated group. ## [](#how-it-works)How it works Each migration component synchronizes on a different schedule: - **Topics**: Sync from source on startup and every 5 minutes by default, including source topics that have no current data (for example, after retention cleanup). Configure the sync interval with [`sync_topic_interval`](#sync_topic_interval), or set it to `0s` to disable periodic sync. When periodic sync is disabled, topics are still created on demand when the first message arrives. - **Schema Registry**: Syncs at startup, then periodically as configured. - **Consumer groups**: Sync in the background, filtered to the topics being migrated. ## [](#guarantees)Guarantees The migrator upholds the following guarantees: - Creates each destination topic with the intended partition count and replication factor. - Never overwrites existing destination topics and logs any partition count mismatches. - Consumer group offsets never rewind. - ACL replication never grants write access at the destination. ## [](#limitations)Limitations - The destination cluster’s Schema Registry must be in `READWRITE` or `IMPORT` mode. - Offset translation is best-effort. - Consumer group migration requires identical partition counts at source and destination. ## [](#metrics)Metrics | Metric Name | Type | Labels | Description | | --- | --- | --- | --- | | Topic migration | | | | | redpanda_migrator_topics_created_total | counter | | Total topics created on destination | | redpanda_migrator_topic_create_errors_total | counter | | Topic creation errors | | redpanda_migrator_topic_create_latency_ns | timer | | Topic creation latency (ns) | | Schema Registry migration | | | | | redpanda_migrator_sr_schemas_created_total | counter | | Schemas created in destination registry | | redpanda_migrator_sr_schema_create_errors_total | counter | | Schema creation errors | | redpanda_migrator_sr_schema_create_latency_ns | timer | | Schema creation latency (ns) | | redpanda_migrator_sr_compatibility_updates_total | counter | | Compatibility level updates applied | | redpanda_migrator_sr_compatibility_update_errors_total | counter | | Compatibility update errors | | redpanda_migrator_sr_compatibility_update_latency_ns | timer | | Compatibility update latency (ns) | | Consumer group migration | | | | | redpanda_migrator_cg_offsets_translated_total | counter | group | Offsets translated per consumer group | | redpanda_migrator_cg_offset_translation_errors_total | counter | group | Offset translation errors per group | | redpanda_migrator_cg_offset_translation_latency_ns | timer | group | Offset translation latency per group (ns) | | redpanda_migrator_cg_offsets_committed_total | counter | group | Offsets committed per consumer group | | redpanda_migrator_cg_offset_commit_errors_total | counter | group | Offset commit errors per group | | redpanda_migrator_cg_offset_commit_latency_ns | timer | group | Offset commit latency per group (ns) | | Consumer lag | | | | | redpanda_lag | gauge | topic, partition | Current consumer lag in messages for each topic partition. Shows difference between high water mark and current consumer position. | ## [](#examples)Examples ### [](#basic-migration)Basic migration Migrate topics, schemas and consumer groups from source to destination. ```yaml input: redpanda_migrator: seed_brokers: ["source:9092"] topics: ["orders", "payments"] consumer_group: "migration" output: redpanda_migrator: seed_brokers: ["destination:9092"] # Write to the same topic name topic: ${! metadata("kafka_topic") } schema_registry: url: "http://dest-registry:8081" translate_ids: true consumer_groups: interval: 1m ``` ### [](#migration-to-redpanda-serverless)Migration to Redpanda Serverless Migrate from Confluent/Kafka to Redpanda Cloud serverless cluster with authentication. ```yaml input: redpanda_migrator: seed_brokers: ["source-kafka:9092"] regexp_topics_include: - '.' regexp_topics_exclude: - '^_' consumer_group: "migrator_cg" schema_registry: url: "http://source-registry:8081" output: redpanda_migrator: seed_brokers: ["serverless-cluster.redpanda.com:9092"] tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "migrator" password: "migrator" schema_registry: url: "https://serverless-cluster.redpanda.com:8081" basic_auth: enabled: true username: "migrator" password: "migrator" translate_ids: true consumer_groups: exclude: - "migrator_cg" # Exclude the migration consumer group itself serverless: true # Enable serverless mode for restricted configurations ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_groups)`consumer_groups` **Type**: `object` ### [](#consumer_groups-enabled)`consumer_groups.enabled` Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed. **Type**: `bool` **Default**: `true` ### [](#consumer_groups-exclude)`consumer_groups.exclude[]` Regular expressions for consumer groups to exclude from offset migration. Takes precedence over include patterns. Useful for excluding system or temporary groups. **Type**: `array` ```yaml # Examples: exclude: [".*-test", ".*-temp", "connect-.*"] # --- exclude: ["dev-.*", "local-.*"] ``` ### [](#consumer_groups-fetch_timeout)`consumer_groups.fetch_timeout` Maximum time to wait for data when fetching records for timestamp-based offset translation. Increase for clusters with low message throughput. **Type**: `string` **Default**: `10s` ```yaml # Examples: fetch_timeout: 1s # Fast clusters # --- fetch_timeout: 10s # Slower clusters ``` ### [](#consumer_groups-include)`consumer_groups.include[]` Regular expressions for consumer groups to include in offset migration. If empty, all groups are included (unless excluded). **Type**: `array` ```yaml # Examples: include: ["prod-.*", "staging-.*"] # --- include: ["app-.*", "service-.*"] ``` ### [](#consumer_groups-interval)`consumer_groups.interval` How often to synchronise consumer group offsets. Regular syncing helps maintain offset accuracy during ongoing migration. **Type**: `string` **Default**: `1m` ```yaml # Examples: interval: 0s # Disabled # --- interval: 30s # Sync every 30 seconds # --- interval: 5m # Sync every 5 minutes ``` ### [](#consumer_groups-only_empty)`consumer_groups.only_empty` Whether to only migrate Empty consumer groups. When false (default), all statuses except Dead are included; when true, only Empty groups are migrated. **Type**: `bool` **Default**: `false` ### [](#headers)`headers` Custom headers to add to migrated records, keyed by header name with interpolated string values. Useful for injecting metadata such as processing timestamps or latency measurements that should surface as header values on the destination cluster. A custom header name that collides with `provenance_header` or `offset_header` is ignored, so those migration-critical headers are always protected. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` ```yaml # Examples: headers: x-migration-latency-ms: ${! timestamp_unix_milli() - meta("kafka_timestamp_ms") } x-migration-processed-at: ${! timestamp_unix_milli() } ``` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable. **Type**: `bool` **Default**: `true` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. Increase this value to improve throughput during migration. For optimal performance, set this to match the total number of partitions being migrated. Setting it higher than the partition count provides no additional benefit, as each partition can only have one in-flight batch at a time. Example: If migrating 100 partitions, set `max_in_flight: 100` for maximum throughput. **Type**: `int` **Default**: `10` ```yaml # Examples: max_in_flight: 64 # For a cluster with 64 partitions # --- max_in_flight: 128 # For multiple topics with combined 128 partitions ``` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. When `idempotent_write` is enabled, this is capped at 5 by the Kafka protocol (and at 1 for Kafka < v1.0.0). When `idempotent_write` is disabled, higher values improve throughput by pipelining requests but may cause out-of-order delivery. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum space in bytes that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#offset_header)`offset_header` The name of a message header to add to migrated records. This header contains the source offset, enabling exact consumer group offset translation during migration. When left empty (default), no offset header is added and consumer groups are migrated using timestamp-based positioning. This approach works well for most cases, but may be imprecise for consumer groups with no committed offsets when multiple records share the same timestamp (timestamps have millisecond resolution). Set this field to enable precise offset translation, especially when migrating consumer groups that are caught up or have minimal lag. Note: This header is only added when consumer group migration is enabled. **Type**: `string` **Default**: `redpanda-migrator-offset` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#provenance_header)`provenance_header` Header name to add to migrated records indicating their source cluster. When set, each migrated message receives a header with this name containing the source cluster’s seed broker addresses, enabling downstream systems to track message origins for auditing, debugging, or multi-cluster orchestration workflows. If empty, no provenance header is added to messages. The header value format is a comma-separated list of the source cluster’s `seed_brokers`. Example: Setting `provenance_header: "rp-source-cluster"` adds a header like `rp-source-cluster: "kafka-1:9092,kafka-2:9092"`. **Type**: `string` **Default**: `redpanda-migrator-provenance` ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods of SASL authentication, which are tried in order. If the broker supports the first mechanism, all connections will use that mechanism. If the first mechanism fails, the client picks the first supported mechanism. Connections fail if the broker does not support any client mechanisms. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-enabled)`schema_registry.enabled` Whether schema registry migration is enabled. When disabled, no schema operations are performed. **Type**: `bool` **Default**: `true` ### [](#schema_registry-exclude)`schema_registry.exclude[]` Regular expressions for schema subjects to exclude from migration. Takes precedence over include patterns. Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: exclude: [".*-test", ".*-temp"] # --- exclude: ["dev-.*", "local-.*"] ``` ### [](#schema_registry-include)`schema_registry.include[]` Regular expressions for schema subjects to include in migration. If empty, all subjects are included (unless excluded). Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: include: ["prod-.*", "staging-.*"] # --- include: ["user-.*", "order-.*"] ``` ### [](#schema_registry-include_deleted)`schema_registry.include_deleted` Whether to include soft-deleted schemas in migration. Useful for complete migration but may not be supported by all schema registries. **Type**: `bool` **Default**: `false` ### [](#schema_registry-interval)`schema_registry.interval` How often to synchronise schema registry subjects. Set to 0s for one-time sync at startup only. **Type**: `string` **Default**: `5m` ```yaml # Examples: interval: 0s # One-time sync only # --- interval: 5m # Sync every 5 minutes # --- interval: 30m # Sync every 30 minutes ``` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-max_parallel_http_requests)`schema_registry.max_parallel_http_requests` Maximum number of parallel HTTP requests to the schema registry. Controls concurrency when syncing multiple schemas. **Type**: `int` **Default**: `10` ### [](#schema_registry-normalize)`schema_registry.normalize` Whether to normalize schemas when creating them in the destination registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-strict)`schema_registry.strict` Error on unknown schema IDs. Only relevant when translate\_ids is true. When false (default), unknown schema IDs are passed through unchanged, allowing migration of topics with mixed message formats. Note: messages with 0-byte prefixes (e.g., protobuf) cannot be distinguished from schema registry headers and may fail when strict is enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-subject)`schema_registry.subject` Template for transforming subject names during migration. Use interpolation to rename subjects systematically. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: prod_${! metadata("schema_registry_subject") } # --- subject: ${! metadata("schema_registry_subject") | replace("dev_", "prod_") } ``` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-translate_ids)`schema_registry.translate_ids` Whether to translate schema IDs during migration. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#schema_registry-versions)`schema_registry.versions` Which schema versions to migrate. 'latest' migrates only the current version, 'all' migrates complete version history for better compatibility. **Type**: `string` **Default**: `all` **Options**: `latest`, `all` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#serverless)`serverless` Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by serverless environments. **Type**: `bool` **Default**: `false` ### [](#sync_topic_acls)`sync_topic_acls` Whether to synchronise topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts. **Type**: `bool` **Default**: `false` ### [](#sync_topic_interval)`sync_topic_interval` How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. Set to 0s to disable periodic sync (topics are still created on first message). **Type**: `string` **Default**: `5m` ```yaml # Examples: sync_topic_interval: 0s # Disable periodic sync # --- sync_topic_interval: 1m # Sync every minute # --- sync_topic_interval: 5m # Sync every 5 minutes ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${! @kafka_topic }` ```yaml # Examples: topic: prod_${! @kafka_topic } ``` ### [](#topic_replication_factor)`topic_replication_factor` The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes. **Type**: `int` ```yaml # Examples: topic_replication_factor: 3 # --- topic_replication_factor: 1 # For single-node clusters ``` --- # Page 354: redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda.md --- # redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/redpanda page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/redpanda.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2026-05-26" --- Sends message data to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. #### Common ```yml outputs: label: "" redpanda: seed_brokers: [] # No default (optional) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) inject_tracing_map: "" # No default (optional) partitioner: "" # No default (optional) idempotent_write: true acks: all compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: 0 max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s ``` ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted. When consuming data from a Redpanda input it is recommended to tune batches from the input config via the `max_yield_batch_bytes` field, or the `unordered_processing.batching` field if appropriate. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is not available. **Type**: `bool` **Default**: `true` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. **Type**: `int` **Default**: `10000` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `256` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. When `idempotent_write` is enabled, this is capped at 5 by the Kafka protocol (and at 1 for Kafka < v1.0.0). When `idempotent_write` is disabled, higher values improve throughput by pipelining requests but may cause out-of-order delivery. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum space (in bytes) that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. Optional when `seed_brokers` is configured in a top-level `redpanda` block. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 355: reject_errored **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/reject_errored.md --- # reject_errored > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: reject_errored latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/reject_errored page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/reject_errored.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/reject_errored.adoc description: Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output. ```yml # Config fields, showing default values output: label: "" reject_errored: null # No default (required) ``` The routing of messages rejected by this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages The most straight forward use case for this output type is to nack messages that have failed their processing steps. In this example our mapping might fail, in which case the messages that failed are rejected and will be nacked by our input: ```yaml input: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.pending pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: reject_errored: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.processed ``` ### [](#dlqing-failed-messages)DLQing Failed Messages Another use case for this output is to send failed messages straight into a dead-letter queue. You use it within a [fallback output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) that allows you to specify where these failed messages should go to next. ```yaml pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: fallback: - reject_errored: http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s ``` --- # Page 356: reject **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/reject.md --- # reject > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: reject latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/reject page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/reject.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/reject.adoc description: Rejects all messages, treating them as though the output destination failed to publish them. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Rejects all messages, treating them as though the output destination failed to publish them. ```yml # Config fields, showing default values output: label: "" reject: "" ``` The routing of messages after this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. To learn when this output could be useful, see \[the [Examples](#examples). ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages This input is particularly useful for routing messages that have failed during processing, where instead of routing them to some sort of dead letter queue we wish to push the error upstream. We can do this with a switch broker: ```yaml output: switch: retry_until_success: false cases: - check: '!errored()' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - output: reject: "processing failed due to: ${! error() }" ``` --- # Page 357: resource **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/resource.md --- # resource > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: resource latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/resource page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/resource.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/resource.adoc description: Resource is an output type that channels messages to a resource output, identified by its name. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Resource is an output type that channels messages to a resource output, identified by its name. ```yml # Config fields, showing default values output: resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml output: broker: pattern: fan_out outputs: - kafka: addresses: [ TODO ] topic: foo - gcp_pubsub: project: bar topic: baz ``` Could also be expressed as: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar output_resources: - label: foo kafka: addresses: [ TODO ] topic: foo - label: bar gcp_pubsub: project: bar topic: baz ``` --- # Page 358: retry **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/retry.md --- # retry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: retry latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/retry page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/retry.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/retry.adoc description: Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or max elapsed time fields are non-zero, either is reached. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or max elapsed time fields are non-zero, either is reached. #### Common ```yml outputs: label: "" retry: output: "" # No default (required) ``` #### Advanced ```yml outputs: label: "" retry: max_retries: 0 backoff: initial_interval: 500ms max_interval: 3s max_elapsed_time: 0s output: "" # No default (required) ``` All messages in Redpanda Connect are always retried on an output error, but this would usually involve propagating the error back to the source of the message, whereby it would be reprocessed before reaching the output layer once again. This output type is useful whenever we wish to avoid reprocessing a message on the event of a failed send. We might, for example, have a deduplication processor that we want to avoid reapplying to the same message more than once in the pipeline. Rather than retrying the same output you may wish to retry the send using a different output target (a dead letter queue). In which case you should instead use the [`fallback`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) output type. ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `0s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `3s` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#output)`output` A child output. **Type**: `output` --- # Page 359: salesforce_sink **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/salesforce_sink.md --- # salesforce_sink > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: salesforce_sink latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/salesforce_sink page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/salesforce_sink.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/salesforce_sink.adoc description: Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. page-git-created-date: "2026-05-01" page-git-modified-date: "2026-08-11" --- Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. Consumes batches of messages and writes them to Salesforce. Each message must have a `topic` field (set by the per-topic processor) and a `data` field containing the Salesforce record fields. The `topic` is used to look up the correct `topic_mappings` entry which defines the sObject, operation, and write mode. **Realtime mode** uses the sObject Collections REST API (synchronous, up to 200 records/call). **Bulk mode** uses the Bulk API 2.0 (asynchronous, polls until complete). #### Common ```yml outputs: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) ``` #### Advanced ```yml outputs: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) api_version: v65.0 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) http: timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` ## [](#fields)Fields ### [](#api_version)`api_version` Salesforce REST API version to target, prefixed with `v`. Affects endpoint paths (`/services/data/{api_version}/…​`) and available fields/objects. Must be supported by your org — check Setup → Company Information. Older versions may lack recent fields. **Type**: `string` **Default**: `v65.0` ```yaml # Examples: api_version: v65.0 # --- api_version: v62.0 ``` ### [](#batch_period)`batch_period` Maximum period to wait before flushing an incomplete batch. **Type**: `string` **Default**: `5s` ### [](#bulk_batch_size)`bulk_batch_size` Number of records per bulk job. Also controls the output batch size. **Type**: `int` **Default**: `1000` ### [](#bulk_poll_interval)`bulk_poll_interval` How often to poll Salesforce for bulk job completion status. **Type**: `string` **Default**: `5s` ### [](#client_id)`client_id` Client ID for the Salesforce Connected App. **Type**: `string` ### [](#client_secret)`client_secret` Client secret for the Salesforce Connected App. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#http)`http` HTTP client configuration for Salesforce REST calls (OAuth token endpoint and, where applicable, data queries). **Type**: `object` ### [](#http-access_log_body_limit)`http.access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#http-access_log_level)`http.access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#http-backoff)`http.backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#http-backoff-initial_interval)`http.backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#http-backoff-max_interval)`http.backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#http-backoff-max_retries)`http.backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#http-disable_http2)`http.disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http-http)`http.http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-http-disable_compression)`http.http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-http-disable_keep_alives)`http.http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-expect_continue_timeout)`http.http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-http-h2)`http.http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-http-h2-max_decoder_header_table_size)`http.http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_encoder_header_table_size)`http.http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-http-h2-max_read_frame_size)`http.http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-http-h2-max_receive_buffer_per_connection)`http.http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-max_receive_buffer_per_stream)`http.http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-http-h2-ping_timeout)`http.http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-http-h2-send_ping_timeout)`http.http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-http-h2-strict_max_concurrent_requests)`http.http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-http-h2-write_byte_timeout)`http.http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-idle_conn_timeout)`http.http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-http-max_conns_per_host)`http.http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-http-max_idle_conns)`http.http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-http-max_idle_conns_per_host)`http.http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-http-max_response_body_bytes)`http.http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-http-max_response_header_bytes)`http.http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-http-read_buffer_size)`http.http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-http-response_header_timeout)`http.http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-http-tls_handshake_timeout)`http.http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-http-write_buffer_size)`http.http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#http-proxy_url)`http.proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#http-tcp)`http.tcp` TCP socket configuration. **Type**: `object` ### [](#http-tcp-connect_timeout)`http.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-tcp-keep_alive)`http.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#http-tcp-keep_alive-count)`http.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#http-tcp-keep_alive-idle)`http.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#http-tcp-keep_alive-interval)`http.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#http-tcp-tcp_user_timeout)`http.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#http-timeout)`http.timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#http-tls)`http.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#http-tls-client_certs)`http.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#http-tls-client_certs-cert)`http.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-cert_file)`http.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key)`http.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-key_file)`http.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#http-tls-client_certs-password)`http.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#http-tls-enable_renegotiation)`http.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#http-tls-enabled)`http.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#http-tls-root_cas)`http.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#http-tls-root_cas_file)`http.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#http-tls-skip_cert_verify)`http.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#http-tps_burst)`http.tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#http-tps_limit)`http.tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#max_concurrent_bulk_jobs)`max_concurrent_bulk_jobs` Maximum number of bulk jobs polling concurrently in the background. Each in-flight job buffers its CSV payload in memory. Lower this value if memory usage is a concern. **Type**: `int` **Default**: `10` ### [](#max_in_flight)`max_in_flight` Maximum number of batches to send concurrently. Increasing this value improves real-time write throughput. **Type**: `int` **Default**: `1` ### [](#org_url)`org_url` Salesforce instance base URL (for example, [https://your-domain.salesforce.com](https://your-domain.salesforce.com)). **Type**: `string` ```yaml # Examples: org_url: https://acme.my.salesforce.com # --- org_url: https://acme--staging.sandbox.my.salesforce.com ``` ### [](#topic_mappings)`topic_mappings[]` Per-topic Salesforce write configuration. Each entry maps a Kafka topic to an sObject and write settings. **Type**: `array` ### [](#topic_mappings-all_or_none)`topic_mappings[].all_or_none` Real-time only: rolls back the entire batch if any record fails. **Type**: `bool` **Default**: `false` ### [](#topic_mappings-external_id_field)`topic_mappings[].external_id_field` External ID field name. Required for upsert operations. **Type**: `string` **Default**: `""` ### [](#topic_mappings-mode)`topic_mappings[].mode` Write mode: `realtime` (sObject Collections API) or `bulk` (Bulk API 2.0). **Type**: `string` **Default**: `realtime` ### [](#topic_mappings-operation)`topic_mappings[].operation` Write operation: insert, update, upsert, or delete. **Type**: `string` **Default**: `upsert` ### [](#topic_mappings-sobject)`topic_mappings[].sobject` Salesforce sObject API name (for example, Account, Contact, MyObject\_\_c). **Type**: `string` ### [](#topic_mappings-topic)`topic_mappings[].topic` Kafka topic name to match against the message’s `topic` field. **Type**: `string` --- # Page 360: schema_registry **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/schema_registry.md --- # schema_registry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: schema_registry latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/schema_registry page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/schema_registry.adoc description: Publishes schemas to SchemaRegistry. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Publishes schemas to a schema registry. This output uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml outputs: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) subject_compatibility_level: "" # No default (optional) backfill_dependencies: true translate_ids: false normalize: true remove_metadata: true remove_rule_set: true input_resource: schema_registry_input tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#performance)Performance The `schema_registry` output sends multiple messages in parallel for improved performance. You can use the `max_in_flight` field to tune the maximum number of in-flight messages, or message batches. ## [](#example)Example This example writes schemas to a schema registry instance and logs errors for existing schemas. ```yaml output: fallback: - schema_registry: url: http://localhost:8082 subject: ${! @schema_registry_subject } - switch: cases: - check: '@fallback_error == "request returned status: 422"' output: drop: {} processors: - log: message: | Subject '${! @schema_registry_subject }' version ${! @schema_registry_version } already has schema: ${! content() } - output: reject: ${! @fallback_error } ``` ## [](#fields)Fields ### [](#backfill_dependencies)`backfill_dependencies` Backfill missing schema references and previous schema versions. If set to `true`, you must also configure a [`schema_registry`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/schema_registry/) input to read source schemas. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#input_resource)`input_resource` The label of the [`schema_registry` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/schema_registry/) from which to read source schemas. **Type**: `string` **Default**: `schema_registry_input` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication for secure data transmission from this component to your schema registry. This feature is in beta and may change in future releases. **Type**: `object` ### [](#jwt-claims)`jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A PEM-encoded file containing a private key that is formatted using either PKCS1 or PKCS8 standards. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The method used to sign the token, such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#normalize)`normalize` Normalize schemas. **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value this component can use to gain access to the schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#remove_metadata)`remove_metadata` Removes metadata fields from schema output. Use this to produce leaner schema definitions for downstream consumers or when metadata is not required. **Type**: `bool` **Default**: `true` ### [](#remove_rule_set)`remove_rule_set` Removes rule set definitions from schema output. Useful for simplifying schemas when rule sets are not required by consumers or applications. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` The subject name. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#subject_compatibility_level)`subject_compatibility_level` The compatibility level for the subject. Can be one of `BACKWARD`, `BACKWARD_TRANSITIVE`, `FORWARD`, `FORWARD_TRANSITIVE`, `FULL`, `FULL_TRANSITIVE`, `NONE`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#translate_ids)`translate_ids` When set to `true`, this field automatically translates the schema ID in each message to match the corresponding schema in the destination schema registry. The updated message is then written to the destination schema registry. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 361: sftp **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sftp.md --- # sftp > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sftp latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/sftp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/sftp.adoc description: Writes files to an SFTP server. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Writes files to an SFTP server. #### Common ```yml outputs: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` In order to have a different path for each object you should use function interpolations described [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. **Type**: `string` **Default**: `all-bytes` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | delim:x | Append each message to the output stream followed by a custom delimiter. | | lines | Append each message to the output stream followed by a line break. | ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The private key used to authenticate with the SFTP server. This field provides an alternative to the [`private_key_file`](#credentials-private_key_file). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the [`private_key`](#credentials-private_key) field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` A passphrase for private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The file to save the messages to on the SFTP server. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 362: slack_post **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/slack_post.md --- # slack_post > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: slack_post latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/slack_post page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/slack_post.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/slack_post.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Posts a new message to a Slack channel using the Slack API method [chat.postMessage](https://api.slack.com/methods/chat.postMessage). ```yml # Common configuration fields, showing default values output: label: "" slack_post: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (optional) text: "" # No default (optional) blocks: "" # No default (optional) markdown: true unfurl_links: false unfurl_media: true link_names: 0 ``` See also: [Examples](#examples) ## [](#fields)Fields ### [](#blocks)`blocks` A Bloblang query that should return a JSON array of [Slack blocks](https://api.slack.com/reference/block-kit/blocks). You can either specify message content in the `text` or `blocks` fields, but not both. **Type**: `string` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to post messages to the target Slack channel. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the target Slack channel. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#link_names)`link_names` When set to `1`, this output finds and links to [user groups](https://api.slack.com/reference/surfaces/formatting#mentioning-groups) mentioned in Slack messages. **Type**: `bool` **Default**: `false` ### [](#markdown)`markdown` When set to `true`, this output accepts message content in Markdown format. **Type**: `bool` **Default**: `true` ### [](#text)`text` The text content of the message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). You can either specify message content in the `text` or `blocks` fields, but not both. **Type**: `string` **Default**: `""` ### [](#thread_ts)`thread_ts` Specify the thread timestamp (`ts` value) of another message to post a reply within the same thread. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#unfurl_links)`unfurl_links` When set to `true`, this output provides previews of linked content in Slack messages. For more information about unfurling links, see the [Slack documentation](https://api.slack.com/reference/messaging/link-unfurling). **Type**: `bool` **Default**: `false` ### [](#unfurl_media)`unfurl_media` When set to `true`, this output provides previews of rich content in Slack messages, such as videos or embedded tweets. **Type**: `bool` **Default**: `true` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 363: slack_reaction **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/slack_reaction.md --- # slack_reaction > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: slack_reaction latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/slack_reaction page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/slack_reaction.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/slack_reaction.adoc description: Add or remove an emoji reaction to a Slack message. page-git-created-date: "2025-07-08" page-git-modified-date: "2026-08-11" --- Add or remove an emoji reaction to a Slack message using [`reactions.add`](https://api.slack.com/methods/reactions.add) and [`reactions.remove`](https://api.slack.com/methods/reactions.remove). ```yaml output: label: "" slack_reaction: bot_token: "" # No default (required) channel_id: "" # No default (required) timestamp: "" # No default (required) emoji: "" # No default (required) action: add max_in_flight: 64 ``` ## [](#fields)Fields ### [](#action)`action` Whether to add or remove the reaction. When set to `add`, the specified emoji reaction is applied to the target message. When set to `remove`, the emoji reaction is removed from the target message. **Type**: `string` **Default**: `add` **Options**: `add`, `remove` ### [](#bot_token)`bot_token` Your Slack Bot User OAuth token used to authenticate the API request. This token must have the necessary `reactions:write` and `channels:read` (or related) scopes. It typically begins with `xoxb-`. **Type**: `string` ### [](#channel_id)`channel_id` The unique Slack channel ID where the target message resides. Channel IDs usually start with `C` for public channels or `G` for private channels. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#emoji)`emoji` The name of the emoji to be added or removed, without surrounding colons. Use the plain emoji name, such as `thumbsup` or `tada`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increasing this value can improve throughput in high-volume scenarios, but be cautious not to exceed Slack’s API rate limits. **Type**: `int` **Default**: `64` ### [](#timestamp)`timestamp` The timestamp of the message to react to. This is a unique identifier for the message, usually obtained from a previous Slack API call (such as `chat.postMessage` or `conversations.history`). It typically looks like a Unix timestamp with a decimal. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 364: snowflake_put **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/snowflake_put.md --- # snowflake_put > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: snowflake_put latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/snowflake_put page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/snowflake_put.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/snowflake_put.adoc description: Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > 💡 **TIP** > > Use the [`snowflake_streaming` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/snowflake_streaming/) for improved performance, cost-effectiveness, and ease of use. Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables. #### Common ```yml outputs: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" compression: AUTO request_id: "" snowpipe: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 1 ``` #### Advanced ```yml outputs: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" upload_parallel_threads: 4 compression: AUTO request_id: "" snowpipe: "" # No default (optional) client_session_keep_alive: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 1 ``` In order to use a different stage and / or Snowpipe for each message, you can use function interpolations as described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). When using batching, messages are grouped by the calculated stage and Snowpipe and are streamed to individual files in their corresponding stage and, optionally, a Snowpipe `insertFiles` REST API call will be made for each individual file. ## [](#credentials)Credentials Two authentication mechanisms are supported: - User/password - Key Pair Authentication ### [](#userpassword)User/password This is a basic authentication mechanism which allows you to PUT data into a stage. However, it is not compatible with Snowpipe. ### [](#key-pair-authentication)Key pair authentication This authentication mechanism allows Snowpipe functionality, but it does require configuring an SSH Private Key beforehand. Please consult the [documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) for details on how to set it up and assign the Public Key to your user. Note that the Snowflake documentation [used to suggest](https://twitter.com/felipehoffa/status/1560811785606684672) using this command: ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 ``` to generate an encrypted SSH private key. However, in this case, it uses an encryption algorithm called `pbeWithMD5AndDES-CBC`, which is part of the PKCS#5 v1.5 and is considered insecure. Due to this, Redpanda Connect does not support it and, if you wish to use password-protected keys directly, you must use PKCS#5 v2.0 to encrypt them by using the following command (as the current Snowflake docs suggest): ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8 ``` If you have an existing key encrypted with PKCS#5 v1.5, you can re-encrypt it with PKCS#5 v2.0 using this command: ```bash openssl pkcs8 -in rsa_key_original.p8 -topk8 -v2 des3 -out rsa_key.p8 ``` Please consult the [pkcs8 command documentation](https://linux.die.net/man/1/pkcs8) for details on PKCS#5 algorithms. ## [](#batching)Batching It’s common to want to upload messages to Snowflake as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) and/or [`compress`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/compress/) processor. For the optimal batch size, please consult the Snowflake [documentation](https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare.html). ## [](#snowpipe)Snowpipe Given a table called `BENTHOS_TBL` with one column of type `variant`: ```sql CREATE OR REPLACE TABLE BENTHOS_DB.PUBLIC.BENTHOS_TBL(RECORD variant) ``` and the following `BENTHOS_PIPE` Snowpipe: ```sql CREATE OR REPLACE PIPE BENTHOS_DB.PUBLIC.BENTHOS_PIPE AUTO_INGEST = FALSE AS COPY INTO BENTHOS_DB.PUBLIC.BENTHOS_TBL FROM (SELECT * FROM @%BENTHOS_TBL) FILE_FORMAT = (TYPE = JSON COMPRESSION = AUTO) ``` you can configure Redpanda Connect to use the implicit table stage `@%BENTHOS_TBL` as the `stage` and `BENTHOS_PIPE` as the `snowpipe`. In this case, you must set `compression` to `AUTO` and, if using message batching, you’ll need to configure an [`archive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) processor with the `concatenate` format. Since the `compression` is set to `AUTO`, the [gosnowflake](https://github.com/snowflakedb/gosnowflake) client library will compress the messages automatically so you don’t need to add a [`compress`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/compress/) processor for message batches. If you add `STRIP_OUTER_ARRAY = TRUE` in your Snowpipe `FILE_FORMAT` definition, then you must use `json_array` instead of `concatenate` as the archive processor format. > 📝 **NOTE** > > Only Snowpipes with `FILE_FORMAT` `TYPE` `JSON` are currently supported. ## [](#snowpipe-troubleshooting)Snowpipe troubleshooting Snowpipe [provides](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html) the `insertReport` and `loadHistoryScan` REST API endpoints which can be used to get information about recent Snowpipe calls. In order to query them, you’ll first need to generate a valid JWT token for your Snowflake account. There are two methods for doing so: - Using the `snowsql` [utility](https://docs.snowflake.com/en/user-guide/snowsql.html): ```bash snowsql --private-key-path rsa_key.p8 --generate-jwt -a -u ``` - Using the Python `sql-api-generate-jwt` [utility](https://docs.snowflake.com/en/developer-guide/sql-api/authenticating.html#generating-a-jwt-in-python): ```bash python3 sql-api-generate-jwt.py --private_key_file_path=rsa_key.p8 --account= --user= ``` Once you successfully generate a JWT token and store it into the `JWT_TOKEN` environment variable, then you can, for example, query the `insertReport` endpoint using `curl`: ```bash curl -H "Authorization: Bearer ${JWT_TOKEN}" "https://.snowflakecomputing.com/v1/data/pipes/../insertReport" ``` If you need to pass in a valid `requestId` to any of these Snowpipe REST API endpoints, you can set a [uuid\_v4()](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#uuid_v4) string in a metadata field called `request_id`, log it via the [`log`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/log/) processor and then configure `request_id: ${ @request_id }` ). Alternatively, you can [enable debug logging](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/logger/about/) and Redpanda Connect will print the Request IDs that it sends to Snowpipe. ## [](#general-troubleshooting)General troubleshooting The underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) requires write access to the default directory to use for temporary files. Please consult the [`os.TempDir`](https://pkg.go.dev/os#TempDir) docs for details on how to change this directory via environment variables. A silent failure can occur due to [this issue](https://github.com/snowflakedb/gosnowflake/issues/701), where the underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) doesn’t return an error and doesn’t log a failure if it can’t figure out the current username. One way to trigger this behavior is by running Redpanda Connect in a Docker container with a non-existent user ID (such as `--user 1000:1000`). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#examples)Examples ### [](#kafka-realtime-brokers)Kafka / realtime brokers Upload message batches from realtime brokers such as Kafka persisting the batch partition and offsets in the stage path and filename similarly to the [Kafka Connector scheme](https://docs.snowflake.com/en/user-guide/kafka-connector-ts.html#step-1-view-the-copy-history-for-the-table) and call Snowpipe to load them into a table. When batching is configured at the input level, it is done per-partition. ```yaml input: redpanda: seed_brokers: - localhost:9092 topics: - foo consumer_group: rpcn max_yield_batch_bytes: 8MB processors: - mapping: | meta kafka_start_offset = meta("kafka_offset").from(0) meta kafka_end_offset = meta("kafka_offset").from(-1) meta batch_timestamp = if batch_index() == 0 { now() } - mapping: | meta batch_timestamp = if batch_index() != 0 { meta("batch_timestamp").from(0) } output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos/BENTHOS_TBL/${! @kafka_partition } file_name: ${! @kafka_start_offset }_${! @kafka_end_offset }_${! meta("batch_timestamp") } upload_parallel_threads: 4 compression: NONE snowpipe: BENTHOS_PIPE ``` ### [](#no-compression)No compression Upload concatenated messages into a `.json` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#parquet-format-with-snappy-compression)Parquet format with snappy compression Upload concatenated messages into a `.parquet` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos file_extension: parquet upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - parquet_encode: schema: - name: ID type: INT64 - name: CONTENT type: BYTE_ARRAY default_compression: snappy ``` ### [](#automatic-compression)Automatic compression Upload concatenated messages compressed automatically into a `.gz` archive file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: AUTO batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#deflate-compression)DEFLATE compression Upload concatenated messages compressed into a `.deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("zlib") ``` ### [](#raw_deflate-compression)RAW_DEFLATE compression Upload concatenated messages compressed into a `.raw_deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: RAW_DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("flate") ``` ## [](#fields)Fields ### [](#account)`account` Account name, which is the same as the [Account Identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#where-are-account-identifiers-used). However, when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier), the Account Identifier is formatted as `..` and this field needs to be populated using the `` part. **Type**: `string` ### [](#batching-2)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_session_keep_alive)`client_session_keep_alive` Enable Snowflake keepalive mechanism to prevent the client session from expiring after 4 hours (error 390114). **Type**: `bool` **Default**: `false` ### [](#cloud)`cloud` Optional cloud platform field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: cloud: aws # --- cloud: gcp # --- cloud: azure ``` ### [](#compression)`compression` Compression type. **Type**: `string` **Default**: `AUTO` | Option | Summary | | --- | --- | | AUTO | Compression (gzip) is applied automatically by the output and messages must contain plain-text JSON. Default file_extension: gz. | | DEFLATE | Messages must be pre-compressed using the zlib algorithm (with zlib header, RFC1950). Default file_extension: deflate. | | GZIP | Messages must be pre-compressed using the gzip algorithm. Default file_extension: gz. | | NONE | No compression is applied and messages must contain plain-text JSON. Default file_extension: json. | | RAW_DEFLATE | Messages must be pre-compressed using the flate algorithm (without header, RFC1951). Default file_extension: raw_deflate. | | ZSTD | Messages must be pre-compressed using the Zstandard algorithm. Default file_extension: zst. | ### [](#database)`database` Database. **Type**: `string` ### [](#file_extension)`file_extension` Stage file extension. Will be derived from the configured `compression` if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: file_extension: csv # --- file_extension: parquet ``` ### [](#file_name)`file_name` Stage file name. Will be equal to the Request ID if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `1` ### [](#password)`password` An optional password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#path)`path` Stage path. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#private_key)`private_key` Your private SSH key. When using encrypted keys, you must also set a value for [`private_key_pass`](#private_key_pass). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#private_key_file)`private_key_file` The path to a file containing your private SSH key. When using encrypted keys, you must also set a value for [`private_key_pass`](#private_key_pass). **Type**: `string` ### [](#private_key_pass)`private_key_pass` The passphrase for your private SSH key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#region)`region` Optional region field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#request_id)`request_id` Request ID. Will be assigned a random UUID (v4) string if not set or empty. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#role)`role` Role. **Type**: `string` ### [](#schema)`schema` Schema. **Type**: `string` ### [](#snowpipe-2)`snowpipe` An optional Snowpipe name. Use the `` part from `..`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#stage)`stage` Stage name. Use either one of the [supported](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html) stage types. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#upload_parallel_threads)`upload_parallel_threads` Specifies the number of threads to use for uploading files. **Type**: `int` **Default**: `4` ### [](#user)`user` Username. **Type**: `string` ### [](#warehouse)`warehouse` Warehouse. **Type**: `string` --- # Page 365: snowflake_streaming **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/snowflake_streaming.md --- # snowflake_streaming > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: snowflake_streaming latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/snowflake_streaming page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/snowflake_streaming.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/snowflake_streaming.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2026-05-26" --- Allows Snowflake to ingest data from your data pipeline using [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview). To help you configure your own `snowflake_streaming` output, this page includes [example data pipelines](#example-pipelines). ### Common ```yml outputs: label: "" snowflake_streaming: account: "" # No default (required) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: "" # No default (required) ignore_nulls: true processors: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ### Advanced ```yml outputs: label: "" snowflake_streaming: account: "" # No default (required) url: "" # No default (optional) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: "" # No default (required) ignore_nulls: true processors: [] # No default (optional) build_options: parallelism: 1 chunk_size: 50000 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 channel_prefix: "" # No default (optional) channel_name: "" # No default (optional) offset_token: "" # No default (optional) commit_backoff: initial_interval: 32ms max_interval: 512ms max_elapsed_time: 60s multiplier: 2 message_format: object timestamp_format: 2006-01-02T15:04:05.999999999Z07:00 ``` ## [](#conversion-of-message-data-into-snowflake-table-rows)Conversion of message data into Snowflake table rows Message data conversion to Snowflake table rows is determined by the: - Output message contents. - [Schema evolution settings](#schema_evolution). - Schema of the [target Snowflake table](#table). The following scenarios highlight how these three factors affect data written to the target table. > 📝 **NOTE** > > For reduced complexity, consider [turning on schema evolution](#schema_evolution), which automatically creates and updates the Snowflake table schema based on message contents. ### [](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off)Scenario: Data and table schema match (schema evolution turned on or off) An output message matches the existing table schema, and the `schema_evolution.enabled` field is set to `true` or `false`. The target Snowflake table has two columns: - `product_id` (NUMBER) - `product_code` (STRING) A pipeline generates the following message: ```json {"product_id": 521, "product_code": “EST-PR”} ``` In this scenario: - The JSON keys in the message (`"product_id"` and `"product_code"`) match column names in the target Snowflake table. - The message values match the column data types. (If there was a data mismatch, the message would be rejected.) - Redpanda Connect inserts the message values into a new row in the target Snowflake table. | product_id | product_code | | --- | --- | | 521 | EST-PR | ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on)Scenario: Data and table schema mismatch (schema evolution turned on) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `true`. The target Snowflake table has the same two columns as the [previous scenario](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off): - `product_id` (NUMBER) - `product_code` (STRING) This time, the pipeline generates the following message: ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match column names in the target Snowflake table. - As schema evolution is enabled, Redpanda Connect adds two new columns to the target table with data types derived from the output message values. For more information about the mapping of data types, see [Supported data formats for Snowflake columns](#supported-data-formats-for-snowflake-columns). - Redpanda Connect inserts the message values into a new table row. | product_id | product_code | product_batch | product_color | | --- | --- | --- | --- | | (null) | (null) | 11111 | yellow | > 📝 **NOTE** > > You can [configure processors](#schema_evolution-processors) to override the schema updates derived from the message values. ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-off)Scenario: Data and table schema mismatch (schema evolution turned off) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `false`. The target Snowflake table has the same two columns: - `product_id` (NUMBER) - `product_code` (STRING) The pipeline generates the same message as the [previous scenario](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on): ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match any existing column names. - Because schema evolution is turned off, Redpanda Connect ignores the extra column names and values and inserts a row of null values. | product_id | product_code | | --- | --- | | (null) | (null) | ## [](#supported-data-formats-for-snowflake-columns)Supported data formats for Snowflake columns The message data from your output must match the columns in the Snowflake table that you want to write data to. The following table shows you the [column data types supported by Snowflake](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types) and how they correspond to the [Bloblang data types](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#type) in Redpanda Connect. | Snowflake column data type | Bloblang data types | | --- | --- | | CHAR, VARCHAR | string | | BINARY | string or bytes | | NUMBER | number, or string where the string is parsed into a number | | FLOAT, including special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity) | number | | BOOLEAN | bool, or number where a non-zero number is true | | TIME, DATE, TIMESTAMP | timestamp, or number where the number is a converted to a Unix timestamp, or string where the string is parsed using RFC 3339 format | | VARIANT, ARRAY, OBJECT | Any data type converted into JSON | | GEOGRAPHY,GEOMETRY | Not supported | ## [](#authentication)Authentication You can authenticate with Snowflake using an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth). Either specify: - A PEM-encoded private key, in the [`private_key` field](#private_key). - The path to a file from which the output can load the private RSA key, in the [`private_key_file` field](#private_key_file). ## [](#performance)Performance For improved performance, this output: - Sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. - Sends messages as a batch. You can configure batches at both the input and output level. For more information, see [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ### [](#batch-sizes)Batch sizes Redpanda recommends that every message batch writes at least 16 MiB of compressed output to Snowflake. You can monitor batch sizes using the `snowflake_compressed_output_size_bytes` metric. ### [](#metrics)Metrics This output emits the following metrics. | Metric name | Description | | --- | --- | | snowflake_compressed_output_size_bytes | The size in bytes of each message batch uploaded to Snowflake. | | snowflake_convert_latency_ns | The time taken to convert messages into the Snowflake column data types. | | snowflake_serialize_latency_ns | The time taken to serialize the converted columnar data into a file for upload to Snowflake. | | snowflake_build_output_latency_ns | The time taken to build the file that is uploaded to Snowflake. This metric is the sum of snowflake_convert_latency_ns + snowflake_serialize_latency_ns. | | snowflake_upload_latency_ns | The time taken to upload the output file to Snowflake. | | snowflake_register_latency_ns | The time taken to register the uploaded output file with Snowflake. | | snowflake_commit_latency_ns | The time taken to commit the uploaded data updates to the target Snowflake table. | ## [](#fields)Fields ### [](#account)`account` The [Snowflake account name to use](https://docs.snowflake.com/en/user-guide/admin-account-identifier#account-name). Use the format `-` where: - The `` is the name of your Snowflake organization. - The `` is the unique name of your account with your Snowflake organization. To find the correct value for this field, run the following query in Snowflake: ```sql WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` **Type**: `string` ```yaml # Examples: account: ORG-ACCOUNT ``` ### [](#batching)`batching` Lets you configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). Type\*: `object` ```yml # Examples batching: byte_size: 5000 count: 0 period: 1s batching: count: 10 period: 1s batching: check: this.contains("END BATCH") count: 0 period: 1m ``` **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#build_options)`build_options` Options for optimizing the build of the output data that is sent to Snowflake. Monitor the `snowflake_build_output_latency_ns` metric to assess whether you need to update these options. **Type**: `object` ### [](#build_options-chunk_size)`build_options.chunk_size` The number of table rows to submit in each chunk for processing. **Type**: `int` **Default**: `50000` ### [](#build_options-parallelism)`build_options.parallelism` The maximum amount of parallel processing to use when building the output for Snowflake. **Type**: `int` **Default**: `1` ### [](#channel_name)`channel_name` The channel name to use when connecting to a Snowflake table. Duplicate channel names cause errors and prevent multiple instances of Redpanda Connect from writing at the same time, and so this field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). Redpanda Connect assumes that a message batch contains messages for a single channel, which means that interpolation is only executed on the first message in each batch. If your pipeline uses an input that is partitioned, such as an Apache Kafka topic, batch messages at the input level to make sure all messages are processed by the same channel. You can specify either the `channel_name` or `channel_prefix`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). **Type**: `string` ```yaml # Examples: channel_name: partition-${!@kafka_partition} ``` ### [](#channel_prefix)`channel_prefix` The prefix to use when creating a channel name for connecting to a Snowflake table. Adding a `channel_prefix` avoids the creation of duplicate channel names, which result in errors and prevent multiple instances of Redpanda Connect from writing at the same time. You can specify either the `channel_prefix` or `channel_name`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. The maximum number of channels open at any time is determined by the value in the `max_in_flight` field. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). **Type**: `string` ```yaml # Examples: channel_prefix: channel-${HOST} ``` ### [](#commit_backoff)`commit_backoff` Control how frequently Snowflake is polled to check if data has been committed. **Type**: `object` ### [](#commit_backoff-initial_interval)`commit_backoff.initial_interval` The initial period to wait between status polls. **Type**: `string` **Default**: `32ms` ### [](#commit_backoff-max_elapsed_time)`commit_backoff.max_elapsed_time` The maximum total time to wait for data to be committed. If zero then no limit is used. **Type**: `string` **Default**: `60s` ### [](#commit_backoff-max_interval)`commit_backoff.max_interval` The maximum period to wait between status polls. **Type**: `string` **Default**: `512ms` ### [](#commit_backoff-multiplier)`commit_backoff.multiplier` The factor by which the poll interval grows on each attempt. **Type**: `float` **Default**: `2` ### [](#database)`database` The Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: database: MY_DATABASE ``` ### [](#init_statement)`init_statement` Optional SQL statements to execute immediately after this output connects to Snowflake for the first time. This is a useful way to initialize tables before processing data. > 📝 **NOTE** > > Make sure your SQL statements are idempotent, so they do not cause issues when run multiple times after service restarts. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS mytable (amount NUMBER); # --- init_statement: |- ALTER TABLE t1 ALTER COLUMN c1 DROP NOT NULL; ALTER TABLE t1 ADD COLUMN a2 NUMBER; ``` ### [](#mapping)`mapping` The [Bloblang `mapping`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) to execute on each message. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `4` ### [](#message_format)`message_format` The format to expect incoming messages from the rest of the pipeline. **Type**: `string` **Default**: `object` | Option | Summary | | --- | --- | | array | Messages are an array of values where each position matches the ordinal of the column in Snowflake. | | object | Messages are JSON or Bloblang objects where each key is the Snowflake column name and the value is the column value. | ```yaml # Examples: message_format: array ``` ### [](#offset_token)`offset_token` The offset token to use for exactly-once delivery of data to a Snowflake table. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). This output assumes that messages within a batch are in increasing order by offset token. When data is sent on a channel, the offset token of each message in the batch is compared to the latest token processed by the channel. If the offset token is lexicographically less than the latest token, it’s assumed the message is a duplicate and is dropped. Messages must be delivered to the output in order, otherwise they are processed as duplicates and dropped. To avoid dropping retried messages if later messages have succeeded in the meantime, use a dead-letter queue to process failed messages. See the [Ingesting data exactly once from Redpanda](#example-pipelines) example. > 📝 **NOTE** > > If you’re using a numeric value as an offset token, pad the value so that it’s lexicographically ordered in its string representation because offset tokens are compared in string form. For more details, see the [Ingesting data exactly once from Redpanda](#example-pipelines) example. For more information about offset tokens, see [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens). **Type**: `string` ```yaml # Examples: offset_token: offset-${!"%016X".format(@kafka_offset)} # --- offset_token: postgres-${!@lsn} ``` ### [](#private_key)`private_key` The PEM-encoded private RSA key to use for authentication with Snowflake. You must specify a value for this field or the `private_key_file` field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#private_key_file)`private_key_file` A `.p8`, PEM-encoded file to load the private RSA key from. You must specify a value for this field or the `private_key` field. **Type**: `string` ### [](#private_key_pass)`private_key_pass` If the RSA key is encrypted, specify the RSA key passphrase. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#role)`role` The role of the user specified in the `user` field. The user’s role must have the [required privileges](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#required-access-privileges) to call the Snowpipe Streaming APIs. For more information about user roles, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management#user-roles). **Type**: `string` ```yaml # Examples: role: ACCOUNTADMIN ``` ### [](#schema)`schema` The schema of the Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: schema: PUBLIC ``` ### [](#schema_evolution)`schema_evolution` Options to control schema updates when messages are written to the Snowflake table. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Whether schema evolution is enabled. When set to `true`, the Snowflake table is automatically created based on the schema of the first message written to it, if the table does not already exist. As new fields are added to subsequent messages in the pipeline, new columns are created in the Snowflake table. Any required columns are marked as `nullable` if new messages do not include data for them. **Type**: `bool` ### [](#schema_evolution-ignore_nulls)`schema_evolution.ignore_nulls` When set to `true` and schema evolution is enabled, new columns that have `null` values _are not_ added to the Snowflake table. This behavior: - Prevents unnecessary schema changes caused by placeholder or incomplete data. - Avoids creating table columns with incorrect data types. > 📝 **NOTE** > > Redpanda does not recommend updating the default setting unless you are confident about the data type of `null` columns in advance. **Type**: `bool` **Default**: `true` ### [](#schema_evolution-processors)`schema_evolution.processors[]` A series of processors to execute when new columns are added to the Snowflake table. You can use these processors to: - Run side effects when the schema evolves. - Enrich the message with additional information to guide the schema changes. For example, a processor could read the schema from the schema registry that a message was produced with and use that schema to determine the data type of the new column in Snowflake. The input to these processors is an object with the value and name of the new column, the original message, and details of the Snowflake table the output writes to. For example: `{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "db_field_name"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}` The output from the processors must be a valid message, which contains a string that specifies the column type for the new column in Snowflake. The metadata remains the same as in the original message that triggered the schema update. **Type**: `array` ```yaml # Examples: processors: - mapping: |- root = match this.value.type() { this == "string" => "STRING" this == "bytes" => "BINARY" this == "number" => "DOUBLE" this == "bool" => "BOOLEAN" this == "timestamp" => "TIMESTAMP" _ => "VARIANT" } ``` ### [](#table)`table` The Snowflake table you want to write data to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table: MY_TABLE ``` ### [](#timestamp_format)`timestamp_format` The format to parse string values for `TIMESTAMP`, `TIMESTAMP_LTZ` and `TIMESTAMP_NTZ` columns. Should be a layout for [time.Parse](https://pkg.go.dev/time#Parse) in Go. **Type**: `string` **Default**: `2006-01-02T15:04:05.999999999Z07:00` ### [](#url)`url` Specify a custom URL to connect to Snowflake. This parameter overrides the default URL, which is automatically generated from the value of `output.snowflake_streaming.account`. By default, the URL is constructed as follows: `[https://.snowflakecomputing.com](https://\.snowflakecomputing.com)`. **Type**: `string` ```yaml # Examples: url: https://org-account.privatelink.snowflakecomputing.com ``` ### [](#user)`user` Specify a user to run the Snowpipe Stream. To learn how to create a user, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management). **Type**: `string` ## [](#example-pipelines)Example pipelines The following examples show you how to ingest, process, and write data to Snowflake from: - A PostgreSQL table using change data capture (CDC) - A Redpanda cluster - A REST API that posts JSON payloads to a HTTP server See also: [Ingest data into Snowflake cookbook](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/snowflake_ingestion/) ### Write data exactly once to a Snowflake table using CDC Send data from a PostgreSQL table and write it to Snowflake exactly once using PostgreSQL logical replication. This example includes some important features: - To make sure that a Snowflake streaming channel does not assume that older data is already committed, the configuration sets a 45-second interval between message batches. This interval prevents a message batch from being sent while another batch is retried. - The log sequence number of each data update from the Write-Ahead Log (WAL) in PostgreSQL makes sure that data is only uploaded once to the `snowflake_streaming` output, and that messages sent to the output are already lexicographically ordered. > 📝 **NOTE** > > To do exactly-once data delivery, it’s important that records are delivered in order to the output, and are correctly partitioned. Before you start, read the [`offset_token`](#offset_token) field description. Alternatively, remove the `offset_token` field to use Redpanda Connect’s default at-least-once delivery model. ```yaml input: postgres_cdc: dsn: postgres://foouser:foopass@localhost:5432/foodb schema: "public" tables: ["my_pg_table"] # Use very large batches. Each batch is sent to Snowflake individually, # so to optimize query performance, use the largest file size # your memory allows batching: count: 50000 period: 45s # Set an interval between message batches to prevent multiple batches # from being in flight at once checkpoint_limit: 1 output: snowflake_streaming: # Using the log sequence number makes sure data is only updated exactly once offset_token: "${!@lsn}" # Sending a single ordered log means you can only send one update # at a time and properly increment the offset_token # and use only a single channel. max_in_flight: 1 account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MY_PG_TABLE" private_key_file: "my/private/key.p8" ``` ### Ingest data exactly once from Redpanda Ingest data from Redpanda using consumer groups, decode the schema using the schema registry, then write the corresponding data into Snowflake. This example includes some important features: - To create multiple Redpanda Connect streams to write to each output table, you need a unique channel prefix per stream. The `channel_prefix` field constructs a unique prefix for each stream using the host name. - To prevent message failures from being retried and changing the order of delivered messages, a dead-letter queue processes them. > 📝 **NOTE** > > To do exactly-once data delivery, it’s important that records are delivered in order to the output, and are correctly partitioned. Before you start, read the [`channel_name`](#channel_name) and [`offset_token`](#offset_token) field descriptions. Alternatively, remove the `offset_token` field to use Redpanda Connect’s default at-least-once delivery model. ```yaml input: redpanda_common: topics: ["my_topic_going_to_snow"] consumer_group: "redpanda_connect_to_snowflake" # Use very large batches. Each batch is sent to Snowflake individually, # so to optimize query performance, use the largest file size # your memory allows fetch_max_bytes: 100MiB fetch_min_bytes: 50MiB partition_buffer_bytes: 100MiB pipeline: processors: - schema_registry_decode: url: "redpanda.example.com:8081" basic_auth: enabled: true username: MY_USER_NAME password: "${TODO}" output: fallback: - snowflake_streaming: # To write an ordered stream of messages, each partition in # Apache Kafka gets its own channel. channel_name: "partition-${!@kafka_partition}" # Offsets are lexicographically sorted in string form by padding with # leading zeros offset_token: offset-${!"%016X".format(@kafka_offset)} account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" schema_evolution: enabled: true # To prevent delivery failures from changing the order of # delivered records, it's important that they are immediately # sent to a dead-letter queue. - retry: output: redpanda_common: topic: "dead_letter_queue" ``` ### HTTP server to push data to Snowflake Create a HTTP server input that receives HTTP PUT requests with JSON payloads. The payloads are buffered locally then written to Snowflake in batches. To create multiple Redpanda Connect streams to write to each output table, you need a unique channel prefix per stream. In this example, the `channel_prefix` field constructs a unique prefix for each stream using the host name. > 📝 **NOTE** > > Using a buffer to immediately respond to the HTTP requests may result in data loss if there are delivery failures between the output and Snowflake. For more information about the configuration of buffers, see [buffers](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/buffers/memory/). Alternatively, remove the buffer entirely to respond to the HTTP request only once the data is written to Snowflake. ```yaml input: http_server: path: /snowflake buffer: memory: # Max inflight data before applying backpressure limit: 524288000 # 50MiB # Batching policy the size of the files sent to Snowflake batch_policy: enabled: true byte_size: 33554432 # 32MiB period: "10s" output: snowflake_streaming: account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" channel_prefix: "snowflake-channel-for-${HOST}" schema_evolution: enabled: true ``` --- # Page 366: splunk_hec **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/splunk_hec.md --- # splunk_hec > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: splunk_hec latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/splunk_hec page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/splunk_hec.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/splunk_hec.adoc description: Publishes messages to a Splunk HTTP Endpoint Collector (HEC). page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Publishes messages to a Splunk HTTP Endpoint Collector (HEC). #### Common ```yml outputs: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#event_host)`event_host` Set the host value to assign to the event data. Overrides existing host field if present. **Type**: `string` ### [](#event_index)`event_index` Set the index value to assign to the event data. Overrides existing index field if present. **Type**: `string` ### [](#event_source)`event_source` Set the source value to assign to the event data. Overrides existing source field if present. **Type**: `string` ### [](#event_sourcetype)`event_sourcetype` Set the sourcetype value to assign to the event data. Overrides existing sourcetype field if present. **Type**: `string` ### [](#gzip)`gzip` Enable gzip compression **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#token)`token` A bot token used for authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#url)`url` Full HTTP Endpoint Collector (HEC) URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/collector/event ``` --- # Page 367: sql_insert **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sql_insert.md --- # sql_insert > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_insert latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/sql_insert page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/sql_insert.adoc description: Inserts a row into an SQL database for each message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Inserts a row into an SQL database for each message. #### Common ```yml outputs: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sql_raw/) instead. There is no dedicated `sql_delete` output. To delete rows, use `sql_raw` with a DELETE statement: ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable query: "DELETE FROM my_table WHERE id = $1" args_mapping: root = [ this.id ] ``` To insert into a table determined at runtime, use `sql_raw` with `unsafe_dynamic_query: true`, which enables Bloblang interpolation in the `query` field. > ⚠️ **CAUTION** > > Interpolating user-supplied values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable unsafe_dynamic_query: true query: 'INSERT INTO ${! this.table_name } (id, value) VALUES ($1, $2)' args_mapping: root = [ this.id, this.value ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of inserts to run in parallel. **Type**: `int` **Default**: `64` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 368: sql_raw **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sql_raw.md --- # sql_raw > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_raw latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/sql_raw page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/sql_raw.adoc description: Executes an arbitrary SQL query for each message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes an arbitrary SQL query for each message. #### Common ```yml outputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` For some scenarios where you might use this output, see [Examples](#examples). ## [](#batch-execution-and-ordered-writes)Batch execution and ordered writes Configure a `batching` policy to accumulate messages and write them together. When you set more than one query (see [Conditional queries](#conditional-queries)), all messages in a batch run in a single database transaction, which reduces round-trips to the database. A single-query configuration runs each message individually without a transaction, preserving per-message error granularity. When consuming from Redpanda or Kafka, this output orders messages by partition within the transaction. This lets you raise `max_in_flight` above `1` to parallelize writes across partitions while preserving consume order within each partition, so you can scale throughput without breaking change-data-capture ordering. Messages without a `kafka_partition` metadata field are treated as partition `0`. This makes `sql_raw` suitable for high-throughput sink pipelines, including CDC, that previously required `max_in_flight: 1` to stay ordered. ## [](#conditional-queries)Conditional queries Use the `queries` field to route each message to a different SQL statement based on its content or metadata, without a preprocessing `mapping` step or `unsafe_dynamic_query`. Set a `when` condition (a Bloblang expression) on each entry: the first query whose `when` evaluates to `true`, or the first query with no `when`, runs for that message. If none of your queries have a `when` condition, every query runs for each message within the same transaction. Because an unconditioned query always matches, Redpanda Connect lints against configuring one before later queries in the list. For example, route change-data-capture tombstones to a `DELETE` and everything else to an upsert: ```yaml output: sql_raw: driver: postgres dsn: "${DSN}" max_in_flight: 8 batching: count: 100 period: 100ms queries: - when: 'root = meta("kafka_tombstone_message") == "true"' query: 'DELETE FROM orders WHERE id = $1' args_mapping: 'root = [ meta("kafka_key").parse_json().id ]' - query: | INSERT INTO orders (id, name, updated_at) VALUES ($1, $2, $3) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at args_mapping: 'root = [ this.id, this.name, this.updated_at ]' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Allows you to configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `array` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of database statements to run in parallel. When consuming from Redpanda or Kafka, messages are ordered by partition within each transaction, so you can raise this above `1` to parallelize writes across partitions while preserving consume order within each partition. This lets high-throughput and change-data-capture pipelines scale without setting `max_in_flight` to `1`. Messages without a `kafka_partition` metadata field are treated as partition `0`. **Type**: `int` **Default**: `64` ### [](#queries)`queries[]` A list of database statements to run in addition to your main [`query`](#query). If you specify multiple queries, they are executed within a single transaction. For more information, see [Examples](#examples). **Type**: `array` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-query)`queries[].query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | |---|---| | `clickhouse` | Dollar sign | | `mysql` | Question mark | | `postgres` | Dollar sign | | `pgx` | Dollar sign | | `mssql` | Question mark | | `sqlite` | Question mark | | `oracle` | Colon | | `snowflake` | Question mark | | `trino` | Question mark | | `gocosmos` | Colon | **Type**: `string` ### [](#queries-when)`queries[].when` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that, when set, is evaluated for each message to determine whether to execute this query. The mapping should return a boolean value. The first query in the list whose `when` condition evaluates to `true` (or that has no `when` condition) is executed. This enables conditional query routing based on message content or metadata without requiring `unsafe_dynamic_query`. **Type**: `string` ```yaml # Examples: when: root = meta("kafka_tombstone_message") == "true" # --- when: root = this.operation == "delete" ``` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) in the query. Great care should be made to ensure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (id, name, topic) VALUES (?, ?, ?);" args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we dynamically create output tables transactionally with inserting a record into the newly created table. ```yaml output: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` ### [](#conditional-cdc-queries-postgresql)Conditional CDC Queries (PostgreSQL) Route messages to different SQL operations based on message metadata. Tombstone messages trigger a DELETE, while all other messages perform an upsert. All operations within a batch execute in a single transaction, ordered by Kafka partition. ```yaml output: sql_raw: driver: postgres dsn: postgres://localhost/postgres max_in_flight: 8 batching: count: 100 period: 100ms queries: - when: 'root = meta("kafka_tombstone_message") == "true"' query: 'DELETE FROM users WHERE id = $1' args_mapping: 'root = [this.id]' - query: | INSERT INTO users (id, name, updated_at) VALUES ($1, $2, $3) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at args_mapping: 'root = [this.id, this.name, this.updated_at]' ``` --- # Page 369: switch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/switch.md --- # switch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: switch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/switch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/switch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/switch.adoc description: The switch output type allows you to route messages to different outputs based on their contents. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- The switch output type allows you to route messages to different outputs based on their contents. #### Common ```yml outputs: label: "" switch: retry_until_success: false cases: [] # No default (required) ``` #### Advanced ```yml outputs: label: "" switch: retry_until_success: false strict_mode: false cases: [] # No default (required) ``` Messages that do not pass the check of a single output case are effectively dropped. In order to prevent this outcome set the field [`strict_mode`](#strict_mode) to `true`, in which case messages that do not pass at least one case are considered failed and will be nacked and/or reprocessed depending on your input. ## [](#examples)Examples ### [](#basic-multiplexing)Basic Multiplexing The most common use for a switch output is to multiplex messages across a range of output destinations. The following config checks the contents of the field `type` of messages and sends `foo` type messages to an `amqp_1` output, `bar` type messages to a `gcp_pubsub` output, and everything else to a `redis_streams` output. Outputs can have their own processors associated with them, and in this example the `redis_streams` output has a processor that enforces the presence of a type field before sending it. ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type | "unknown" ``` ### [](#control-flow)Control Flow The `continue` field allows messages that have passed a case to be tested against the next one also. This can be useful when combining non-mutually-exclusive case checks. In the following example a message that passes both the check of the first case as well as the second will be routed to both. ```yaml output: switch: cases: - check: 'this.user.interests.contains("walks").catch(false)' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/people_what_think_good continue: true - check: 'this.user.dislikes.contains("videogames").catch(false)' output: gcp_pubsub: project: people topic: that_i_dont_want_to_hang_with ``` ## [](#fields)Fields ### [](#cases)`cases[]` A list of switch cases, outlining outputs that can be routed to. **Type**: `array` ```yaml # Examples: cases: - check: this.urls.contains("http://benthos.dev") continue: true output: cache: key: ${!json("id")} target: foo - output: s3: bucket: bar path: ${!json("id")} ``` ### [](#cases-check)`cases[].check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should be routed to the case output. If left empty the case always passes. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#cases-continue)`cases[].continue` Indicates whether, if this case passes for a message, the next case should also be tested. **Type**: `bool` **Default**: `false` ### [](#cases-output)`cases[].output` An [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/) for messages that pass the check to be routed to. **Type**: `output` ### [](#retry_until_success)`retry_until_success` If a selected output fails to send a message this field determines whether it is reattempted indefinitely. If set to false the error is instead propagated back to the input level. If a message can be routed to >1 outputs it is usually best to set this to true in order to avoid duplicate messages being routed to an output. **Type**: `bool` **Default**: `false` ### [](#strict_mode)`strict_mode` This field determines whether an error should be reported if no condition is met. If set to true, an error is propagated back to the input level. The default behavior is false, which will drop the message. **Type**: `bool` **Default**: `false` --- # Page 370: sync_response **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sync_response.md --- # sync_response > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sync_response latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/sync_response page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/sync_response.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/sync_response.adoc description: Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type. ```yml # Config fields, showing default values output: label: "" sync_response: {} ``` --- # Page 371: timeplus **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/timeplus.md --- # timeplus > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: timeplus page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/outputs/timeplus page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/outputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/outputs/timeplus.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-11-05" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Sends messages to a data stream on [Timeplus Enterprise (Cloud or Self-Hosted)](https://docs.timeplus.com/) using the [Ingest API](https://docs.timeplus.com/ingest-api), or directly to the `timeplusd` component in Timeplus Enterprise. #### Common ```yml # Common configuration fields, showing default values output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml # All configuration fields, showing default values output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This output only accepts structured messages. All messages must: - Contain the same keys. - Use a structure that matches the schema of the destination data stream. If your upstream data source or pipeline returns unstructured messages, such as strings, you can configure an output processor to transform the messages. See the [Unstructured messages](#unstructured-messages) section for examples. ## [](#examples)Examples #### Timeplus Enterprise (Cloud) You must [generate an API key](https://docs.timeplus.com/apikey) using the web console of Timeplus Enterprise (Cloud). ```yaml output: timeplus: workspace: stream: apikey: ``` Replace the following placeholders with your own values: - ``: The ID of the workspace you want to send messages to. - ``: The name of the destination data stream. - ``: The API key for the Ingest API. #### Timeplus Enterprise (Self-Hosted) You must specify the username, password, and URL of the application server. ```yaml output: timeplus: url: http://localhost:8000 workspace: stream: username: password: ``` Replace the following placeholders with your own values: - ``: The ID of the workspace you want to send messages to. - ``: The name of the destination data stream. - ``: The username for the Timeplus application server. - ``: The password for the Timeplus application server. #### timeplusd You must specify the HTTP port for `timeplusd`. ```yaml output: timeplus: url: http://localhost:3218 stream: username: password: ``` Replace the following placeholders with your own values: - ``: The name of the destination data stream. - ``: The username for the Timeplus application server. - ``: The password for the Timeplus application server. ### [](#unstructured-messages)Unstructured messages If your upstream data source or pipeline returns unstructured messages, such as strings, you can configure an output processor to transform them into structured messages and then pass them to the output. In the following example, the `mapping` processor creates a field called `raw`, and uses the functions `content().string()` to store the original string content into it, thereby creating structured messages. If you use this example, you must also add the `raw` field name to the destination data stream, so that your message structure matches the schema of your destination data stream. ```yaml output: timeplus: workspace: stream: apikey: processors: - mapping: | root = {} root.raw = content().string() ``` ## [](#fields)Fields ### [](#target)`target` The destination platform. For Timeplus Enterprise (Cloud or Self-Hosted), enter `timeplus`, or `timeplusd` for the `timeplusd` component. **Type**: `string` **Default**: `timeplus` **Options**: `timeplus`, `timeplusd` ### [](#url)`url` The URL of your Timeplus instance, which should always include the schema and host. **Type**: `string` **Default**: `[https://us-west-2.timeplus.cloud](https://us-west-2.timeplus.cloud)` ```yml # Examples url: http://localhost:8000 url: http://127.0.0.1:3218 ``` ### [](#workspace)`workspace` The ID of the workspace you want to send messages to. This field is required if the `target` field is set to `timeplus`. **Type**: `string` ### [](#stream)`stream` The name of the destination data stream. Make sure the schema of the data stream matches this output. **Type**: `string` ### [](#apikey)`apikey` The API key for the Ingest API. You need to generate this in the web console of Timeplus Enterprise (Cloud). This field is required if you are sending messages to Timeplus Enterprise (Cloud). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#username)`username` The username for the Timeplus application server. This field is required if you are sending messages to Timeplus Enterprise (Self-Hosted) or `timeplusd`. **Type**: `string` ### [](#password)`password` The password for the Timeplus application server. This field is required if you are sending messages to Timeplus Enterprise (Self-Hosted) or `timeplusd`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#batching)`batching` Configure a [batching policy](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). **Type**: `object` ```yml # Examples batching: byte_size: 5000 count: 0 period: 1s batching: count: 10 period: 1s batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-byte_size)`batching.byte_size` The amount of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. **Type**: `string` **Default**: `""` ```yml # Examples period: 1s period: 1m period: 500ms ``` ### [](#batching-check)`batching.check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yml # Examples check: this.type == "end_of_transaction" ``` ### [](#batching-processors)`batching.processors` For aggregating and archiving message batches, you can add a list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to a batch as it is flushed. All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches. **Type**: `array` ```yml # Examples processors: - archive: format: concatenate processors: - archive: format: lines processors: - archive: format: json_array ``` --- # Page 372: a2a_message **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/a2a_message.md --- # a2a_message > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: a2a_message latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/a2a_message page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/a2a_message.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/a2a_message.adoc description: Sends messages to an A2A (Agent-to-Agent) protocol agent and returns the response. page-git-created-date: "2026-02-18" page-git-modified-date: "2026-08-11" --- Sends messages to an A2A (Agent-to-Agent) protocol agent and returns the response. This processor enables Redpanda Connect pipelines to communicate with A2A protocol agents. Currently only JSON-RPC transport is supported. The processor sends a message to the agent and polls for task completion. The agent’s response is returned as the processor output. For more information about the A2A protocol, see [https://a2a-protocol.org/latest/specification](https://a2a-protocol.org/latest/specification) #### Common ```yml processors: label: "" a2a_message: agent_card_url: "" # No default (required) prompt: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" a2a_message: agent_card_url: "" # No default (required) prompt: "" # No default (optional) final_message_only: true ``` ## [](#fields)Fields ### [](#agent_card_url)`agent_card_url` URL for the A2A agent card. Can be either a base URL (e.g., `[https://example.com](https://example.com)`) or a full path to the agent card (e.g., `[https://example.com/.well-known/agent.json](https://example.com/.well-known/agent.json)`). If no path is provided, defaults to `/.well-known/agent.json`. Authentication uses OAuth2 from environment variables. **Type**: `string` ### [](#final_message_only)`final_message_only` If true, returns only the text from the final agent message (concatenated from all text parts). If false, returns the complete Message or Task object as structured data with full history, artifacts, and metadata. Example with final\_message\_only: true (default): ```none Here is the answer to your question... ``` Example with final\_message\_only: false: ```json { "id": "task-123", "contextId": "ctx-456", "status": { "state": "completed" }, "history": [ {"role": "user", "parts": [{"text": "Your question"}]}, {"role": "agent", "parts": [{"text": "Here is the answer to your question..."}]} ], "artifacts": [] } ``` **Type**: `bool` **Default**: `true` ### [](#prompt)`prompt` The user prompt to send to the agent. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 373: Processors **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about.md --- # Processors > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Processors latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect processors are functions applied to messages passing through a pipeline. The function signature allows a processor to mutate or drop messages depending on the content of the message. There are many types on offer but the most powerful are the [`mapping`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) and [`mutation`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mutation/) processors. Processors are set via config, and depending on where in the config they are placed they will be run either immediately after a specific input (set in the input section), on all messages (set in the pipeline section) or before a specific output (set in the output section). Most processors apply to all messages and can be placed in the pipeline section: ```yaml pipeline: threads: 1 processors: - label: my_cool_mapping mapping: | root.message = this root.meta.link_count = this.links.length() ``` The `threads` field in the pipeline section determines how many parallel processing threads are created. You can read more about parallel processing in the [pipeline guide](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/processing_pipelines/). ## [](#labels)Labels Processors have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple nested processors, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/). ## [](#error-handling)Error handling Some processors have conditions whereby they might fail. Rather than throw these messages into the abyss Redpanda Connect still attempts to send these messages onwards, and has mechanisms for filtering, recovering or dead-letter queuing messages that have failed which can be read about [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ### [](#error-logs)Error logs Errors that occur during processing can be roughly separated into two groups; those that are unexpected intermittent errors such as connectivity problems, and those that are logical errors such as bad input data or unmatched schemas. All processing errors result in the messages being flagged as failed, [error metrics](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/) increasing for the given errored processor, and debug level logs being emitted that describe the error. Only errors that are known to be intermittent are also logged at the error level. The reason for this behavior is to prevent noisy logging in cases where logical errors are expected and will likely be [handled in config](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). However, this can also sometimes make it easy to miss logical errors in your configs when they lack error handling. If you suspect you are experiencing processing errors and do not wish to add error handling yet then a quick and easy way to expose those errors is to enable debug level logs with the cli flag `--log.level=debug` or by setting the level in config: ```yaml logger: level: DEBUG ``` ## [](#using-processors-as-outputs)Using processors as outputs It might be the case that a processor that results in a side effect, such as the [`sql_insert`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sql_insert/) or [`redis`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/redis/) processors, is the only side effect of a pipeline, and therefore could be considered the output. In such cases it’s possible to place these processors within a [`reject` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/reject/) so that they behave the same as regular outputs, where success results in dropping the message with an acknowledgement and failure results in a nack (or retry): ```yaml output: reject: 'failed to send data: ${! error() }' processors: - try: - redis: url: tcp://localhost:6379 command: sadd args_mapping: 'root = [ this.key, this.value ]' - mapping: root = deleted() ``` The way this works is that if your processor with the side effect (`redis` in this case) succeeds then the final `mapping` processor deletes the message which results in an acknowledgement. If the processor fails then the `try` block exits early without executing the `mapping` processor and instead the message is routed to the `reject` output, which nacks the message with an error message containing the error obtained from the `redis` processor. ## [](#batching-and-multiple-part-messages)Batching and multiple-part messages All Redpanda Connect processors support multiple-part messages, which are synonymous with batches. This enables [windowed processing](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/windowed_processing/) capabilities. Many processors are able to perform their behaviors on specific parts of a message batch, or on all parts, and have a field `parts` for specifying an array of part indexes they should apply to. If the list of target parts is empty these processors will be applied to all message parts. Part indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if part = -1 then the selected part will be the last part of the message, if part = -2 then the part before the last element will be selected, and so on. Some processors such as [`dedupe`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/dedupe/) act across an entire batch, when instead we might like to perform them on individual messages of a batch. In this case the [`for_each`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each/) processor can be used. You can read more about batching [in this document](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). --- # Page 374: archive **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive.md --- # archive > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: archive latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/archive page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/archive.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/archive.adoc description: Archives all the messages of a batch into a single message according to the selected archive format. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Archives all the messages of a batch into a single message according to the selected archive format. ```yml # Config fields, showing default values label: "" archive: format: "" # No default (required) path: "" ``` Some archive formats (such as tar, zip) treat each archive item (message part) as a file with a path. Since message parts only contain raw data a unique path must be generated for each part. This can be done by using function interpolations on the 'path' field as described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). For types that aren’t file based (such as binary) the file field is ignored. The resulting archived message adopts the metadata of the _first_ message part of the batch. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#format)`format` The archiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Archive messages to a binary blob format. | | concatenate | Join the raw contents of each message into a single binary message. | | json_array | Attempt to parse each message as a JSON document and append the result to an array, which becomes the contents of the resulting message. | | lines | Join the raw contents of each message and insert a line break between each one. | | tar | Archive messages to a unix standard tape archive. | | zip | Archive messages to a zip file. | ### [](#path)`path` The path to set for each message in the archive (when applicable). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: path: ${!count("files")}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}-${!json("id")}.json ``` ## [](#examples)Examples ### [](#tar-archive)Tar Archive If we had JSON messages in a batch each of the form: ```json {"doc":{"id":"foo","body":"hello world 1"}} ``` And we wished to tar archive them, setting their filenames to their respective unique IDs (with the extension `.json`), our config might look like this: ```yaml pipeline: processors: - archive: format: tar path: ${!json("doc.id")}.json ``` --- # Page 375: avro **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/avro.md --- # avro > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: avro latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/avro page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/avro.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/avro.adoc description: Performs Avro based operations on messages based on a schema. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Performs Avro based operations on messages based on a schema. ```yml # Config fields, showing default values label: "" avro: operator: "" # No default (required) encoding: textual schema: "" schema_path: "" ``` > ⚠️ **WARNING** > > If you are consuming or generating messages using a schema registry service then it is likely this processor will fail as those services require messages to be prefixed with the identifier of the schema version being used. Instead, try the [`schema_registry_encode`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/schema_registry_encode/) and [`schema_registry_decode`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/schema_registry_decode/) processors. ## [](#operators)Operators ### [](#to_json)`to_json` Converts Avro documents into a JSON structure. This makes it easier to manipulate the contents of the document within Benthos. The encoding field specifies how the source documents are encoded. ### [](#from_json)`from_json` Attempts to convert JSON documents into Avro documents according to the specified encoding. ## [](#fields)Fields ### [](#encoding)`encoding` An Avro encoding format to use for conversions to and from a schema. **Type**: `string` **Default**: `textual` **Options**: `textual`, `binary`, `single` ### [](#operator)`operator` The [operator](#operators) to execute **Type**: `string` **Options**: `to_json`, `from_json` ### [](#schema)`schema` A full Avro schema to use. **Type**: `string` **Default**: `""` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. URLs must begin with `file://` or `http://`. Note that `file://` URLs must use absolute paths (e.g. `[file:///absolute/path/to/spec.avsc](file:///absolute/path/to/spec.avsc)`); relative paths are not supported. **Type**: `string` **Default**: `""` ```yaml # Examples: schema_path: file:///path/to/spec.avsc # --- schema_path: http://localhost:8081/path/to/spec/versions/1 ``` --- # Page 376: aws_bedrock_chat **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_bedrock_chat.md --- # aws_bedrock_chat > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_bedrock_chat latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/aws_bedrock_chat page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/aws_bedrock_chat.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/aws_bedrock_chat.adoc description: Generates responses to messages in a chat conversation, using the AWS Bedrock API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates responses to messages in a chat conversation, using the [AWS Bedrock API](https://aws.amazon.com/bedrock/). #### Common ```yml processors: label: "" aws_bedrock_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" aws_bedrock_chat: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) stop: [] # No default (optional) top_p: "" # No default (optional) ``` This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). ## [](#fields)Fields ### [](#credentials)`credentials` Configure which AWS credentials to use (optional). For more information, see [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` The external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials you want to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials you want to use. You must enter this value when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the generated response. **Type**: `int` ### [](#model)`model` The model ID to use. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-text-express-v1 # --- model: anthropic.claude-3-5-sonnet-20240620-v1:0 # --- model: cohere.command-text-v14 # --- model: meta.llama3-1-70b-instruct-v1:0 # --- model: mistral.mistral-large-2402-v1:0 ``` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stop)`stop[]` A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the AWS Bedrock LLM. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#temperature)`temperature` The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options. A higher value makes the model more likely to choose lower-probability options. **Type**: `float` ### [](#top_p)`top_p` The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of `0.8`, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence. **Type**: `float` --- # Page 377: aws_bedrock_embeddings **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_bedrock_embeddings.md --- # aws_bedrock_embeddings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_bedrock_embeddings page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/aws_bedrock_embeddings page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/aws_bedrock_embeddings.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/aws_bedrock_embeddings.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-10-16" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Generates vector embeddings from text prompts, using the [AWS Bedrock API](https://aws.amazon.com/bedrock/). #### Common ```yaml # Common config fields, showing default values label: "" aws_bedrock_embeddings: model: amazon.titan-embed-text-v1 # No default (required) text: "" # No default (optional) ``` #### Advanced ```yaml # All config fields, showing default values label: "" aws_bedrock_embeddings: region: "" endpoint: "" credentials: from_ec2_role: false role: "" role_external_id: "" model: amazon.titan-embed-text-v1 # No default (required) text: "" # No default (optional) ``` This processor sends text prompts to your chosen large language model (LLM), which generates vector embeddings for them using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). ## [](#fields)Fields ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#input_type)`input_type` Specifies the type of input passed to the model. Required by Cohere embedding models; ignored by Amazon Titan models. **Type**: `string` | Option | Summary | | --- | --- | | classification | Used for embeddings passed through a text classifier. | | clustering | Used for the embeddings run through a clustering algorithm. | | search_document | Used for embeddings stored in a vector database for search use-cases. | | search_query | Used for embeddings of search queries run against a vector DB to find relevant documents. | ### [](#model)`model` The ID of the LLM that you want to use to generate vector embeddings. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-embed-text-v1 # --- model: amazon.titan-embed-text-v2:0 # --- model: cohere.embed-english-v3 # --- model: cohere.embed-multilingual-v3 # --- model: cohere.embed-v4:0 ``` ### [](#region)`region` The region in which your AWS resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#text)`text` The prompt you want to generate a vector embedding for. The processor submits the entire payload as a string. **Type**: `string` --- # Page 378: aws_dynamodb_partiql **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_dynamodb_partiql.md --- # aws_dynamodb_partiql > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_dynamodb_partiql latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/aws_dynamodb_partiql page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/aws_dynamodb_partiql.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/aws_dynamodb_partiql.adoc description: Executes a PartiQL expression against a DynamoDB table for each message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a PartiQL expression against a DynamoDB table for each message. #### Common ```yml processors: label: "" aws_dynamodb_partiql: query: "" # No default (required) args_mapping: "" ``` #### Advanced ```yml processors: label: "" aws_dynamodb_partiql: query: "" # No default (required) unsafe_dynamic_query: false use_batch: true args_mapping: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Both writes or reads are supported, when the query is a read the contents of the message will be replaced with the result. This processor is more efficient when messages are pre-batched as the whole batch will be executed in a single call. ## [](#examples)Examples ### [](#insert)Insert The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages: ```yaml pipeline: processors: - aws_dynamodb_partiql: query: "INSERT INTO footable VALUE {'foo':'?','bar':'?','baz':'?'}" args_mapping: | root = [ { "S": this.foo }, { "S": meta("kafka_topic") }, { "S": this.document.content }, ] ``` ### [](#query-a-gsi-for-a-single-record)Query a GSI for a single record The following example looks up a single record from the table footable using the global secondary index index\_name, matching on the field bar. BatchExecuteStatement can’t query a GSI, so use\_batch is disabled: ```yaml pipeline: processors: - aws_dynamodb_partiql: query: "SELECT * FROM \"footable\".\"index_name\" WHERE bar = ?" use_batch: false args_mapping: | root = [ { "S": this.bar }, ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that, for each message, creates a list of arguments to use with the query. **Type**: `string` **Default**: `""` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#query)`query` A PartiQL query to execute for each message. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable dynamic queries that support interpolation functions. **Type**: `bool` **Default**: `false` ### [](#use_batch)`use_batch` Whether to execute all messages in a batch as a single `BatchExecuteStatement` call. Set this to `false` to execute one `ExecuteStatement` call per message instead, which is required for PartiQL `SELECT` queries against a global secondary index (GSI), because `BatchExecuteStatement` does not support querying a GSI. Only the first result row is used when a query returns multiple items. **Type**: `bool` **Default**: `true` --- # Page 379: aws_lambda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_lambda.md --- # aws_lambda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: aws_lambda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/aws_lambda page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/aws_lambda.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/aws_lambda.adoc description: Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message. #### Common ```yml processors: label: "" aws_lambda: parallel: false function: "" # No default (required) ``` #### Advanced ```yml processors: label: "" aws_lambda: parallel: false function: "" # No default (required) rate_limit: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) timeout: 5s retries: 3 ``` The `rate_limit` field can be used to specify a rate limit [resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to cap the rate of requests across parallel components service wide. In order to map or encode the payload to a specific request body, and map the response back into the original payload instead of replacing it entirely, you can use the [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/). ## [](#error-handling)Error handling When Redpanda Connect is unable to connect to the AWS endpoint or is otherwise unable to invoke the target lambda function it will retry the request according to the configured number of retries. Once these attempts have been exhausted the failed message will continue through the pipeline with it’s contents unchanged, but flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). However, if the invocation of the function is successful but the function itself throws an error, then the message will have it’s contents updated with a JSON payload describing the reason for the failure, and a metadata field `lambda_function_error` will be added to the message allowing you to detect and handle function errors with a [`branch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/): ```yaml pipeline: processors: - branch: processors: - aws_lambda: function: foo result_map: | root = if meta().exists("lambda_function_error") { throw("Invocation failed due to %v: %v".format(this.errorType, this.errorMessage)) } else { this } output: switch: retry_until_success: false cases: - check: errored() output: reject: ${! error() } - output: resource: somewhere_else ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). ## [](#examples)Examples ### [](#branched-invoke)Branched Invoke This example uses a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) to map a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update ``` ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#function)`function` The function to invoke. **Type**: `string` ### [](#parallel)`parallel` Whether messages of a batch should be dispatched in parallel. **Type**: `bool` **Default**: `false` ### [](#rate_limit)`rate_limit` An optional [`rate_limit`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle invocations by. **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts for each message. **Type**: `int` **Default**: `3` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait before abandoning an invocation. **Type**: `string` **Default**: `5s` --- # Page 380: azure_cosmosdb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/azure_cosmosdb.md --- # azure_cosmosdb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: azure_cosmosdb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/azure_cosmosdb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/azure_cosmosdb.adoc description: Creates or updates messages as JSON documents in Azure CosmosDB. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). ### Common ```yml processors: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) ``` ### Advanced ```yml processors: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) enable_content_response_on_write: true ``` When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: - `activity_id` - `request_charge` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ## [](#examples)Examples ### [](#patch-documents)Patch documents Query documents from a container and patch them. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish processors: - mapping: | root = "" meta habitat = json("habitat") meta id = this.id - azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! meta("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" # Return the updated document enable_content_response_on_write: true ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#enable_content_response_on_write)`enable_content_response_on_write` Enable content response on write operations. To save some bandwidth, set this to false if you don’t need to receive the updated message(s) from the server, in which case the processor will not modify the content of the messages which are fed into it. Applies to every operation except Read. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Read | Read operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `array` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 381: benchmark **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/benchmark.md --- # benchmark > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: benchmark latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/benchmark page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/benchmark.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/benchmark.adoc description: Logs basic throughput statistics of messages that pass through this processor. page-git-created-date: "2024-12-16" page-git-modified-date: "2026-05-26" --- Logs throughput statistics for processed messages, and provides a summary of those statistics over the lifetime of the processor. ```yml # Configuration fields, showing default values label: "" benchmark: interval: 5s count_bytes: true ``` ## [](#throughput-statistics)Throughput statistics This processor logs the following rolling statistics at a [configurable interval](#interval) to help you to understand the current performance of your pipeline: - The number of messages processed per second. - The number of bytes processed per second (optional). For example: ```bash INFO rolling stats: 1 msg/sec, 407 B/sec ``` When the processor shuts down, it also logs a summary of the number and size of messages processed during its lifetime. For example: ```bash INFO total stats: 1.00186 msg/sec, 425 B/sec ``` ## [](#fields)Fields ### [](#count_bytes)`count_bytes` Whether to measure the number of bytes per second of throughput. If set to `true`, Redpanda Connect must serialize structured data to count the number of bytes processed, which can unnecessarily degrade performance if serialization is not required elsewhere in your pipeline. **Type**: `bool` **Default**: `true` ### [](#interval)`interval` How often to emit rolling statistics. Set to `0`, if you only want to log summary statistics when the processor shuts down. **Type**: `string` **Default**: `5s` --- # Page 382: bloblang **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/bloblang.md --- # bloblang > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: bloblang latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/bloblang page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/bloblang.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/bloblang.adoc description: Executes a Bloblang mapping on messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) mapping on messages. ```yml # Config fields, showing default values label: "" bloblang: "" ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information see [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#component-rename)Component rename This processor was recently renamed to the [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) in order to make the purpose of the processor more prominent. It is still valid to use the existing `bloblang` name but eventually it will be deprecated and replaced by the new name in example configs. ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the fans to only those with an obsession score above 0.5, giving us: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - bloblang: | root = this root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - bloblang: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired fallback behavior, which you can read about in [Error handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling.adoc). --- # Page 383: bounds_check **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/bounds_check.md --- # bounds_check > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: bounds_check latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/bounds_check page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/bounds_check.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/bounds_check.adoc description: Removes messages (and batches) that do not fit within certain size boundaries. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Removes messages (and batches) that do not fit within certain size boundaries. #### Common ```yml processors: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 ``` #### Advanced ```yml processors: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 max_parts: 100 min_parts: 1 ``` ## [](#fields)Fields ### [](#max_part_size)`max_part_size` The maximum size of a message to allow (in bytes) **Type**: `int` **Default**: `1073741824` ### [](#max_parts)`max_parts` The maximum size of message batches to allow (in message count) **Type**: `int` **Default**: `100` ### [](#min_part_size)`min_part_size` The minimum size of a message to allow (in bytes) **Type**: `int` **Default**: `1` ### [](#min_parts)`min_parts` The minimum size of message batches to allow (in message count) **Type**: `int` **Default**: `1` --- # Page 384: branch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch.md --- # branch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: branch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/branch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/branch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/branch.adoc description: The branch processor allows you to create a new request message via a Bloblang mapping, execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- The `branch` processor allows you to create a new request message via a [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/), execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping. ```yml # Config fields, showing default values label: "" branch: request_map: "" processors: [] # No default (required) result_map: "" ``` This is useful for preserving the original message contents when using processors that would otherwise replace the entire contents. ## [](#metadata)Metadata Metadata fields that are added to messages during branch processing will not be automatically copied into the resulting message. In order to do this you should explicitly declare in your `result_map` either a wholesale copy with `meta = metadata()`, or selective copies with `meta foo = metadata("bar")` and so on. It is also possible to reference the metadata of the origin message in the `result_map` using the [`@` operator](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#metadata). ## [](#error-handling)Error handling If the `request_map` fails the child processors will not be executed. If the child processors themselves result in an (uncaught) error then the `result_map` will not be executed. If the `result_map` fails the message will remain unchanged. Under any of these conditions standard [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/) can be used in order to filter, DLQ or recover the failed messages. ## [](#conditional-branching)Conditional branching If the root of your request map is set to `deleted()` then the branch processors are skipped for the given message, this allows you to conditionally branch messages. ## [](#fields)Fields ### [](#processors)`processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `array` ### [](#request_map)`request_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#result_map)`result_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ## [](#examples)Examples ### [](#http-request)HTTP Request This example strips the request message into an empty body, grabs an HTTP payload, and places the result back into the original message at the path `image.pull_count`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: root.image.pull_count = this.pull_count # Example input: {"id":"foo","some":"pre-existing data"} # Example output: {"id":"foo","some":"pre-existing data","image":{"pull_count":1234}} ``` ### [](#non-structured-results)Non Structured Results When the result of your branch processors is unstructured and you wish to simply set a resulting field to the raw output use the content function to obtain the raw bytes of the resulting message and then coerce it into your value type of choice: ```yaml pipeline: processors: - branch: request_map: 'root = this.document.id' processors: - cache: resource: descriptions_cache key: ${! content() } operator: get result_map: root.document.description = content().string() # Example input: {"document":{"id":"foo","content":"hello world"}} # Example output: {"document":{"id":"foo","content":"hello world","description":"this is a cool doc"}} ``` ### [](#lambda-function)Lambda Function This example maps a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update # Example input: {"doc":{"id":"foo","body":"hello world"},"user":{"name":"fooey"}} # Output matches the input, which is unchanged ``` ### [](#conditional-caching)Conditional Caching This example caches a document by a message ID only when the type of the document is a foo: ```yaml pipeline: processors: - branch: request_map: | meta id = this.id root = if this.type == "foo" { this.document } else { deleted() } processors: - cache: resource: TODO operator: set key: ${! @id } value: ${! content() } ``` --- # Page 385: cache **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cache.md --- # cache > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cache latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/cache page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/cache.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/cache.adoc description: Performs operations against a cache resource for each message, allowing you to store or retrieve data within message payloads. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Performs operations against a [cache resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) for each message, allowing you to store or retrieve data within message payloads. #### Common ```yml processors: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ttl: "" # No default (optional) ``` For use cases where you wish to cache the result of processors, consider using the [`cached` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cached/) instead. This processor will interpolate functions within the `key` and `value` fields individually for each message. This allows you to specify dynamic keys and values based on the contents of the message payloads and metadata. You can find a list of functions in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#deduplication)Deduplication Deduplication can be done using the add operator with a key extracted from the message payload, since it fails when a key already exists we can remove the duplicates using a [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/): ```yaml pipeline: processors: - cache: resource: foocache operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } cache_resources: - label: foocache redis: url: tcp://TODO:6379 ``` ### [](#deduplication-batch-wide)Deduplication Batch-Wide Sometimes it’s necessary to deduplicate a batch of messages (also known as a window) by a single identifying value. This can be done by introducing a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/), which executes the cache only once on behalf of the batch, in this case with a value make from a field extracted from the first and last messages of the batch: ```yaml pipeline: processors: # Try and add one message to a cache that identifies the whole batch - branch: request_map: | root = if batch_index() == 0 { json("id").from(0) + json("meta.tail_id").from(-1) } else { deleted() } processors: - cache: resource: foocache operator: add key: ${! content() } value: t # Delete all messages if we failed - mapping: | root = if errored().from(0) { deleted() } ``` ### [](#hydration)Hydration It’s possible to enrich payloads with content previously stored in a cache by using the [`branch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) processor: ```yaml pipeline: processors: - branch: processors: - cache: resource: foocache operator: get key: '${! json("message.document_id") }' result_map: 'root.message.document = this' # NOTE: If the data stored in the cache is not valid JSON then use # something like this instead: # result_map: 'root.message.document = content().string()' cache_resources: - label: foocache memcached: addresses: [ "TODO:11211" ] ``` ## [](#fields)Fields ### [](#key)`key` A key to use with the cache. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#operator)`operator` The [operation](#operators) to perform with the cache. **Type**: `string` **Options**: `set`, `add`, `get`, `delete`, `exists` ### [](#resource)`resource` The [`cache` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to target with this processor. **Type**: `string` ### [](#ttl)`ttl` The time to live (TTL) of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, those that do will have a configuration field `default_ttl`, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` ### [](#value)`value` A value to use with the cache (when applicable). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ## [](#operators)Operators ### [](#set)`set` Set a key in the cache to a value. If the key already exists the contents are overridden. ### [](#add)`add` Set a key in the cache to a value. If the key already exists the action fails with a 'key already exists' error, which can be detected with [processor error handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ### [](#get)`get` Retrieve the contents of a cached key and replace the original message payload with the result. If the key does not exist the action fails with an error, which can be detected with [processor error handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ### [](#exists)`exists` Check whether a specific key is in the cache and replace the original message payload with `true` if the key exists, or `false` if it doesn’t. ### [](#delete)`delete` Delete a key and its contents from the cache. If the key does not exist the action is a no-op and will not fail with an error. --- # Page 386: cached **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cached.md --- # cached > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cached latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/cached page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/cached.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/cached.adoc description: Cache the result of applying one or more processors to messages identified by a key. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Cache the result of applying one or more processors to messages identified by a key. If the key already exists within the cache the contents of the message will be replaced with the cached result instead of applying the processors. This component is therefore useful in situations where an expensive set of processors need only be executed periodically. ```yml # Config fields, showing default values label: "" cached: cache: "" # No default (required) skip_on: errored() # No default (optional) key: my_foo_result # No default (required) ttl: "" # No default (optional) processors: [] # No default (required) ``` The format of the data when stored within the cache is a custom and versioned schema chosen to balance performance and storage space. It is therefore not possible to point this processor to a cache that is pre-populated with data that this processor has not created itself. ## [](#examples)Examples ### [](#cached-enrichment)Cached Enrichment In the following example we want to we enrich messages consumed from Kafka with data specific to the origin topic partition, we do this by placing an `http` processor within a `branch`, where the HTTP URL contains interpolation functions with the topic and partition in the path. However, it would be inefficient to make this HTTP request for every single message as the result is consistent for all data of a given topic partition. We can solve this by placing our enrichment call within a `cached` processor where the key contains the topic and partition, resulting in messages that originate from the same topic/partition combination using the cached result of the prior. ```yaml pipeline: processors: - branch: processors: - cached: key: '${! meta("kafka_topic") }-${! meta("kafka_partition") }' cache: foo_cache processors: - mapping: 'root = ""' - http: url: http://example.com/enrichment/${! meta("kafka_topic") }/${! meta("kafka_partition") } verb: GET result_map: 'root.enrichment = this' cache_resources: - label: foo_cache memory: # Disable compaction so that cached items never expire compaction_interval: "" ``` ### [](#periodic-global-enrichment)Periodic Global Enrichment In the following example we enrich all messages with the same data obtained from a static URL with an `http` processor within a `branch`. However, we expect the data from this URL to change roughly every 10 minutes, so we configure a `cached` processor with a static key (since this request is consistent for all messages) and a TTL of `10m`. ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - cached: key: static_foo cache: foo_cache ttl: 10m processors: - http: url: http://example.com/get/foo.json verb: GET result_map: 'root.foo = this' cache_resources: - label: foo_cache memory: {} ``` ## [](#fields)Fields ### [](#cache)`cache` The cache resource to read and write processor results from. **Type**: `string` ### [](#key)`key` A key to be resolved for each message, if the key already exists in the cache then the cached result is used, otherwise the processors are applied and the result is cached under this key. The key could be static and therefore apply generally to all messages or it could be an interpolated expression that is potentially unique for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: my_foo_result # --- key: ${! this.document.id } # --- key: ${! meta("kafka_key") } # --- key: ${! meta("kafka_topic") } ``` ### [](#processors)`processors[]` The list of processors whose result will be cached. **Type**: `array` ### [](#skip_on)`skip_on` A condition that can be used to skip caching the results from the processors. **Type**: `string` ```yaml # Examples: skip_on: errored() ``` ### [](#ttl)`ttl` An optional expiry period to set for each cache entry. Some caches only have a general TTL and will therefore ignore this setting. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 387: catch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch.md --- # catch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: catch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/catch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/catch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/catch.adoc description: Applies a list of child processors _only_ when a previous processing step has failed. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Applies a list of child processors _only_ when a previous processing step has failed. ```yml # Config fields, showing default values label: "" catch: [] ``` Behaves similarly to the [`for_each`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, processors are only applied to messages that failed a processing step prior to the catch. For example, with the following config: ```yaml pipeline: processors: - resource: foo - catch: - resource: bar - resource: baz ``` If the processor `foo` fails for a particular message, that message will be fed into the processors `bar` and `baz`. Messages that do not fail for the processor `foo` will skip these processors. When messages leave the catch block their fail flags are cleared. This processor is useful for when it’s possible to recover failed messages, or when special actions (such as logging/metrics) are required before dropping them. More information about error handling can be found in [Error Handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). --- # Page 388: cohere_chat **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cohere_chat.md --- # cohere_chat > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cohere_chat page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/cohere_chat page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/cohere_chat.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/cohere_chat.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-10-16" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Generates responses to messages in a chat conversation, using the [Cohere API](https://docs.cohere.com/docs/chat-api) and external tools. ### Common ```yml processors: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) response_format: text json_schema: "" # No default (optional) max_tool_calls: 10 tools: [] ``` ### Advanced ```yml processors: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) response_format: text json_schema: "" # No default (optional) schema_registry: url: "" # No default (required) subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: "" # No default (optional) frequency_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) seed: "" # No default (optional) stop: [] # No default (optional) max_tool_calls: 10 tools: [] ``` This processor sends the contents of user prompts to the Cohere API, which generates responses using all available context, including supplementary data provided by external tools. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` field to customize it. To learn more about chat completion, see the [Cohere API documentation](https://docs.cohere.com/docs/chat-api). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#frequency_penalty)`frequency_penalty` A number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#json_schema)`json_schema` The JSON schema to use when responding in `json_schema` format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the chat completion. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of tool calls the model can perform. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the Cohere large language model (LLM) you want to use. **Type**: `string` ```yaml # Examples: model: command-r-plus # --- model: command-r # --- model: command # --- model: command-light ``` ### [](#presence_penalty)`presence_penalty` A number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Choose the model’s output format. If `json_schema` is specified, then you must also configure a `json_schema` or `schema_registry`. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas from when responding in `json_schema` format. Schemas themselves must be in JSON format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Configure JSON Web Token (JWT) authentication for secure data transmission from your schema registry to this component. This feature is in beta and may change in future releases. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests to the schema registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` The refresh rate for fetching the latest schema. If not specified the schema does not refresh. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name to fetch the schema for. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` If specified, Redpanda Connect makes a best effort to sample deterministically. Repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed. **Type**: `int` ### [](#stop)`stop[]` Specify up to four sequences to stop the API from generating further tokens. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Choose a sampling temperature between `0` and `2`: - Higher values, such as `0.8` make the output more random. - Lower values, such as `0.2` make the output more focused and deterministic. Redpanda recommends adding a value for this field or `top_p`, but not both. **Type**: `float` ### [](#tools)`tools[]` External tools that the model can invoke, such as functions, APIs, or web browsing. You can define a series of processors that describe these tools, enabling the model to use agent-like behavior to decide when and how to invoke them to enhance response generation. **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `array` ### [](#top_p)`top_p` An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. For example, a `top_p` of `0.1` means only the tokens comprising the top 10% probability mass are sampled. Redpanda recommends adding a value for this field or `temperature`, but not both. **Type**: `float` ## [](#example)Example In this pipeline configuration, the Command R+ model executes a number of processors, which make a tool call to retrieve weather data for a specific city. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - cohere_chat: auth_token: my_cohere_api_token model: command-r-plus prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to look up the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: User-Agent: curl/8.11.1 # Returns a text string from the weather website output: stdout: {} ``` --- # Page 389: cohere_embeddings **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cohere_embeddings.md --- # cohere_embeddings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cohere_embeddings page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/cohere_embeddings page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/cohere_embeddings.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/cohere_embeddings.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-10-16" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Generates vector embeddings to represent input text, using the [Cohere API](https://docs.cohere.com/docs/embeddings). ```yml # Configuration fields, showing default values label: "" cohere_embeddings: base_url: https://api.cohere.com auth_token: "" # No default (required) model: embed-english-v3.0 # No default (required) text_mapping: "" # No default (optional) input_type: search_document dimensions: "" # No default (optional) ``` This processor sends text strings to your chosen large language model (LLM), which generates vector embeddings for them using the Cohere API. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` field to customize it. To learn more about vector embeddings, see the [Cohere API documentation](https://docs.cohere.com/docs/embeddings). ## [](#examples)Examples ### [](#store-embedding-vectors-in-qdrant)Store embedding vectors in Qdrant Compute embeddings for some generated data and store it within xrefs:component:outputs/qdrant.adoc\[Qdrant\] ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - cohere_embeddings: model: embed-english-v3 api_key: "${COHERE_API_KEY}" text_mapping: "root = this.text" output: qdrant: grpc_host: localhost:6334 collection_name: "example_collection" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#dimensions)`dimensions` The number of dimensions (numerical values) in each vector embedding generated by this processor. This parameter only supports [`embed-v4.0`](https://docs.cohere.com/v2/docs/embeddings) and newer models. **Type**: `int` ### [](#input_type)`input_type` The type of text input passed to the model. **Type**: `string` **Default**: `search_document` | Option | Summary | | --- | --- | | classification | Used for embeddings passed through a text classifier. | | clustering | Used for the embeddings run through a clustering algorithm. | | search_document | Used for embeddings stored in a vector database for search use-cases. | | search_query | Used for embeddings of search queries run against a vector DB to find relevant documents. | ### [](#model)`model` The name of the Cohere LLM you want to use. **Type**: `string` ```yaml # Examples: model: embed-english-v3.0 # --- model: embed-english-light-v3.0 # --- model: embed-multilingual-v3.0 # --- model: embed-multilingual-light-v3.0 ``` ### [](#text_mapping)`text_mapping` The text you want to generate a vector embedding for. By default, the processor submits the entire payload as a string. **Type**: `string` --- # Page 390: cohere_rerank **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cohere_rerank.md --- # cohere_rerank > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: cohere_rerank latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/cohere_rerank page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/cohere_rerank.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/cohere_rerank.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2026-05-26" --- Sends document strings to the [Cohere API](https://docs.cohere.com/reference/rerank), which returns them [ranked by their relevance to a specified query](https://docs.cohere.com/docs/rerank-2). The output of this processor is an array of strings, ordered by their relevance to the query. ```yml # Configuration fields, showing default values label: "" cohere_rerank: base_url: https://api.cohere.com api_key: "" # No default (required) model: rerank-v3.5 # No default (required) query: "" # No default (required) documents: "" # No default (required) top_n: 0 max_tokens_per_doc: 4096 ``` ## [](#metadata)Metadata - `relevance_scores`: An array of scores for each input document that indicates how relevant it is to the query. The scores are in the same order as the documents in the input. The higher the score, the more relevant the document. ## [](#examples)Examples ### [](#rerank-some-documents-based-on-a-query)Rerank some documents based on a query Rerank some documents based on a query ```yaml input: generate: interval: 1s mapping: | root = { "query": fake("sentence"), "docs": [fake("paragraph"), fake("paragraph"), fake("paragraph")], } pipeline: processors: - cohere_rerank: model: rerank-v3.5 api_key: "${COHERE_API_KEY}" query: "${!this.query}" documents: "root = this.docs" output: stdout: {} ``` ## [](#fields)Fields ### [](#api_key)`api_key` Your API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#documents)`documents` A list of text strings that are compared to the specified query. For optimal performance: - Send fewer than 1000 documents in a single request - Send structured data in YAML format **Type**: `string` ### [](#max_tokens_per_doc)`max_tokens_per_doc` This processor automatically truncates long documents to the specified number of tokens. **Type**: `int` **Default**: `4096` ### [](#model)`model` The name of the Cohere LLM you want to use. **Type**: `string` ```yaml # Examples: model: rerank-v3.5 ``` ### [](#query)`query` The search query you want to execute. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#top_n)`top_n` The number of documents to return when the query is executed. If set to `0`, all documents are returned. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `0` --- # Page 391: compress **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/compress.md --- # compress > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: compress latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/compress page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/compress.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/compress.adoc description: "Compresses messages according to the selected algorithm. Supported compression algorithms are: [flate gzip lz4 pgzip snappy zlib]." page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Compresses messages according to the selected algorithm. Supported compression algorithms are: \[flate gzip lz4 pgzip snappy zlib\] ```yml # Config fields, showing default values label: "" compress: algorithm: "" # No default (required) level: -1 ``` The 'level' field might not apply to all algorithms. ## [](#fields)Fields ### [](#algorithm)`algorithm` The compression algorithm to use. **Type**: `string` **Options**: `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` ### [](#level)`level` The level of compression to use. May not be applicable to all algorithms. **Type**: `int` **Default**: `-1` --- # Page 392: decompress **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/decompress.md --- # decompress > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: decompress latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/decompress page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/decompress.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/decompress.adoc description: "Decompresses messages according to the selected algorithm. Supported decompression algorithms are: [bzip2 flate gzip lz4 pgzip snappy zlib]." page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Decompresses messages according to the selected algorithm. Supported decompression algorithms are: \[bzip2 flate gzip lz4 pgzip snappy zlib\] ```yml # Config fields, showing default values label: "" decompress: algorithm: "" # No default (required) ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` The decompression algorithm to use. **Type**: `string` **Options**: `bzip2`, `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` --- # Page 393: dedupe **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/dedupe.md --- # dedupe > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: dedupe latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/dedupe page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/dedupe.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/dedupe.adoc description: Deduplicates messages by storing a key value in a cache using the add operator. If the key already exists within the cache it is dropped. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Deduplicates messages by storing a key value in a cache using the `add` operator. If the key already exists within the cache it is dropped. ```yml # Config fields, showing default values label: "" dedupe: cache: "" # No default (required) key: ${! meta("kafka_key") } # No default (required) drop_on_err: true ``` Caches must be configured as resources, for more information check out the [cache documentation](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/). When using this processor with an output target that might fail you should always wrap the output within an indefinite [`retry`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/retry/) block. This ensures that during outages your messages aren’t reprocessed after failures, which would result in messages being dropped. ## [](#batch-deduplication)Batch deduplication This processor enacts on individual messages only, in order to perform a deduplication on behalf of a batch (or window) of messages instead use the [`cache` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cache/#examples). ## [](#delivery-guarantees)Delivery guarantees Performing deduplication on a stream using a distributed cache voids any at-least-once guarantees that it previously had. This is because the cache will preserve message signatures even if the message fails to leave the Redpanda Connect pipeline, which would cause message loss in the event of an outage at the output sink followed by a restart of the Redpanda Connect instance (or a server crash, etc). This problem can be mitigated by using an in-memory cache and distributing messages to horizontally scaled Redpanda Connect pipelines partitioned by the deduplication key. However, in situations where at-least-once delivery guarantees are important it is worth avoiding deduplication in favour of implement idempotent behavior at the edge of your stream pipelines. ## [](#fields)Fields ### [](#cache)`cache` The [`cache` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) to target with this processor. **Type**: `string` ### [](#drop_on_err)`drop_on_err` Whether messages should be dropped when the cache returns a general error such as a network issue. **Type**: `bool` **Default**: `true` ### [](#key)`key` An interpolated string yielding the key to deduplicate by for each message. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! meta("kafka_key") } # --- key: ${! content().hash("xxhash64") } ``` ## [](#examples)Examples ### [](#deduplicate-based-on-kafka-key)Deduplicate based on Kafka key The following configuration demonstrates a pipeline that deduplicates messages based on the Kafka key. ```yaml pipeline: processors: - dedupe: cache: keycache key: ${! meta("kafka_key") } cache_resources: - label: keycache memory: default_ttl: 60s ``` --- # Page 394: for_each **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each.md --- # for_each > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: for_each latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/for_each page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/for_each.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/for_each.adoc description: A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message. ```yml # Config fields, showing default values label: "" for_each: [] ``` This is useful for forcing batch wide processors such as [`dedupe`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/dedupe/) or interpolations such as the `value` field of the `metadata` processor to execute on individual message parts of a batch instead. Please note that most processors already process per message of a batch, and this processor is not needed in those cases. --- # Page 395: gcp_bigquery_select **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/gcp_bigquery_select.md --- # gcp_bigquery_select > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_bigquery_select latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/gcp_bigquery_select page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/gcp_bigquery_select.adoc description: Executes a SELECT query against BigQuery and replaces messages with the rows returned. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a `SELECT` query against BigQuery and replaces messages with the rows returned. ```yml # Config fields, showing default values label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" # No default (optional) table: bigquery-public-data.samples.shakespeare # No default (required) columns: [] # No default (required) where: type = ? and created_at > ? # No default (optional) job_labels: {} args_mapping: root = [ "article", now().ts_format("2006-01-02") ] # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` ## [](#examples)Examples ### [](#word-count)Word count Given a stream of English terms, enrich the messages with the word count from Shakespeare’s public works: ```yaml pipeline: processors: - branch: processors: - gcp_bigquery_select: project: test-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: word = ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: root = [ this.term ] result_map: | root.count = this.get("0.total_count") ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A list of labels to add to the query job. **Type**: `object` **Default**: `{}` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#project)`project` GCP project where the query job will execute. **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` Fully-qualified BigQuery table name to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 396: gcp_vertex_ai_chat **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/gcp_vertex_ai_chat.md --- # gcp_vertex_ai_chat > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_vertex_ai_chat latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/gcp_vertex_ai_chat page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/gcp_vertex_ai_chat.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/gcp_vertex_ai_chat.adoc description: Generates responses to messages in a chat conversation, using the Vertex AI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates responses to messages in a chat conversation, using the [Vertex API AI](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform). #### Common ```yml processors: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: "" # No default (optional) max_tokens: "" # No default (optional) response_format: text tools: [] ``` #### Advanced ```yml processors: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: "" # No default (optional) max_tokens: "" # No default (optional) response_format: text top_p: "" # No default (optional) top_k: "" # No default (optional) stop: [] # No default (optional) presence_penalty: "" # No default (optional) frequency_penalty: "" # No default (optional) max_tool_calls: 10 tools: [] ``` This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the Vertex AI API. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs). ## [](#fields)Fields ### [](#attachment)`attachment` Additional data like an image to send with the prompt to the model. The result of the mapping must be a byte array, and the content type is automatically detected. **Type**: `string` ```yaml # Examples: attachment: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#credentials_json)`credentials_json` An optional field to set a Google Service Account Credentials JSON. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Historical messages to include in the chat request. The result of the bloblang query should be an array of objects of the form of \[{"role": "", "content":""}\], where role is "user" or "model". **Type**: `string` ### [](#location)`location` Specify the location of a fine tuned model. For base models, you can omit this field. **Type**: `string` ```yaml # Examples: location: us-central1 ``` ### [](#max_tokens)`max_tokens` The maximum number of output tokens to generate per message. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential tool calls. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the LLM to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: gemini-1.5-pro-001 # --- model: gemini-1.5-flash-001 ``` ### [](#presence_penalty)`presence_penalty` Positive values penalize new tokens if they appear in the text already, increasing the model’s likelihood to include new topics. **Type**: `float` ### [](#project)`project` The GCP project ID to use. **Type**: `string` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` The format of the generated response. You must also prompt the model to output the appropriate response type. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#stop)`stop[]` Sets the stop sequences to use. When this pattern is encountered the LLM stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the Vertex AI LLM. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Controls the randomness of predictions. **Type**: `float` ### [](#tools)`tools[]` The tools to allow the LLM to invoke. This allows building subpipelines that the LLM can choose to invoke to execute agentic-like actions. **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `array` ### [](#top_k)`top_k` Enables top-k sampling (optional). **Type**: `float` ### [](#top_p)`top_p` Enables nucleus sampling (optional). **Type**: `float` --- # Page 397: gcp_vertex_ai_embeddings **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/gcp_vertex_ai_embeddings.md --- # gcp_vertex_ai_embeddings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_vertex_ai_embeddings page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/gcp_vertex_ai_embeddings page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/gcp_vertex_ai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/gcp_vertex_ai_embeddings.adoc # Beta release status page-beta: "true" page-git-created-date: "2024-10-16" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- Generates vector embeddings to represent a text string, using the [Vertex AI API](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings). ```yml # Configuration fields, showing default values label: "" gcp_vertex_ai_embeddings: project: "" # No default (required) credentials_json: "" # No default (optional) location: us-central1 model: text-embedding-004 # No default (required) task_type: RETRIEVAL_DOCUMENT text: "" # No default (optional) output_dimensions: 0 # No default (optional) ``` This processor sends text strings to the Vertex AI API, which generates vector embeddings for them. By default, the processor submits the entire payload of each message as a string, unless you use the `text` field to customize it. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings). ## [](#fields)Fields ### [](#credentials_json)`credentials_json` Set your Google Service Account Credentials as JSON. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#location)`location` The location of the Vertex AI large language model (LLM) that you want to use. **Type**: `string` **Default**: `us-central1` ### [](#model)`model` The name of the LLM to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: text-embedding-004 # --- model: text-multilingual-embedding-002 ``` ### [](#output_dimensions)`output_dimensions` The maximum length of a generated vector embedding. If this value is set, generated embeddings are truncated to this size. **Type**: `int` ### [](#project)`project` The ID of your Google Cloud project. **Type**: `string` ### [](#task_type)`task_type` Use the following options to optimize embeddings that the model generates for specific use cases. **Type**: `string` **Default**: `RETRIEVAL_DOCUMENT` | Option | Summary | | --- | --- | | CLASSIFICATION | optimize for being able classify texts according to preset labels | | CLUSTERING | optimize for clustering texts based on their similarities | | FACT_VERIFICATION | optimize for queries that are proving or disproving a fact such as "apples grow underground" | | QUESTION_ANSWERING | optimize for search proper questions such as "Why is the sky blue?" | | RETRIEVAL_DOCUMENT | optimize for documents that will be searched (also known as a corpus) | | RETRIEVAL_QUERY | optimize for queries such as "What is the best fish recipe?" or "best restaurant in Chicago" | | SEMANTIC_SIMILARITY | optimize for text similarity | ### [](#text)`text` The text you want to generate vector embeddings for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 398: google_drive_download **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/google_drive_download.md --- # google_drive_download > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: google_drive_download latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/google_drive_download page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/google_drive_download.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/google_drive_download.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2026-05-26" --- Downloads files from Google Drive that contain matching file IDs. Try out the [example pipeline on this page](#example), which downloads all files from your Google Drive. ### Common ```yml processors: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) shared_drives: false ``` ### Advanced ```yml processors: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) export_mime_types: application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" shared_drives: false ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#export_mime_types)`export_mime_types` Maps Google Drive MIME types to [supported file export formats](https://developers.google.com/workspace/drive/api/guides/ref-export-formats). The MIME type is the key, and the export format is the value. **Type**: `object` **Default**: ```yaml application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" ``` ```yaml # Examples: export_mime_types: application/vnd.google-apps.document: application/pdf application/vnd.google-apps.drawing: application/pdf application/vnd.google-apps.presentation: application/pdf application/vnd.google-apps.spreadsheet: application/pdf # --- export_mime_types: application/vnd.google-apps.document: application/vnd.openxmlformats-officedocument.wordprocessingml.document application/vnd.google-apps.drawing: image/svg+xml application/vnd.google-apps.presentation: application/vnd.openxmlformats-officedocument.presentationml.presentation application/vnd.google-apps.spreadsheet: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet ``` ### [](#file_id)`file_id` The ID of the file to download from Google Drive. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#mime_type)`mime_type` The [MIME type](https://developers.google.com/workspace/drive/api/guides/mime-types) of the file for download. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives. **Type**: `bool` **Default**: `false` ## [](#example)Example This example downloads all files from a Google Drive. ```yaml input: stdin: {} pipeline: processors: - google_drive_search: query: "${!content().string()}" - mutation: 'meta path = this.name' - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" output: file: path: "${!@path}" codec: all-bytes ``` --- # Page 399: google_drive_list_labels **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/google_drive_list_labels.md --- # google_drive_list_labels > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: google_drive_list_labels latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/google_drive_list_labels page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/google_drive_list_labels.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/google_drive_list_labels.adoc description: Lists labels for a file in Google Drive. page-git-created-date: "2025-05-19" page-git-modified-date: "2026-05-26" --- Lists [labels](https://developers.google.com/workspace/drive/api/guides/about-labels) for files on a Google Drive. ```yml # Configuration fields, showing default values label: "" google_drive_list_labels: credentials_json: "" # No default (optional) ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` --- # Page 400: google_drive_search **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/google_drive_search.md --- # google_drive_search > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: google_drive_search latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/google_drive_search page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/google_drive_search.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/google_drive_search.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2026-05-26" --- Searches Google Drive for files that match a specified query and emits the results as a batch of messages. Each message contains the [metadata of a Google Drive file](https://developers.google.com/workspace/drive/api/reference/rest/v3/files#File). Try out the [example pipeline on this page](#example), which searches for and downloads all Google Drive files that match the specified query. ```yml # Configuration fields, showing default values label: "" google_drive_search: credentials_json: "" # No default (optional) query: "" # No default (required) projection: - id - name - mimeType - size - labelInfo include_label_ids: "" # No default (optional) max_results: 64 ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#include_label_ids)`include_label_ids` A comma delimited list of label IDs to include in the Google Drive search result. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_results)`max_results` The maximum number of search results to return. **Type**: `int` **Default**: `64` ### [](#projection)`projection[]` Partial fields to include in the Google Drive search result. **Type**: `array` **Default**: ```yaml - "id" - "name" - "mimeType" - "size" - "labelInfo" ``` ### [](#query)`query` Specify a search query to locate matching files in Google Drive. This field supports: - The same query syntax as the Google Drive UI - [Bloblang interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) for dynamic query generation **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives in the result. **Type**: `bool` **Default**: `false` ## [](#example)Example This example searches Google Drive for files matching a query and downloads each file to a specified location. It uses the `google_drive_search` processor to perform the search and the [`google_drive_download` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/google_drive_download/) to retrieve the files. ```yaml input: stdin: {} pipeline: processors: - google_drive_search: query: "${!content().string()}" - mutation: 'meta path = this.name' - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" output: file: path: "${!@path}" codec: all-bytes ``` --- # Page 401: group_by_value **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by_value.md --- # group_by_value > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: group_by_value latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/group_by_value page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/group_by_value.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/group_by_value.adoc description: Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a function interpolated string evaluated per message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a [function interpolated string](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) evaluated per message. ```yml # Config fields, showing default values label: "" group_by_value: value: ${! meta("kafka_key") } # No default (required) ``` This allows you to group messages using arbitrary fields within their content or metadata, process them individually, and send them to unique locations as per their group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#value)`value` The interpolated string to group based on. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: value: ${! meta("kafka_key") } # --- value: ${! json("foo.bar") }-${! meta("baz") } ``` ## [](#examples)Examples If we were consuming Kafka messages and needed to group them by their key, archive the groups, and send them to S3 with the key as part of the path we could achieve that with the following: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_key") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_key") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` --- # Page 402: group_by **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by.md --- # group_by > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: group_by latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/group_by page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/group_by.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/group_by.adoc description: Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a Bloblang query. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Splits a [batch of messages](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/) into N batches, where each resulting batch contains a group of messages determined by a [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/). ```yml # Config fields, showing default values label: "" group_by: [] # No default (required) ``` Once the groups are established a list of processors are applied to their respective grouped batch, which can be used to label the batch as per their grouping. Messages that do not pass the check of any specified group are placed in their own group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message belongs to a given group. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") # --- check: true ``` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to execute on the newly formed group. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a batch of messages that we wish to split into a group of foos and everything else, which should be sent to different output destinations based on those groupings. We also need to send the foos as a tar gzip archive. For this purpose we can use the `group_by` processor with a [`switch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/switch/) output: ```yaml pipeline: processors: - group_by: - check: content().contains("this is a foo") processors: - archive: format: tar - compress: algorithm: gzip - mapping: 'meta grouping = "foo"' output: switch: cases: - check: meta("grouping") == "foo" output: gcp_pubsub: project: foo_prod topic: only_the_foos - output: gcp_pubsub: project: somewhere_else topic: no_foos_here ``` --- # Page 403: http **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/http.md --- # http > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: http latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/http page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/http.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/http.adoc page-git-created-date: "2025-03-04" page-git-modified-date: "2026-05-26" --- Performs a HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response. #### Common ```yml processors: label: "" http: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s parallel: false ``` #### Advanced ```yml processors: label: "" http: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false batch_as_multipart: false parallel: false ``` ## [](#rate-limit-requests)Rate limit requests You can use the `rate_limit` field to specify a [rate limit resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/), which restricts the number of requests processed service-wide, regardless of how many components you run in parallel. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#map-payloads-with-the-branch-processor)Map payloads with the branch processor You can use the [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) to transform or encode the payload into a specific request body format, and map the response back into the original payload instead of replacing it entirely. This example uses a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) to strip the request message into an empty body (`request_map: 'root = ""'`), grab an HTTP payload, and place the result back into the original message at the path `repo.status`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: 'root.repo.status = this' ``` ## [](#response-codes)Response codes HTTP response codes in the 200-299 range indicate a successful response. You can use the [`successful_on`](#successful_on) field to add more success status codes. HTTP status codes in the 300-399 range are redirects. The [`follow_redirects` field](#follow_redirects) determines how these responses are handled. If a request returns a response code that matches an entry in: - The [`backoff_on` field](#backoff_on), the request is retried after increasing intervals. - The [`drop_on` field](#drop_on), the request is immediately treated as a failure. ## [](#add-metadata-to-errors)Add metadata to errors If a request returns an error response code, this processor sets a `http_status_code` metadata field in the resulting message. > 💡 **TIP** > > You can use the [`extract_headers`](#extract_headers) field to define rules for copying headers into messages generated from the response. ## [](#error-handling)Error handling When all retry attempts for a message are exhausted, this processor cancels the attempt. By default, the failed message continues through the pipeline unchanged unless you configure other error-handling. For example, you might want to drop failed messages or route them to a dead letter queue. For more information, see [Error Handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure, and trigger retries with an increasing backoff period between attempts. **Type**: `array` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all message in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure, where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `array` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether to follow redirects, including all responses with HTTP status codes in the 300-399 range. If set to `false`, the response message includes only the body, status, and headers from the redirect response, and this processor does not make a request to the URL specified in the `Location` header. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys should be added to the HTTP request as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#parallel)`parallel` When processing batched messages, this field determines whether messages in the batch are sent in parallel. If set to `false`, messages are sent serially. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `array` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 404: insert_part **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/insert_part.md --- # insert_part > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: insert_part latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/insert_part page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/insert_part.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/insert_part.adoc description: Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end. ```yml # Config fields, showing default values label: "" insert_part: index: -1 content: "" ``` The index can be negative, and if so the message will be inserted from the end counting backwards starting from -1. E.g. if index = -1 then the new message will become the last of the batch, if index = -2 then the new message will be inserted before the last message, and so on. If the negative index is greater than the length of the existing batch it will be inserted at the beginning. The new message will have metadata copied from the first pre-existing message of the batch. This processor will interpolate functions within the 'content' field, you can find a list of functions [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#content)`content` The content of the message being inserted. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#index)`index` The index within the batch to insert the message at. **Type**: `int` **Default**: `-1` --- # Page 405: jira **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/jira.md --- # jira > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: jira latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/jira page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/jira.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/jira.adoc description: Queries Jira resources and returns structured data. page-git-created-date: "2025-11-03" page-git-modified-date: "2026-08-11" --- > ⚠️ **WARNING: Deprecated in 4.100.0** > > Deprecated in 4.100.0 > > This component is deprecated and will be removed in the next major version release. To stream Jira issues, comments, or changelog entries, use the [`jira` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/jira/) instead. Queries Jira resources and returns structured data. #### Common ```yaml processors: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s ``` #### Advanced ```yaml processors: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. Supports querying the following Jira resources: - Issues (JQL queries) - Issue transitions - Users - Roles - Project versions - Project categories - Project types - Projects The processor authenticates using basic authentication with username and API token. Input messages should contain valid Jira queries in JSON format. ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#api_token)`api_token` The Jira API token for the specified account. You can generate an API token from your [Atlassian account settings](https://id.atlassian.com/manage-profile/security/api-tokens). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` The base URL of the Jira instance (for example, `[https://your-domain.atlassian.net](https://your-domain.atlassian.net)`). **Type**: `string` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#max_results_per_page)`max_results_per_page` The maximum number of results to return per page when calling the Jira API. [Pagination](https://docs.atlassian.com/software/jira/docs/api/REST/9.17.0/#pagination) in the Jira API is zero-based, so the first page starts at `0`. **Type**: `int` **Default**: `50` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#username)`username` The username or email address of the Jira account. **Type**: `string` ## [](#examples)Examples ### [](#minimal-configuration)Minimal configuration Basic Jira processor setup with required fields only ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" ``` ### [](#full-configuration-with-tuning)Full configuration with tuning Complete configuration with pagination and timeout settings ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 200 timeout: "30s" ``` --- # Page 406: jmespath **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/jmespath.md --- # jmespath > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: jmespath latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/jmespath page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/jmespath.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/jmespath.adoc description: Executes a http://jmespath.org/[JMESPath query] on JSON documents and replaces the message with the resulting document. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a [JMESPath query](http://jmespath.org/) on JSON documents and replaces the message with the resulting document. ```yml # Config fields, showing default values label: "" jmespath: query: "" # No default (required) ``` > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try native Redpanda Connect mapping with the [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/). ## [](#fields)Fields ### [](#query)`query` The JMESPath query to apply to messages. **Type**: `string` nclude::connect:components:partial$examples/processors/jmespath.adoc\[\] --- # Page 407: jq **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/jq.md --- # jq > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: jq latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/jq page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/jq.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/jq.adoc description: Transforms and filters messages using jq queries. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Transforms and filters messages using jq queries. #### Common ```yml processors: label: "" jq: query: "" # No default (required) ``` #### Advanced ```yml processors: label: "" jq: query: "" # No default (required) raw: false output_raw: false ``` > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try out native Redpanda Connect mapping with the [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/). The provided query is executed on each message, targeting either the contents as a structured JSON value or as a raw string using the field `raw`, and the message is replaced with the query result. Message metadata is also accessible within the query from the variable `$metadata`. This processor uses the [gojq library](https://github.com/itchyny/gojq), and therefore does not require jq to be installed as a dependency. However, this also means there are some [differences in how these queries are executed](https://github.com/itchyny/gojq#difference-to-jq) versus the jq cli. If the query does not emit any value then the message is filtered, if the query returns multiple values then the resulting message will be an array containing all values. The full query syntax is described in [jq’s documentation](https://stedolan.github.io/jq/manual/). ## [](#error-handling)Error handling Queries can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#fields)Fields ### [](#output_raw)`output_raw` Whether to output raw text (unquoted) instead of JSON strings when the emitted values are string types. **Type**: `bool` **Default**: `false` ### [](#query)`query` The jq query to filter and transform messages with. **Type**: `string` ### [](#raw)`raw` Whether to process the input as a raw string instead of as JSON. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#mapping)Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - jq: query: '{Cities: .locations | map(select(.state == "WA").name) | sort | join(", ") }' ``` --- # Page 408: json_schema **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/json_schema.md --- # json_schema > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: json_schema latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/json_schema page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/json_schema.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/json_schema.adoc description: Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using error handling methods. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ```yml # Config fields, showing default values label: "" json_schema: schema: "" # No default (optional) schema_path: "" # No default (optional) ``` Please refer to the [JSON Schema website](https://json-schema.org/) for information and tutorials regarding the syntax of the schema. ## [](#fields)Fields ### [](#schema)`schema` A schema to apply. Use either this or the `schema_path` field. **Type**: `string` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. **Type**: `string` ## [](#examples)Examples With the following JSONSchema document: ```json { "$id": "https://example.com/person.schema.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "Person", "type": "object", "properties": { "firstName": { "type": "string", "description": "The person's first name." }, "lastName": { "type": "string", "description": "The person's last name." }, "age": { "description": "Age in years which must be equal to or greater than zero.", "type": "integer", "minimum": 0 } } } ``` And the following Redpanda Connect configuration: ```yaml pipeline: processors: - json_schema: schema_path: "file://path_to_schema.json" - catch: - log: level: ERROR message: "Schema validation failed due to: ${!error()}" - mapping: 'root = deleted()' # Drop messages that fail ``` If a payload being processed looked like: ```json {"firstName":"John","lastName":"Doe","age":-21} ``` Then a log message would appear explaining the fault and the payload would be dropped. --- # Page 409: log **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/log.md --- # log > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: log latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/log page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/log.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/log.adoc description: Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in Bloblang queries which allows you to log the contents and metadata of messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in [Bloblang queries](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) which allows you to log the contents and metadata of messages. ```yml # Config fields, showing default values label: "" log: level: INFO fields_mapping: |- # No default (optional) root.reason = "cus I wana" root.id = this.id root.age = this.user.age.number() root.kafka_topic = meta("kafka_topic") message: "" ``` The `level` field determines the log level of the printed events and can be any of the following values: TRACE, DEBUG, INFO, WARN, ERROR. ## [](#structured-fields)Structured fields It’s also possible add custom fields to logs when the format is set to a structured form such as `json` or `logfmt` with the config field [`fields_mapping`](#fields_mapping): ```yaml pipeline: processors: - log: level: DEBUG message: hello world fields_mapping: | root.reason = "cus I wana" root.id = this.id root.age = this.user.age root.kafka_topic = meta("kafka_topic") ``` ## [](#fields)Fields ### [](#fields_mapping)`fields_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that can be used to specify extra fields to add to the log. If log fields are also added with the `fields` field then those values will override matching keys from this mapping. **Type**: `string` ```yaml # Examples: fields_mapping: |- root.reason = "cus I wana" root.id = this.id root.age = this.user.age.number() root.kafka_topic = meta("kafka_topic") ``` ### [](#level)`level` The log level to use. **Type**: `string` **Default**: `INFO` **Options**: `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE` ### [](#message)`message` The message to print. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 410: mapping **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping.md --- # mapping > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mapping latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/mapping page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/mapping.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/mapping.adoc description: Executes a Bloblang mapping on messages, creating a new document that replaces (or filters) the original message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) mapping on messages, creating a new document that replaces (or filters) the original message. ```yml # Config fields, showing default values label: "" mapping: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. Note: This processor is equivalent to the [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/bloblang/#component-rename) one. The latter will be deprecated in a future release. ## [](#input-document-immutability)Input document immutability Mapping operates by creating an entirely new object during assignments, this has the advantage of treating the original referenced document as immutable and therefore queryable at any stage of your mapping. For example, with the following mapping: ```bloblang root.id = this.id root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"id":"party-2024","invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we mutate the value of `invitees` in the resulting document by filtering out objects with a lower mood. However, even after doing so we’re still able to reference the unchanged original contents of this value from the input document in order to populate a second field. Within this mapping we also have the flexibility to reference the mutable mapped document by using the keyword `root` (i.e. `root.invitees`) on the right-hand side instead. Mapping documents is advantageous in situations where the result is a document with a dramatically different shape to the input document, since we are effectively rebuilding the document in its entirety and might as well keep a reference to the unchanged input document throughout. However, in situations where we are only performing minor alterations to the input document, the rest of which is unchanged, it might be more efficient to use the [`mutation` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mutation/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mapping: | root.id = this.id root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mapping: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 411: metric **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/metric.md --- # metric > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: metric latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/metric page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/metric.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/metric.adoc description: Emit custom metrics by extracting values from messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Emit custom metrics by extracting values from messages. ```yml # Config fields, showing default values label: "" metric: type: "" # No default (required) name: "" # No default (required) labels: {} # No default (optional) value: "" ``` This processor works by evaluating an [interpolated field `value`](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) for each message and updating a emitted metric according to the [type](#types). Custom metrics such as these are emitted along with Redpanda Connect internal metrics, where you can customize where metrics are sent, which metric names are emitted and rename them as/when appropriate. For more information see the [metrics docs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/). ## [](#fields)Fields ### [](#labels)`labels` A map of label names and values that can be used to enrich metrics. Labels are not supported by some metric destinations, in which case the metrics series are combined. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` ```yaml # Examples: labels: topic: ${! meta("kafka_topic") } type: ${! json("doc.type") } ``` ### [](#name)`name` The name of the metric to create, this must be unique across all Redpanda Connect components otherwise it will overwrite those other metrics. **Type**: `string` ### [](#type)`type` The metric [type](#types) to create. **Type**: `string` **Options**: `counter`, `counter_by`, `gauge`, `timing` ### [](#value)`value` For some metric types specifies a value to set, increment. Certain metrics exporters such as Prometheus support floating point values, but those that do not will cast a floating point value into an integer. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#counter)Counter In this example we emit a counter metric called `Foos`, which increments for every message processed, and we label the metric with some metadata about where the message came from and a field from the document that states what type it is. We also configure our metrics to emit to CloudWatch, and explicitly only allow our custom metric and some internal Redpanda Connect metrics to emit. ```yaml pipeline: processors: - metric: name: Foos type: counter labels: topic: ${! meta("kafka_topic") } partition: ${! meta("kafka_partition") } type: ${! json("document.type").or("unknown") } metrics: mapping: | root = if ![ "Foos", "input_received", "output_sent" ].contains(this) { deleted() } aws_cloudwatch: namespace: ProdConsumer ``` ### [](#gauge)Gauge In this example we emit a gauge metric called `FooSize`, which is given a value extracted from JSON messages at the path `foo.size`. We then also configure our Prometheus metric exporter to only emit this custom metric and nothing else. We also label the metric with some metadata. ```yaml pipeline: processors: - metric: name: FooSize type: gauge labels: topic: ${! meta("kafka_topic") } value: ${! json("foo.size") } metrics: mapping: 'if this != "FooSize" { deleted() }' prometheus: {} ``` ## [](#types)Types ### [](#counter-2)`counter` Increments a counter by exactly 1, the contents of `value` are ignored by this type. ### [](#counter_by)`counter_by` If the contents of `value` can be parsed as a positive integer value then the counter is incremented by this value. For example, the following configuration will increment the value of the `count.custom.field` metric by the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: counter_by name: CountCustomField value: ${!json("field.some.value")} ``` ### [](#gauge-2)`gauge` If the contents of `value` can be parsed as a positive integer value then the gauge is set to this value. For example, the following configuration will set the value of the `gauge.custom.field` metric to the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: gauge name: GaugeCustomField value: ${!json("field.some.value")} ``` ### [](#timing)`timing` Equivalent to `gauge` where instead the metric is a timing. It is recommended that timing values are recorded in nanoseconds in order to be consistent with standard Redpanda Connect timing metrics, as in some cases these values are automatically converted into other units such as when exporting timings as histograms with Prometheus metrics. --- # Page 412: mongodb **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mongodb.md --- # mongodb > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mongodb latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/mongodb page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/mongodb.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/mongodb.adoc description: Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads. #### Common ```yml processors: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false ``` #### Advanced ```yml processors: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false json_marshal_mode: canonical ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, `update-one`, and `aggregate`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. This output uses `filter_map` to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#json_marshal_mode)`json_marshal_mode` Controls the format of the output message (optional). **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation. That is, conversion from relaxed format to BSON can lose type information. | ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `insert-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one`, `find-one`, `aggregate` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username required to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` requests acknowledgement from MongoDB, which is created when write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` requests acknowledgement, which write operations propagate to the specified number of MongoDB instances. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 413: mutation **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mutation.md --- # mutation > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: mutation latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/mutation page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/mutation.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/mutation.adoc description: Executes a Bloblang mapping and directly transforms the contents of messages, mutating (or deleting) them. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) mapping and directly transforms the contents of messages, mutating (or deleting) them. ```yml # Config fields, showing default values label: "" mutation: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#input-document-mutability)Input document mutability A mutation is a mapping that transforms input documents directly, this has the advantage of reducing the need to copy the data fed into the mapping. However, this also means that the referenced document is mutable and therefore changes throughout the mapping. For example, with the following Bloblang: ```bloblang root.rejected = this.invitees.filter(i -> i.mood < 0.5) root.invitees = this.invitees.filter(i -> i.mood >= 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we create a field `rejected` by copying the array field `invitees` and filtering out objects with a high mood. We then overwrite the field `invitees` by filtering out objects with a low mood, resulting in two array fields that are each a subset of the original. If we were to reverse the ordering of these assignments like so: ```bloblang root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Then the new field `rejected` would be empty as we have already mutated `invitees` to exclude the objects that it would be populated by. We can solve this problem either by carefully ordering our assignments or by capturing the original array using a variable (`let invitees = this.invitees`). Mutations are advantageous over a standard mapping in situations where the result is a document with mostly the same shape as the input document, since we can avoid unnecessarily copying data from the referenced input document. However, in situations where we are creating an entirely new document shape it can be more convenient to use the traditional [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the error is logged and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mutation: | root.description = deleted() root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mutation: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 414: nats_kv **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/nats_kv.md --- # nats_kv > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_kv latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/nats_kv page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/nats_kv.adoc description: Perform operations on a NATS key-value bucket. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Perform operations on a NATS key-value bucket. #### Common ```yml processors: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) ``` #### Advanced ```yml processors: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) revision: "" # No default (optional) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#kv-operations)KV operations The NATS KV processor supports many KV operations using the [`operation`](#operation) field. Along with `get`, `put`, and `delete`, this processor supports atomic operations like `update` and `create`, as well as utility operations like `purge`, `history`, and `keys`. ## [](#metadata)Metadata This processor adds the following metadata fields to each message, depending on the chosen `operation`: ### [](#get-get_revision)get, get_revision - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_delta` - `nats_kv_operation` - `nats_kv_created` ### [](#create-update-delete-purge)create, update, delete, purge - `nats_kv_key` - `nats_kv_bucket` - `nats_kv_revision` - `nats_kv_operation` ### [](#keys)keys - `nats_kv_bucket` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. Supports [wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) for the `history` and `keys` operations. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.* # --- key: foo.> # --- key: foo.${! json("meta.type") } ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#operation)`operation` The operation to perform on the KV bucket. **Type**: `string` | Option | Summary | | --- | --- | | create | Adds the key/value pair if it does not exist. Returns an error if it already exists. | | delete | Deletes the key/value pair, but keeps historical values. | | get | Returns the latest value for key. | | get_revision | Returns the value of key for the specified revision. | | history | Returns historical values of key as an array of objects containing the following fields: key, value, bucket, revision, delta, operation, created. | | keys | Returns the keys in the bucket which match the keys_filter as an array of strings. | | purge | Deletes the key/value pair and all historical values. | | put | Places a new value for the key into the store. | | update | Updates the value for key only if the revision matches the latest revision. | ### [](#revision)`revision` The revision of the key to operate on. Used for `get_revision` and `update` operations. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: revision: 42 # --- revision: ${! @nats_kv_revision } ``` ### [](#timeout)`timeout` The maximum period to wait on an operation before aborting and returning an error. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 415: nats_request_reply **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/nats_request_reply.md --- # nats_request_reply > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: nats_request_reply latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/nats_request_reply page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/nats_request_reply.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/nats_request_reply.adoc description: Sends a message to a NATS subject and expects a reply, from a NATS subscriber acting as a responder, back. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sends a message to a NATS subject and expects a reply back from a NATS subscriber acting as a responder. #### Common ```yml processors: label: "" nats_request_reply: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s ``` #### Advanced ```yml processors: label: "" nats_request_reply: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) inbox_prefix: "" # No default (optional) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `nats_subject` - `nats_sequence_stream` - `nats_sequence_consumer` - `nats_num_delivered` - `nats_num_pending` - `nats_domain` - `nats_timestamp_unix_nano` You can access these metadata fields using [function interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `object` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inbox_prefix)`inbox_prefix` Set an explicit inbox prefix for the response subject **Type**: `string` ```yaml # Examples: inbox_prefix: _INBOX_joe ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata-2)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#timeout)`timeout` A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as 300ms, -1.5h or 2h45m. Valid time units are ns, us (or µs), ms, s, m, h. **Type**: `string` **Default**: `3s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 416: noop **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/noop.md --- # noop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: noop latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/noop page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/noop.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/noop.adoc description: Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option. ```yml # Config fields, showing default values label: "" noop: {} ``` --- # Page 417: ollama_chat **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_chat.md --- # ollama_chat > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: ollama_chat latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/ollama_chat page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/ollama_chat.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/ollama_chat.adoc description: Generates responses to messages in a chat conversation, using the Ollama API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Ollama connectors are currently only available on BYOC GCP clusters. > ⚠️ **CAUTION** > > When Redpanda Connect runs a data pipeline with a Ollama processor in it, Redpanda Cloud deploys a GPU-powered instance for the exclusive use of that pipeline. As pricing is based on resource consumption, this can have cost implications. Generates responses to messages in a chat conversation using the Ollama API and external tools. #### Common ```yml processors: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: "" # No default (optional) temperature: "" # No default (optional) save_prompt_metadata: false history: "" # No default (optional) tools: [] runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: "" # No default (optional) temperature: "" # No default (optional) num_keep: "" # No default (optional) seed: "" # No default (optional) top_k: "" # No default (optional) top_p: "" # No default (optional) repeat_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) frequency_penalty: "" # No default (optional) stop: [] # No default (optional) save_prompt_metadata: false history: "" # No default (optional) max_tool_calls: 3 tools: [] runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor sends prompts to your chosen Ollama large language model (LLM) and generates text from the responses using the Ollama API and external tools. By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [examples](#examples). ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set - the directory to download the Ollama binary and use as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set - the URL to download the Ollama binary from. Defaults to the official Ollama GitHub release for this platform. **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Include historical messages in a chat request. You must use a Bloblang query to create an array of objects in the form of `[{"role": "", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, `assistant`, or `tool`. - `content` is the text of the original messages. **Type**: `string` ### [](#image)`image` An optional image to submit along with the [`prompt`](#prompt) value. The result is a byte array. **Type**: `string` ```yaml # Examples: image: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#max_tokens)`max_tokens` The maximum number of tokens to predict and output. Limiting the amount of output means that requests are processed faster and have a fixed limit on the cost. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential calls you can make to external tools to retrieve additional information to answer a prompt. **Type**: `int` **Default**: `3` ### [](#model)`model` The name of the Ollama LLM to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: llama3.1 # --- model: gemma2 # --- model: qwen2 # --- model: phi3 ``` ### [](#num_keep)`num_keep` Specify the number of tokens from the initial prompt to retain when the model resets its internal context. By default, this value is set to `4`. Use `-1` to retain all tokens from the initial prompt. **Type**: `int` ### [](#presence_penalty)`presence_penalty` Positive values penalize new tokens if they have appeared in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#repeat_penalty)`repeat_penalty` Sets how strongly to penalize repetitions. A higher value, for example 1.5, will penalize repetitions more strongly. A lower value, for example 0.9, will be more lenient. **Type**: `float` ### [](#response_format)`response_format` The format of the response the Ollama model generates. If specifying JSON output, then the `prompt` should specify that the output should be in JSON as well. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` This option allows offloading some layers to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Set the number of threads to use during generation. For optimal performance, it is recommended to set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. This is only support on unix systems and allows loading only the necessary parts of the model as needed. **Type**: `bool` ### [](#save_prompt_metadata)`save_prompt_metadata` Set to `true` to save the prompt value to a metadata field (`@prompt`) on the corresponding output message. If you use the `system_prompt` field, its value is also saved to an `@system_prompt` metadata field on each output message. **Type**: `bool` **Default**: `false` ### [](#seed)`seed` Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. **Type**: `int` ```yaml # Examples: seed: 42 ``` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave the field blank and the processor starts and runs a local Ollama server or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#stop)`stop[]` Sets the stop sequences to use. When this pattern is encountered, the LLM stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the Ollama LLM. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` The temperature of the model. Increasing the temperature makes the model answer more creatively. **Type**: `int` ### [](#tools)`tools[]` The external tools the LLM can invoke, such as functions, APIs, or web browsing. You can build a series of processors that include definitions of these tools, and the specified LLM can choose when to invoke them to help answer a prompt. For more information, see [examples](#examples). **Type**: `array` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `array` ### [](#top_k)`top_k` Reduces the probability of generating nonsense. A higher value, for example `100`, will give more diverse answers. A lower value, for example `10`, will be more conservative. **Type**: `int` ### [](#top_p)`top_p` Works together with `top-k`. A higher value, for example 0.95, will lead to more diverse text. A lower value, for example 0.5, will generate more focused and conservative text. **Type**: `float` ## [](#examples)Examples ### [](#use-llava-to-analyze-an-image)Use Llava to analyze an image This example fetches image URLs from stdin and has a multimodal LLM describe the image. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - http: verb: GET url: "${!content().string()}" - ollama_chat: model: llava prompt: "Describe the following image" image: "root = content()" output: stdout: codec: lines ``` ### [](#use-subpipelines-as-tool-calls)Use subpipelines as tool calls This example allows llama3.2 to execute a subpipeline as a tool call to get more data. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - ollama_chat: model: llama3.2 prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to lookup the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: # Spoof curl user-ageent to get a plaintext text User-Agent: curl/8.11.1 output: stdout: {} ``` --- # Page 418: ollama_embeddings **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_embeddings.md --- # ollama_embeddings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: ollama_embeddings latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/ollama_embeddings page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/ollama_embeddings.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/ollama_embeddings.adoc description: Generates vector embeddings from text, using the Ollama API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Ollama connectors are currently only available on BYOC GCP clusters. > ⚠️ **CAUTION** > > When Redpanda Connect runs a data pipeline with a Ollama processor in it, Redpanda Cloud deploys a GPU-powered instance for the exclusive use of that pipeline. As pricing is based on resource consumption, this can have cost implications. Generates vector embeddings from text, using the Ollama API. #### Common ```yml processors: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor sends text to your chosen Ollama large language model (LLM) and creates vector embeddings, using the Ollama API. Vector embeddings are long arrays of numbers that represent values or objects, in this case text. By default, the processor starts and runs a locally installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs). ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set - the directory to download the ollama binary and use as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set - the URL to download the ollama binary from. Defaults to the official Ollama GitHub release for this platform. **Type**: `string` ### [](#model)`model` The name of the Ollama LLM to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: nomic-embed-text # --- model: mxbai-embed-large # --- model: snowflake-artic-embed # --- model: all-minilm ``` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to processor. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` This option allows offloading some layers to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Set the number of threads to use during generation. For optimal performance, it is recommended to set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. This is only support on unix systems and allows loading only the necessary parts of the model as needed. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave the field blank and the processor starts and runs a local Ollama server or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#text)`text` The text you want to create vector embeddings for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 419: ollama_moderation **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_moderation.md --- # ollama_moderation > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: ollama_moderation page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/ollama_moderation page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/ollama_moderation.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/ollama_moderation.adoc # Beta release status page-beta: "true" page-git-created-date: "2025-01-28" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- > 📝 **NOTE** > > Ollama connectors are currently only available on BYOC GCP clusters. > ⚠️ **CAUTION** > > When Redpanda Connect runs a data pipeline with a Ollama processor in it, Redpanda Cloud deploys a GPU-powered instance for the exclusive use of that pipeline. As pricing is based on resource consumption, this can have cost implications. Generates responses to messages in a chat conversation using the Ollama API, and checks the responses to make sure they do not violate [safety or security standards](https://mlcommons.org/2024/04/mlc-aisafety-v0-5-poc/). #### Common ```yml processors: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor checks the safety of responses from your chosen large language model (LLM) using either [Llama Guard 3](https://ollama.com/library/llama-guard3) or [ShieldGemma](https://ollama.com/library/shieldgemma). By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [Examples](#examples). To check the safety of your prompts, see the [`ollama_chat` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_chat/#examples) documentation. ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If the `server_address` is not set, download the Ollama binary to this directory and use it as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform. **Type**: `string` ### [](#model)`model` The name of the Ollama LLM to use. **Type**: `string` | Option | Summary | | --- | --- | | llama-guard3 | When using llama-guard3, two pieces of metadata is added: @safe with the value of yes or no and the second being @category for the safety category violation. For more information see the Llama Guard 3 Model Card. | | shieldgemma | When using shieldgemma, the model output is a single piece of metadata of @safe with a value of yes or no if the response is not in violation of its defined safety policies. | ```yaml # Examples: model: llama-guard3 # --- model: shieldgemma ``` ### [](#prompt)`prompt` The prompt you used to generate a response from an LLM. If you’re using the `ollama_chat` processor, you can set the `save_prompt_metadata` field to save the contents of your prompts. You can then run them through `ollama_moderation` processor to check the model responses for safety. For more details, see [Examples](#examples). You can also check the safety of your prompts. For more information, see the [`ollama_chat` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_chat/#examples) documentation. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response)`response` The LLM’s response that you want to check for safety. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. Set to `true` to load only the necessary parts of the model into memory. This setting is only supported on Unix systems. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ## [](#examples)Examples ### [](#use-llama-guard-3-classify-a-llm-response)Use Llama Guard 3 classify a LLM response This example uses Llama Guard 3 to check if another model responded with a safe or unsafe content. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - ollama_chat: model: llava prompt: "${!content().string()}" save_prompt_metadata: true - ollama_moderation: model: llama-guard3 prompt: "${!@prompt}" response: "${!content().string()}" - mapping: | root.response = content().string() root.is_safe = @safe output: stdout: codec: lines ``` --- # Page 420: openai_chat_completion **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_chat_completion.md --- # openai_chat_completion > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_chat_completion latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_chat_completion page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_chat_completion.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_chat_completion.adoc description: Generates responses to messages in a chat conversation, using the OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates responses to messages in a chat conversation, using the OpenAI API and external tools. #### Common ```yml processors: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) description: "" # No default (optional) schema: "" # No default (required) tools: [] # No default (required) ``` #### Advanced ```yml processors: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) description: "" # No default (optional) schema: "" # No default (required) schema_registry: url: "" # No default (required) name_prefix: schema_registry_id_ subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: "" # No default (optional) frequency_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) seed: "" # No default (optional) stop: [] # No default (optional) tools: [] # No default (required) ``` This processor sends user prompts to the OpenAI API, and the specified large language model (LLM) generates responses using all available context, including supplementary data provided by [external tools](#tools). By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about chat completion, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/chat-completions), and [Examples](#Examples). ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Include messages from a prior conversation. You must use a Bloblang query to create an array of objects in the form of `[{"role": "user", "content": ""}, {"role":"assistant", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, or `assistant`. - `content` is the text of the original messages. For more information, see [Examples](#Examples). **Type**: `string` ### [](#image)`image` An optional image to submit along with the prompt. The result of the Bloblang mapping must be a byte array. **Type**: `string` ```yaml # Examples: image: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#json_schema)`json_schema` The JSON schema used by the model when generating responses in `json_schema` format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs/supported-schemas). **Type**: `object` ### [](#json_schema-description)`json_schema.description` An optional description, which helps the model understand the schema’s purpose. **Type**: `string` ### [](#json_schema-name)`json_schema.name` The name of the JSON schema to use. **Type**: `string` ### [](#json_schema-schema)`json_schema.schema` The JSON schema for the model to use when generating the output. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to generate for chat completion. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: gpt-4o # --- model: gpt-4o-mini # --- model: gpt-4 # --- model: gpt4-turbo ``` ### [](#presence_penalty)`presence_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens if they have appeared in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The user prompt for which a response is generated. By default, the processor sends the entire payload as a string unless customized using this field. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Specify the configured [model’s](#model) output format. If you choose the `json_schema` option, you must also configure a `json_schema` or `schema_registry`. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas for model responses in `json_schema` format. Schemas must be in JSON format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs/supported-schemas). **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` The key/value pairs that identify the type of token and signing algorithm (optional). **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-name_prefix)`schema_registry.name_prefix` A prefix to add to the schema registry name. To form the complete schema registry name, the schema ID is appended as a suffix. **Type**: `string` **Default**: `schema_registry_id_` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests to the schema registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` How frequently to poll the schema registry for updates. If not specified, the schema does not refresh automatically. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name used to fetch the schema from the schema registry. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` When set to a specific number, Redpanda Connect attempts to generate consistent responses for requests that use the same prompt, seed, and parameters. **Type**: `int` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#stop)`stop[]` Specify up to four stop sequences to use. When the model encounters a stop pattern, it stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Choose a sampling temperature between `0` and `2`: - Higher values, such as `0.8` make the output more random. - Lower values, such as `0.2` make the output more focused and deterministic. Redpanda recommends adding a value for this field or [`top_p`](#top_p), but not both. **Type**: `float` ### [](#tools)`tools[]` External tools the model can invoke, such as functions, APIs, or web browsing. You can build a series of processors that include definitions of these tools, and the specified model can choose when to invoke them to help answer a prompt. For more information, see [Examples](#Examples). > 📝 **NOTE** > > If you don’t want to use external tools, enter an empty array `tools:[]`. **Type**: `array` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` **Default**: `[]` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `array` ### [](#top_p)`top_p` An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. For example, a `top_p` of `0.1` means only the tokens comprising the top 10% probability mass are sampled. Redpanda recommends adding a value for this field or `temperature`, but not both. **Type**: `float` ### [](#user)`user` A unique identifier that represents the end-user generating the prompt. This value can help OpenAI monitor and detect [platform abuse](https://openai.com/policies/usage-policies/). This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` nclude::connect:components:partial$examples/processors/openai\_chat\_completion.adoc\[\] --- # Page 421: openai_embeddings **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_embeddings.md --- # openai_embeddings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_embeddings latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_embeddings page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_embeddings.adoc description: Generates vector embeddings to represent input text, using the OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates vector embeddings to represent input text, using the OpenAI API. ```yml # Config fields, showing default values label: "" openai_embeddings: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: text-embedding-3-large # No default (required) text_mapping: "" # No default (optional) ``` This processor sends text strings to the OpenAI API, which generates vector embeddings. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` configuration field to customize it. To learn more about vector embeddings, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/embeddings). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#dimensions)`dimensions` The number of dimensions the resulting output embeddings should have. Only supported in `text-embedding-3` and later models. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: text-embedding-3-large # --- model: text-embedding-3-small # --- model: text-embedding-ada-002 ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#text_mapping)`text_mapping` The text you want to generate a vector embedding for. By default, the processor submits the entire payload as a string. **Type**: `string` --- # Page 422: openai_image_generation **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_image_generation.md --- # openai_image_generation > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_image_generation latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_image_generation page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_image_generation.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_image_generation.adoc description: Generates an image from a text description and other attributes, using OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates an image from a text description and other attributes, using OpenAI API. #### Common ```yml processors: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) quality: "" # No default (optional) size: "" # No default (optional) style: "" # No default (optional) ``` This processor sends an image description and other attributes, such as image size and quality to the OpenAI API, which generates an image. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about image generation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/images). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: dall-e-3 # --- model: dall-e-2 ``` ### [](#prompt)`prompt` A text description of the image you want to generate. The `prompt` field accepts a maximum of 1000 characters for `dall-e-2` and 4000 characters for `dall-e-3`. **Type**: `string` ### [](#quality)`quality` The quality of the image to generate. Use `hd` to create images with finer details and greater consistency across the image. This parameter is only supported for `dall-e-3` models. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: quality: standard # --- quality: hd ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#size)`size` The size of the generated image. Choose from `256x256`, `512x512`, or `1024x1024` for `dall-e-2`. Choose from `1024x1024`, `1792x1024`, or `1024x1792` for `dall-e-3` models. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: size: 1024x1024 # --- size: 512x512 # --- size: 1792x1024 # --- size: 1024x1792 ``` ### [](#style)`style` The style of the generated image. Choose from `vivid` or `natural`. Vivid causes the model to lean towards generating hyperreal and dramatic images. Natural causes the model to produce more natural, less hyperreal looking images. This parameter is only supported for `dall-e-3`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: style: vivid # --- style: natural ``` --- # Page 423: openai_speech **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_speech.md --- # openai_speech > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_speech latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_speech page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_speech.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_speech.adoc description: Generates audio from a text description and other attributes, using OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates audio from a text description and other attributes, using OpenAI API. #### Common ```yml processors: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) ``` #### Advanced ```yml processors: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) response_format: "" # No default (optional) ``` This processor sends a text description and other attributes, such as a voice type and format to the OpenAI API, which generates audio. By default, the processor submits the entire payload of each message as a string, unless you use the `input` configuration field to customize it. To learn more about turning text into spoken audio, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/text-to-speech). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#input)`input` A text description of the audio you want to generate. The `input` field accepts a maximum of 4096 characters. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: tts-1 # --- model: tts-1-hd ``` ### [](#response_format)`response_format` The format to generate audio in. Default is `mp3`. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: response_format: mp3 # --- response_format: opus # --- response_format: aac # --- response_format: flac # --- response_format: wav # --- response_format: pcm ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#voice)`voice` The type of voice to use when generating the audio. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: voice: alloy # --- voice: echo # --- voice: fable # --- voice: onyx # --- voice: nova # --- voice: shimmer ``` --- # Page 424: openai_transcription **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_transcription.md --- # openai_transcription > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_transcription latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_transcription page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_transcription.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_transcription.adoc description: Generates a transcription of spoken audio in the input language, using the OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Generates a transcription of spoken audio in the input language, using the OpenAI API. #### Common ```yml processors: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) ``` #### Advanced ```yml processors: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) language: "" # No default (optional) prompt: "" # No default (optional) ``` This processor sends an audio file object along with the input language to OpenAI API to generate a transcription. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about audio transcription, see the: [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#file)`file` The audio file object (not file name) to transcribe, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#language)`language` The language of the input audio. Supplying the input language in ISO-639-1 format improves accuracy and latency. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: language: en # --- language: fr # --- language: de # --- language: zh ``` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 425: openai_translation **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/openai_translation.md --- # openai_translation > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: openai_translation latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/openai_translation page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/openai_translation.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/openai_translation.adoc description: Translates spoken audio into English, using the OpenAI API. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Translates spoken audio into English, using the OpenAI API. #### Common ```yml processors: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) prompt: "" # No default (optional) ``` This processor sends an audio file object to OpenAI API to generate a translation. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about translation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#file)`file` The audio file object (not file name) to translate, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 426: parallel **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/parallel.md --- # parallel > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: parallel latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/parallel page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/parallel.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/parallel.adoc description: A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the for_each processor), but where each message is processed in parallel. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the [`for_each`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each/) processor), but where each message is processed in parallel. ```yml # Config fields, showing default values label: "" parallel: cap: 0 processors: [] # No default (required) ``` The field `cap`, if greater than zero, caps the maximum number of parallel processing threads. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching in [Message Batching](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#cap)`cap` The maximum number of messages to have processing at a given time. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to apply. **Type**: `array` --- # Page 427: parquet_decode **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/parquet_decode.md --- # parquet_decode > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: parquet_decode latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/parquet_decode page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/parquet_decode.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/parquet_decode.adoc description: Decodes Parquet files into a batch of structured messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Decodes [Parquet files](https://parquet.apache.org/docs/) into a batch of structured messages. ```yml # Configuration fields, showing default values label: "" parquet_decode: handle_logical_types: v1 ``` ## [](#fields)Fields ### [](#handle_logical_types)`handle_logical_types` Set to `v2` to enable enhanced decoding of logical types, or keep the default value (`v1`) to ignore logical type metadata when decoding values. In Parquet format, logical types are represented using standard physical types along with metadata that provides additional context. For example, UUIDs are stored as a `FIXED_LEN_BYTE_ARRAY` physical type, but the schema metadata identifies them as UUIDs. By enabling `v2`, this processor uses the metadata descriptions of logical types to produce more meaningful values during decoding. > 📝 **NOTE** > > For backward compatibility, this field enables logical-type handling for the specified Parquet format version, and all earlier versions. When creating new pipelines, Redpanda recommends that you use the newest documented version. **Type**: `string` **Default**: `v1` | Option | Summary | | --- | --- | | v1 | No special handling of logical types | | v2 | TIMESTAMP - decodes as an RFC3339 string describing the time. If the isAdjustedToUTC flag is set to true in the parquet file, the time zone will be set to UTC. If it is set to false the time zone will be set to local time.UUID - decodes as a string, i.e. 00112233-4455-6677-8899-aabbccddeeff. | ```yaml # Examples: handle_logical_types: v2 ``` ## [](#examples)Examples ### [](#reading-parquet-files-from-aws-s3)Reading Parquet Files from AWS S3 In this example we consume files from AWS S3 as they’re written by listening onto an SQS queue for upload events. We make sure to use the `to_the_end` scanner which means files are read into memory in full, which then allows us to use a `parquet_decode` processor to expand each file into a batch of messages. Finally, we write the data out to local files as newline delimited JSON. ```yaml input: aws_s3: bucket: TODO prefix: foos/ scanner: to_the_end: {} sqs: url: TODO processors: - parquet_decode: {} output: file: codec: lines path: './foos/${! meta("s3_key") }.jsonl' ``` --- # Page 428: parquet_encode **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/parquet_encode.md --- # parquet_encode > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: parquet_encode latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/parquet_encode page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/parquet_encode.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/parquet_encode.adoc description: Encodes Parquet files from a batch of structured messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Encodes [Parquet files](https://parquet.apache.org/docs/) from a batch of structured messages. #### Common ```yml processors: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed ``` #### Advanced ```yml processors: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed default_encoding: DELTA_LENGTH_BYTE_ARRAY default_timestamp_unit: NANOSECOND ``` ## [](#fields)Fields ### [](#default_compression)`default_compression` The default compression type to use for fields. **Type**: `string` **Default**: `uncompressed` **Options**: `uncompressed`, `snappy`, `gzip`, `brotli`, `zstd`, `lz4raw` ### [](#default_encoding)`default_encoding` The default encoding type to use for fields. A custom default encoding is only necessary when consuming data with libraries that do not support `DELTA_LENGTH_BYTE_ARRAY`. **Type**: `string` **Default**: `DELTA_LENGTH_BYTE_ARRAY` **Options**: `DELTA_LENGTH_BYTE_ARRAY`, `PLAIN` ### [](#default_timestamp_unit)`default_timestamp_unit` The precision used when encoding TIMESTAMP logical types. The default `NANOSECOND` matches historical behaviour, but `TIMESTAMP(NANOS)` is not readable by Apache Spark (Databricks), AWS Athena or DuckDB; set this to `MICROSECOND` (or `MILLISECOND`) when writing Parquet files intended for consumption by those engines. **Type**: `string` **Default**: `NANOSECOND` **Options**: `NANOSECOND`, `MICROSECOND`, `MILLISECOND` ### [](#schema)`schema[]` Parquet schema. **Type**: `array` ### [](#schema-fields)`schema[].fields[]` A list of child fields. **Type**: `array` ```yaml # Examples: fields: - name: foo type: INT64 - name: bar type: BYTE_ARRAY ``` ### [](#schema-name)`schema[].name` The name of the column. **Type**: `string` ### [](#schema-optional)`schema[].optional` Whether the field is optional. **Type**: `bool` **Default**: `false` ### [](#schema-repeated)`schema[].repeated` Whether the field is repeated. **Type**: `bool` **Default**: `false` ### [](#schema-type)`schema[].type` The type of the column, only applicable for leaf columns with no child fields. Some logical types can be specified here such as UTF8. **Type**: `string` **Options**: `BOOLEAN`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BYTE_ARRAY`, `UTF8`, `TIMESTAMP`, `BSON`, `ENUM`, `JSON`, `UUID` ### [](#schema_metadata)`schema_metadata` Optionally specify a metadata field containing a schema definition to use for encoding instead of a statically defined schema. For batches of messages, the first message’s schema will be applied to all subsequent messages of the batch. **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#writing-parquet-files-to-aws-s3)Writing Parquet Files to AWS S3 In this example we use the batching mechanism of an `aws_s3` output to collect a batch of messages in memory, which then converts it to a parquet file and uploads it. ```yaml output: aws_s3: bucket: TODO path: 'stuff/${! timestamp_unix() }-${! uuid_v4() }.parquet' batching: count: 1000 period: 10s processors: - parquet_encode: schema: - name: id type: INT64 - name: weight type: DOUBLE - name: content type: BYTE_ARRAY default_compression: zstd ``` --- # Page 429: parse_log **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/parse_log.md --- # parse_log > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: parse_log latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/parse_log page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/parse_log.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/parse_log.adoc description: Parses common log formats into structured data. This is easier and often much faster than grok. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Parses common log [Formats](#formats) into [structured data](#codecs). #### Common ```yml processors: label: "" parse_log: format: "" # No default (required) ``` #### Advanced ```yml processors: label: "" parse_log: format: "" # No default (required) best_effort: true allow_rfc3339: true default_year: current default_timezone: UTC ``` ## [](#fields)Fields ### [](#allow_rfc3339)`allow_rfc3339` Also accept timestamps in rfc3339 format while parsing. Applicable to format `syslog_rfc3164`. **Type**: `bool` **Default**: `true` ### [](#best_effort)`best_effort` Still returns partially parsed messages even if an error occurs. **Type**: `bool` **Default**: `true` ### [](#default_timezone)`default_timezone` Sets the strategy to decide the timezone for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. This value should follow the [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation) format. **Type**: `string` **Default**: `UTC` ### [](#default_year)`default_year` Sets the strategy used to set the year for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. When set to `current` the current year will be set, when set to an integer that value will be used. Leave this field empty to not set a default year at all. **Type**: `string` **Default**: `current` ### [](#format)`format` A common log [format](#formats) to parse. **Type**: `string` **Options**: `syslog_rfc5424`, `syslog_rfc3164` ## [](#codecs)Codecs Currently the only supported structured data codec is `json`. ## [](#formats)Formats ### [](#syslog_rfc5424)`syslog_rfc5424` Attempts to parse a log following the [Syslog RFC5424](https://tools.ietf.org/html/rfc5424) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `version` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) - `structureddata` (object) ### [](#syslog_rfc3164)`syslog_rfc3164` Attempts to parse a log following the [Syslog rfc3164](https://tools.ietf.org/html/rfc3164) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) --- # Page 430: processors **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/processors.md --- # processors > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: processors latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/processors page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/processors.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/processors.adoc description: A processor grouping several sub-processors. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A processor grouping several sub-processors. ```yml # Config fields, showing default values label: "" processors: [] ``` This processor is useful in situations where you want to collect several processors under a single resource identifier, whether it is for making your configuration easier to read and navigate, or for improving the testability of your configuration. The behavior of child processors will match exactly the behavior they would have under any other processors block. ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a collection of processors who cover a specific functionality. We could use this processor to group them together and make it easier to read and mock during testing by giving the whole block a label: ```yaml pipeline: processors: - label: my_super_feature processors: - log: message: "Let's do something cool" - archive: format: json_array - mapping: root.items = this ``` --- # Page 431: protobuf **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/protobuf.md --- # protobuf > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: protobuf latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/protobuf page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/protobuf.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/protobuf.adoc description: Performs conversions to or from a protobuf message. This processor uses reflection, meaning conversions can be made directly from the target .proto files. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Handles conversions between JSON documents and protobuf messages using reflection, which allows you to make conversions from or to the target `.proto` files. For more information about JSON mapping of protobuf messages, see [ProtoJSON Format](https://protobuf.dev/programming-guides/json/) and [Examples](#examples). ```yml # Configuration fields, showing default values label: "" protobuf: operator: "" # No default (required) message: "" # No default (required) discard_unknown: false use_proto_names: false import_paths: [] use_enum_numbers: false ``` ## [](#performance-considerations)Performance considerations Processing protobuf messages using reflection is less performant than using generated native code. For scenarios where performance is critical, consider using [Redpanda Connect plugins](https://github.com/benthosdev/benthos-plugin-example). ## [](#operators)Operators ### [](#to_json)`to_json` Converts protobuf messages into a generic JSON structure, which makes it easier to manipulate the contents of the JSON document within Redpanda Connect. ### [](#from_json)`from_json` Attempts to create a target protobuf message from a generic JSON structure. ## [](#fields)Fields ### [](#bsr)`bsr[]` Buf Schema Registry configuration. Either this field or `import_paths` must be populated. Note that this field is an array, and multiple BSR configurations can be provided. **Type**: `array` **Default**: `[]` ### [](#bsr-api_key)`bsr[].api_key` Buf Schema Registry server API key, can be left blank for a public registry. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#bsr-module)`bsr[].module` Module to fetch from a Buf Schema Registry e.g. 'buf.build/exampleco/mymodule'. **Type**: `string` ### [](#bsr-url)`bsr[].url` Buf Schema Registry URL, leave blank to extract from module. **Type**: `string` **Default**: `""` ### [](#bsr-version)`bsr[].version` Version to retrieve from the Buf Schema Registry, leave blank for latest. **Type**: `string` **Default**: `""` ### [](#discard_unknown)`discard_unknown` When set to `true`, the `from_json` operator discards fields that are unknown to the schema. **Type**: `bool` **Default**: `false` ### [](#import_paths)`import_paths[]` A list of directories that contain `.proto` files, including all definitions required for parsing the target message. If left empty, the current directory is used. This processor imports all `.proto` files listed within specified or default directories. **Type**: `array` **Default**: `[]` ### [](#message)`message` The fully-qualified name of the protobuf message to convert from or to JSON. **Type**: `string` ### [](#operator)`operator` The [operator](#operators) to execute. **Type**: `string` **Options**: `to_json`, `from_json`, `decode` ### [](#use_enum_numbers)`use_enum_numbers` When set to `true`, the `to_json` operator deserializes enumeration fields as their numerical values instead of their string names. For example, an enum field with a value of `ENUM_VALUE_ONE` is represented as `1` in the JSON output. **Type**: `bool` **Default**: `false` ### [](#use_proto_names)`use_proto_names` When set to `true`, the `to_json` operator deserializes fields exactly as named in schema file. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#json-to-protobuf-using-schema-from-disk)JSON to Protobuf using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#protobuf-to-json-using-schema-from-disk)Protobuf to JSON using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#json-to-protobuf-using-buf-schema-registry)JSON to Protobuf using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxx ``` ### [](#protobuf-to-json-using-buf-schema-registry)Protobuf to JSON using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxxx ``` --- # Page 432: qdrant **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/qdrant.md --- # qdrant > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: qdrant latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/qdrant page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/qdrant.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/qdrant.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2026-05-26" --- Query items within a [Qdrant collection](https://qdrant.tech/documentation/concepts/collections/) and filter the returned results. #### Common ```yml processors: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` #### Advanced ```yml processors: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token to use for authentication, which defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#collection_name)`collection_name` The name of the Qdrant collection you want to query. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#filter)`filter` Specify additional filtering to perform on returned results. Mappings must return [a valid filter](https://qdrant.tech/documentation/concepts/filtering/) using the proto3-encoded form. **Type**: `string` ```yaml # Examples: filter: |- root.must = [ {"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}}, {"field":{"key": "city", "match": {"text": "London"}}}, ] # --- filter: |- root.must = [ {"field":{"key": "city", "match": {"text": "London"}}}, ] root.must_not = [ {"field":{"color": "city", "match": {"text": "red"}}}, ] ``` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#limit)`limit` The maximum number of points to return from the collection. **Type**: `int` **Default**: `10` ### [](#payload_fields)`payload_fields[]` The fields to include or exclude in returned results. Use this field in combination with `payload_filter`. **Type**: `array` **Default**: `[]` ### [](#payload_filter)`payload_filter` Whether to include or exclude the fields specified in `payload_fields` from the returned results. **Type**: `string` **Default**: `include` | Option | Summary | | --- | --- | | exclude | Exclude the payload fields specified in payload_fields. | | include | Include the payload fields specified in payload_fields. | ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` A mapping to extract search vectors from the returned document. **Type**: `string` ```yaml # Examples: vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}} # --- vector_mapping: root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]} # --- vector_mapping: root = {"some_dense": [0.352,0.532,0.532,0.234]} ``` --- # Page 433: rate_limit **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/rate_limit.md --- # rate_limit > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: rate_limit latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/rate_limit page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/rate_limit.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/rate_limit.adoc description: Throttles the throughput of a pipeline according to a specified rate_limit resource. Rate limits are shared across components and therefore apply globally to all processing pipelines. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Throttles the throughput of a pipeline according to a specified [`rate_limit`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/) resource. Rate limits are shared across components and therefore apply globally to all processing pipelines. ```yml # Config fields, showing default values label: "" rate_limit: resource: "" # No default (required) ``` ## [](#fields)Fields ### [](#resource)`resource` The target [`rate_limit` resource](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about/). **Type**: `string` --- # Page 434: redis_script **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/redis_script.md --- # redis_script > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis_script latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/redis_script page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/redis_script.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/redis_script.adoc description: Performs actions against Redis using LUA scripts. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Performs actions against Redis using [LUA scripts](https://redis.io/docs/latest/develop/programmability/eval-intro/). #### Common ```yml processors: label: "" redis_script: url: "" # No default (required) script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) ``` #### Advanced ```yml processors: label: "" redis_script: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) retries: 3 retry_period: 500ms ``` Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/). ## [](#examples)Examples ### [](#running-a-script)Running a script The following example will use a script execution to get next element from a sorted set and set its score with timestamp unix nano value. ```yaml pipeline: processors: - redis_script: url: TODO script: | local value = redis.call("ZRANGE", KEYS[1], '0', '0') if next(elements) == nil then return '' end redis.call("ZADD", "XX", KEYS[1], ARGV[1], value) return value keys_mapping: 'root = [ meta("key") ]' args_mapping: 'root = [ timestamp_unix_nano() ]' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of arguments required for the specified Redis script. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), "hardcoded_value" ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#keys_mapping)`keys_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of keys matching in size to the number of arguments required for the specified Redis script. **Type**: `string` ```yaml # Examples: keys_mapping: root = [ this.key ] # --- keys_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#script)`script` A script to use for the target operator. It has precedence over the 'command' field. **Type**: `string` ```yaml # Examples: script: return redis.call('set', KEYS[1], ARGV[1]) ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 435: redis **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/redis.md --- # redis > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/redis page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/redis.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/redis.adoc description: Performs actions against Redis that aren't possible using a cache processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a branch processor. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Performs actions against Redis that aren’t possible using a [`cache`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cache/) processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/). #### Common ```yml processors: label: "" redis: url: "" # No default (required) command: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] command: "" # No default (optional) args_mapping: "" # No default (optional) retries: 3 retry_period: 500ms ``` ## [](#examples)Examples ### [](#querying-cardinality)Querying Cardinality If given payloads containing a metadata field `set_key` it’s possible to query and store the cardinality of the set for each message using a [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) in order to augment rather than replace the message contents: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: scard args_mapping: 'root = [ meta("set_key") ]' result_map: 'root.cardinality = this' ``` ### [](#running-total)Running Total If we have JSON data containing number of friends visited during covid 19: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2} {"name":"bob","month":"feb","year":2019,"friends_visited":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1} ``` We can add a field that contains the running total number of friends visited: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10,"total":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2,"total":8} {"name":"bob","month":"feb","year":2019,"friends_visited":3,"total":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1,"total":4} ``` Using the `incrby` command: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: incrby args_mapping: 'root = [ this.name, this.friends_visited ]' result_map: 'root.total = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of arguments required for the specified Redis command. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command to execute. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: command: scard # --- command: incrby # --- command: ${! meta("command") } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 436: resource **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/resource.md --- # resource > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: resource latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/resource page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/resource.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/resource.adoc description: Resource is a processor type that runs a processor resource identified by its label. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Resource is a processor type that runs a processor resource identified by its label. ```yml # Config fields, showing default values resource: "" ``` This processor allows you to reference the same configured processor resource in multiple places, and can also tidy up large nested configs. For example, the config: ```yaml pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` Is equivalent to: ```yaml pipeline: processors: - resource: foo_proc processor_resources: - label: foo_proc mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` --- # Page 437: retry **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/retry.md --- # retry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: retry latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/retry page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/retry.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/retry.adoc description: Attempts to execute a series of child processors until success. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Attempts to execute a series of child processors until success. ```yml # Config fields, showing default values label: "" retry: backoff: initial_interval: 500ms max_interval: 10s max_elapsed_time: 1m processors: [] # No default (required) parallel: false max_retries: 0 ``` Executes child processors and if a resulting message is errored then, after a specified backoff period, the same original message will be attempted again through those same processors. If the child processors result in more than one message then the retry mechanism will kick in if _any_ of the resulting messages are errored. It is important to note that any mutations performed on the message during these child processors will be discarded for the next retry, and therefore it is safe to assume that each execution of the child processors will always be performed on the data as it was when it first reached the retry processor. By default the retry backoff has a specified [`max_elapsed_time`](#backoffmax_elapsed_time), if this time period is reached during retries and an error still occurs these errored messages will proceed through to the next processor after the retry (or your outputs). Normal [error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/) can be used on these messages. In order to avoid permanent loops any error associated with messages as they first enter a retry processor will be cleared. ## [](#metadata)Metadata This processor adds the following metadata fields to each message: - `retry_count` - The number of retry attempts. - `backoff_duration` - The total time (in nanoseconds) elapsed while performing retries. > ⚠️ **CAUTION: Batching** > > Batching > > If you wish to wrap a batch-aware series of processors then take a look at the [batching section](#batching). ## [](#examples)Examples ### [](#stop-ignoring-me-taz)Stop ignoring me Taz Here we have a config where I generate animal noises and send them to Taz via HTTP. Taz has a tendency to stop his servers whenever I dispatch my animals upon him, and therefore these HTTP requests sometimes fail. However, I have the retry processor and with this super power I can specify a back off policy and it will ensure that for each animal noise the HTTP processor is attempted until either it succeeds or my Redpanda Connect instance is stopped. I even go as far as to zero-out the maximum elapsed time field, which means that for each animal noise I will wait indefinitely, because I really really want Taz to receive every single animal noise that he is entitled to. ```yaml input: generate: interval: 1s mapping: 'root.noise = [ "woof", "meow", "moo", "quack" ].index(random_int(min: 0, max: 3))' pipeline: processors: - retry: backoff: initial_interval: 100ms max_interval: 5s max_elapsed_time: 0s processors: - http: url: 'http://example.com/try/not/to/dox/taz' verb: POST output: # Drop everything because it's junk data, I don't want it lol drop: {} ``` ## [](#fields)Fields ### [](#backoff)`backoff` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `1m` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#max_retries)`max_retries` The maximum number of retry attempts before the request is aborted. Setting this value to `0` will result in unbounded number of retries. **Type**: `int` **Default**: `0` ### [](#parallel)`parallel` When processing batches of messages these batches are ignored and the processors apply to each message sequentially. However, when this field is set to `true` each message will be processed in parallel. Caution should be made to ensure that batch sizes do not surpass a point where this would cause resource (CPU, memory, API limits) contention. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to execute on each message. **Type**: `array` ## [](#batching)Batching When messages are batched the child processors of a retry are executed for each individual message in isolation, performed serially by default but in parallel when the field [`parallel`](#parallel) is set to `true`. This is an intentional limitation of the retry processor and is done in order to ensure that errors are correctly associated with a given input message. Otherwise, the archiving, expansion, grouping, filtering and so on of the child processors could obfuscate this relationship. If the target behavior of your retried processors is "batch aware", in that you wish to perform some processing across the entire batch of messages and repeat it in the event of errors, you can use an [`archive` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) to collapse the batch into an individual message. Then, within these child processors either perform your batch aware processing on the archive, or use an [`unarchive` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/unarchive/) in order to expand the single message back out into a batch. For example, if the retry processor were being used to wrap an HTTP request where the payload data is a batch archived into a JSON array it should look something like this: ```yaml pipeline: processors: - archive: format: json_array - retry: processors: - http: url: example.com/nope verb: POST - unarchive: format: json_array ``` --- # Page 438: schema_registry_decode **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/schema_registry_decode.md --- # schema_registry_decode > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: schema_registry_decode latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/schema_registry_decode page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/schema_registry_decode.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/schema_registry_decode.adoc description: Automatically decodes and validates messages with schemas from a Confluent Schema Registry service. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Automatically decodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml processors: label: "" schema_registry_decode: avro: raw_unions: "" # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false mapping: "" # No default (optional) store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true json: coerce_data: false cache_duration: 10m url: "" # No default (required) default_schema_id: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" schema_registry_decode: avro: raw_unions: "" # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false mapping: "" # No default (optional) store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true json: coerce_data: false cache_duration: 10m url: "" # No default (required) default_schema_id: "" # No default (optional) oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Decodes messages automatically from a schema stored within a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by extracting a schema ID from the message and obtaining the associated schema from the registry. If a message fails to match against the schema then it will remain unchanged and the error can be caught using [error-handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro.raw_unions`](#avro-raw_unions) to `true`. ## [](#protobuf-format)Protobuf format This processor decodes Protobuf messages to JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ## [](#metadata)Metadata This processor also adds the following metadata to each outgoing message: schema\_id: the ID of the schema in the schema registry that was associated with the message. ## [](#fields)Fields ### [](#avro)`avro` Configuration for how to decode schemas that are of type AVRO. **Type**: `object` ### [](#avro-mapping)`avro.mapping` Define a custom mapping to apply to the JSON representation of Avro schemas. You can use mappings to convert custom types emitted by other tools, such as Debezium, into standard Avro types. **Type**: `string` ```yaml # Examples: mapping: |- map isDebeziumTimestampType { root = this.type == "long" && this."connect.name" == "io.debezium.time.Timestamp" && !this.exists("logicalType") } map debeziumTimestampToAvroTimestamp { let mapped_fields = this.fields.or([]).map_each(item -> item.apply("debeziumTimestampToAvroTimestamp")) root = match { this.type == "record" => this.assign({"fields": $mapped_fields}) this.type.type() == "array" => this.assign({"type": this.type.map_each(item -> item.apply("debeziumTimestampToAvroTimestamp"))}) # Add a logical type so that it's decoded as a timestamp instead of a long. this.type.type() == "object" && this.type.apply("isDebeziumTimestampType") => this.merge({"type":{"logicalType": "timestamp-millis"}}) _ => this } } root = this.apply("debeziumTimestampToAvroTimestamp") ``` ### [](#avro-preserve_logical_types)`avro.preserve_logical_types` Choose whether to: - Transform logical types into their primitive type (default). For example, decimals become raw bytes and timestamps become plain integers. - Preserve logical types. Set to `true` to preserve logical types. **Type**: `bool` **Default**: `false` ### [](#avro-raw_unions)`avro.raw_unions` Whether Avro messages should be decoded into normal JSON (JSON that meets the expectations of regular internet JSON) rather than [Avro JSON](https://avro.apache.org/docs/current/specification/). If set to `false`, Avro messages are decoded as [Avro JSON](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodec). For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would be decoded as: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. If set to `true`, Avro messages are decoded as [standard JSON](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull). For example, the same union schema `["null","string","Transaction"]` is decoded as: - A `null` as JSON `null` - The string `"a"` as `"a"` - A `Transaction` instance as `{…​}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. For more details on the difference between standard JSON and Avro JSON, see the [comment in Goavro](https://github.com/linkedin/goavro/blob/5ec5a5ee7ec82e16e6e2b438d610e1cab2588393/union.go#L224-L249) and the [underlying library used for Avro serialization](https://github.com/linkedin/goavro). **Type**: `bool` ### [](#avro-store_schema_metadata)`avro.store_schema_metadata` Optionally store the schema used to decode messages as a metadata field under the given name. This field can later be referenced in other components such as a `parquet_encode` processor in order to automatically infer their schema. **Type**: `string` ### [](#avro-translate_kafka_connect_types)`avro.translate_kafka_connect_types` Only valid if preserve\_logical\_types is true. This decodes various Kafka Connect types into their bloblang equivalents when not representable by standard logical types according to the Avro standard. Types that are currently translated: | Type Name | Bloblang Type | Description | | --- | --- | --- | | io.debezium.time.Date | timestamp | Date without time (days since epoch) | | io.debezium.time.Timestamp | timestamp | Timestamp without timezone (milliseconds since epoch) | | io.debezium.time.MicroTimestamp | timestamp | Timestamp with microsecond precision | | io.debezium.time.NanoTimestamp | timestamp | Timestamp with nanosecond precision | | io.debezium.time.ZonedTimestamp | timestamp | Timestamp with timezone (ISO-8601 format) | | io.debezium.time.Year | timestamp at January 1st at 00:00:00 | Year value | | io.debezium.time.Time | timestamp at the unix epoch | Time without date (milliseconds past midnight) | | io.debezium.time.MicroTime | timestamp at the unix epoch | Time with microsecond precision | | io.debezium.time.NanoTime | timestamp at the unix epoch | Time with nanosecond precision | **Type**: `bool` **Default**: `false` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#cache_duration)`cache_duration` The duration after which a cached schema is considered stale and is removed from the cache. **Type**: `string` **Default**: `10m` ```yaml # Examples: cache_duration: 1h # --- cache_duration: 5m ``` ### [](#default_schema_id)`default_schema_id` This schema ID is used when a message’s schema header cannot be read (`ErrBadHeader`). If this value is not set, schema header errors are returned. This configuration does not work with protobuf schemas. > 💡 **TIP** > > You can also use the [`with_schema_registry_header`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#with_schema_registry_header) bloblang function to add a schema ID to messages. **Type**: `int` ### [](#json)`json` Configuration for how to decode schemas that are of type JSON. **Type**: `object` ### [](#json-coerce_data)`json.coerce_data` Whether decoded values should be coerced to match the types declared in the JSON Schema. By default JSON Schema decoding only validates the message and leaves it untouched, which means numbers are later interpreted as floating point (`double`) and date-time values as strings. When set to `true` the decoder rebuilds the message so that values match the schema: `integer` fields become 64-bit integers, `number` fields stay floating point, `string` fields with `format: date-time` become timestamps, and any `default` values declared in the schema are applied to absent fields. This is useful for downstream components that infer their schema from the decoded values, such as the `iceberg` outputs, which will then create `bigint` columns for integer fields rather than `double`. Note that, unlike the default behavior, this is no longer a read-only operation: the message contents are transformed. Because coercion is stricter than validation, a message that passes validation may still fail coercion (for example an integer that overflows a 64-bit value, or a `date-time` string that is not valid RFC 3339), in which case the error can be caught using [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). **Type**: `bool` **Default**: `false` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#protobuf)`protobuf` Configuration for how to decode schemas that are of type PROTOBUF. **Type**: `object` ### [](#protobuf-emit_default_values)`protobuf.emit_default_values` Whether to emit default-valued primitive fields, empty lists, and empty maps. emit\_unpopulated takes precedence over emit\_default\_values **Type**: `bool` **Default**: `false` ### [](#protobuf-emit_unpopulated)`protobuf.emit_unpopulated` Whether to emit unpopulated fields. It does not emit unpopulated oneof fields or unpopulated extension fields. **Type**: `bool` **Default**: `false` ### [](#protobuf-serialize_to_json)`protobuf.serialize_to_json` If messages should be serialized to JSON bytes. If false then the message is kept in decoded form, which means that 64 bit integers are not converted to strings and types for bytes and google.protobuf.Timestamp are preserved (as they are not serialized to JSON strings). **Type**: `bool` **Default**: `true` ### [](#protobuf-use_enum_numbers)`protobuf.use_enum_numbers` Emits enum values as numbers. **Type**: `bool` **Default**: `false` ### [](#protobuf-use_proto_names)`protobuf.use_proto_names` Use proto field name instead of lowerCamelCase name. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 439: schema_registry_encode **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/schema_registry_encode.md --- # schema_registry_encode > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: schema_registry_encode latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/schema_registry_encode page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/schema_registry_encode.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/schema_registry_encode.adoc description: Automatically encodes and validates messages with schemas from a Confluent Schema Registry service. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Automatically encodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml processors: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) avro: raw_json: "" # No default (optional) record_name: "" namespace: "" ``` #### Advanced ```yml processors: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) normalize: true avro: raw_json: "" # No default (optional) record_name: "" namespace: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Encodes messages automatically from schemas obtains from a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by polling the service for the latest schema version for target subjects. If a message fails to encode under the schema then it will remain unchanged and the error can be caught using [error-handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when encoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can consume documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro_raw_json`](#avro_raw_json) to `true`. ### [](#known-issues)Known issues Important! There is an outstanding issue in the [avro serializing library](https://github.com/linkedin/goavro) that Redpanda Connect uses which means it [doesn’t encode logical types correctly](https://github.com/linkedin/goavro/issues/252). It’s still possible to encode logical types that are in-line with the spec if `avro_raw_json` is set to true, though now of course non-logical types will not be in-line with the spec. ## [](#protobuf-format)Protobuf format This processor encodes Protobuf messages either from any format parsed within Redpanda Connect (encoded as JSON by default), or from raw JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ### [](#multiple-message-support)Multiple message support When a target subject presents a Protobuf schema that contains multiple messages it becomes ambiguous which message definition a given input data should be encoded against. In such scenarios Redpanda Connect will attempt to encode the data against each of them and select the first to successfully match against the data, this process currently **ignores all nested message definitions**. In order to speed up this exhaustive search the last known successful message will be attempted first for each subsequent input. We will be considering alternative approaches in future so please [get in touch](https://redpanda.com/slack) with thoughts and feedback. ## [](#fields)Fields ### [](#avro)`avro` Configuration for Avro encoding. **Type**: `object` ### [](#avro-namespace)`avro.namespace` The Avro namespace for the root record type when encoding from a common schema (schema\_metadata mode). **Type**: `string` **Default**: `""` ### [](#avro-raw_json)`avro.raw_json` Whether messages encoded in Avro format should be parsed as normal JSON rather than Avro JSON. Overrides the deprecated top-level `avro_raw_json` when set. **Type**: `bool` ### [](#avro-record_name)`avro.record_name` The name to use for the root Avro record type when encoding from a common schema (schema\_metadata mode). If empty, derived from the subject. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#format)`format` The encoding format to use when converting a common schema from metadata. Required when `schema_metadata` is set. **Type**: `string` **Options**: `avro`, `json_schema` ### [](#jwt)`jwt` (beta) Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#normalize)`normalize` Whether to normalize the schema before registering with the schema registry (schema\_metadata mode only). **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#refresh_period)`refresh_period` The period after which a schema is refreshed for each subject, this is done by polling the schema registry service. **Type**: `string` **Default**: `10m` ```yaml # Examples: refresh_period: 60s # --- refresh_period: 1h ``` ### [](#schema_metadata)`schema_metadata` When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by `format`, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead. **Type**: `string` **Default**: `""` ### [](#subject)`subject` The schema subject to derive schemas from. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo # --- subject: ${! meta("kafka_topic") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 440: select_parts **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/select_parts.md --- # select_parts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: select_parts latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/select_parts page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/select_parts.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/select_parts.adoc description: Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored. ```yml # Config fields, showing default values label: "" select_parts: parts: [] ``` The selected parts are added to the new message batch in the same order as the selection array. E.g. with 'parts' set to \[ 2, 0, 1 \] and the message parts \[ '0', '1', '2', '3' \], the output will be \[ '2', '0', '1' \]. If none of the selected parts exist in the input batch (resulting in an empty output message) the batch is dropped entirely. Message indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if index = -1 then the selected part will be the last part of the message, if index = -2 then the part before the last element with be selected, and so on. This processor is only applicable to [batched messages](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#parts)`parts[]` An array of message indexes of a batch. Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. **Type**: `array` **Default**: `[]` --- # Page 441: slack_thread **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/slack_thread.md --- # slack_thread > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: slack_thread latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/slack_thread page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/slack_thread.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/slack_thread.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Reads a Slack thread using the Slack API method [conversations.replies](https://api.slack.com/methods/conversations.replies). ```yml # Common configuration fields, showing default values label: "" slack_thread: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (required) ``` ## [](#fields)Fields ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to read messages from the Slack channel specified in `channel_id`. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the Slack channel from which to read threads. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#thread_ts)`thread_ts` The timestamp of the parent message of the thread you want to read. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 442: sleep **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sleep.md --- # sleep > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sleep latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/sleep page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/sleep.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/sleep.adoc description: Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the duration field, you can find a list of functions here. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the `duration` field, you can find a list of functions [here](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). ```yml # Config fields, showing default values label: "" sleep: duration: "" # No default (required) ``` ## [](#fields)Fields ### [](#duration)`duration` The duration of time to sleep for each execution. This field supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 443: split **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/split.md --- # split > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: split latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/split page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/split.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/split.adoc description: Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field byte_size is non-zero, then by total size in bytes (which ever limit is reached first). page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field `byte_size` is non-zero, then by total size in bytes (which ever limit is reached first). ```yml # Config fields, showing default values label: "" split: size: 1 byte_size: 0 ``` This processor is for breaking batches down into smaller ones. In order to break a single message out into multiple messages use the [`unarchive` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/unarchive/). If there is a remainder of messages after splitting a batch the remainder is also sent as a single batch. For example, if your target size was 10, and the processor received a batch of 95 message parts, the result would be 9 batches of 10 messages followed by a batch of 5 messages. ## [](#fields)Fields ### [](#byte_size)`byte_size` An optional target of total message bytes. **Type**: `int` **Default**: `0` ### [](#size)`size` The target number of messages. **Type**: `int` **Default**: `1` --- # Page 444: sql_insert **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sql_insert.md --- # sql_insert > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_insert latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/sql_insert page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/sql_insert.adoc description: Inserts rows into an SQL database for each message, and leaves the message unchanged. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Inserts rows into an SQL database for each message, and leaves the message unchanged. #### Common ```yml processors: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) ``` #### Advanced ```yml processors: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the insert fails to execute then the message will still remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml pipeline: processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sql_raw/) instead. To use Bloblang interpolation inside ``sql_raw’s `query`` field, you must enable `unsafe_dynamic_query: true`. > ⚠️ **CAUTION** > > Interpolating unsanitized values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 445: sql_raw **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sql_raw.md --- # sql_raw > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_raw latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/sql_raw page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/sql_raw.adoc description: Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. #### Common ```yml processors: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) exec_only: "" # No default (optional) queries: [] # No default (optional) ``` #### Advanced ```yml processors: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) exec_only: "" # No default (optional) queries: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). For some scenarios where you might use this processor, see [Examples](#examples). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#exec_only)`exec_only` Whether to discard the [`query`](#query) result. Set to `true` to leave the message contents unchanged, which is useful when you are executing inserts, updates, and so on. By default, the message contents are kept for the last query executed, and previous queries don’t change the results. **Type**: `bool` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#queries)`queries[]` A list of database statements to run in addition to your main [`query`](#query). If you specify multiple queries, they are executed within a single transaction. For more information, see [Examples](#examples). **Type**: `array` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-exec_only)`queries[].exec_only` Whether the query result should be discarded. When set to `true` the message contents will remain unchanged, which is useful in cases where you are executing inserts, updates, etc. By default this is true for the last query, and previous queries don’t change the results. If set to true for any query but the last one, the subsequent `args_mappings` input is overwritten. **Type**: `bool` ### [](#queries-query)`queries[].query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | |---|---| | `clickhouse` | Dollar sign | | `mysql` | Question mark | | `postgres` | Dollar sign | | `pgx` | Dollar sign | | `mssql` | Question mark | | `sqlite` | Question mark | | `oracle` | Colon | | `snowflake` | Question mark | | `trino` | Question mark | | `gocosmos` | Colon | **Type**: `string` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); # --- query: SELECT * FROM footable WHERE user_id = $1; ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries) in the query. Great care should be made to ensure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages. ```yaml pipeline: processors: - sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?);" args_mapping: '[ document.foo, document.bar, meta("kafka_topic") ]' exec_only: true ``` ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - branch: processors: - sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT * FROM footable WHERE user_id = $1;" args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" - sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` --- # Page 446: sql_select **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sql_select.md --- # sql_select > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sql_select latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/sql_select page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/sql_select.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/sql_select.adoc description: Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value. #### Common ```yml processors: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#examples)Examples ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message `user.id`. A [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`: ```yaml pipeline: processors: - branch: processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: user_id = ? args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` An optional prefix to prepend to the query (before SELECT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` The table to query. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks, and will automatically be converted to dollar syntax when the postgres or clickhouse drivers are used. **Type**: `string` ```yaml # Examples: where: meow = ? and woof = ? # --- where: user_id = ? ``` --- # Page 447: string_split **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/string_split.md --- # string_split > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: string_split latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/string_split page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/string_split.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/string_split.adoc description: Splits a string by a delimiter into an array. Generally, using bloblang's `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. page-git-created-date: "2026-04-08" page-git-modified-date: "2026-08-11" --- Splits a string by a delimiter into an array. Generally, using bloblang’s `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. #### Common ```yml processors: label: "" string_split: delimiter: empty_as_null: false ``` #### Advanced ```yml processors: label: "" string_split: delimiter: emit_bytes: false empty_as_null: false ``` ## [](#fields)Fields ### [](#delimiter)`delimiter` The delimiter to split the string by. **Type**: `string` **Default**: \` \` ### [](#emit_bytes)`emit_bytes` When true, the output will be bloblang bytes instead of strings. **Type**: `bool` **Default**: `false` ### [](#empty_as_null)`empty_as_null` When true, empty strings resulting from the split are converted to null. **Type**: `bool` **Default**: `false` --- # Page 448: switch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/switch.md --- # switch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: switch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/switch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/switch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/switch.adoc description: Conditionally processes messages based on their contents. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Conditionally processes messages based on their contents. ```yml # Config fields, showing default values label: "" switch: [] # No default (required) ``` For each switch case a [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) is checked and, if the result is true (or the check is empty) the child processors are executed on the message. ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether a message should have the processors of this case executed on it. If left empty the case always passes. If the check mapping throws an error the message will be flagged [as having failed](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/) and will not be tested against any other cases. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#continue)`continue` Indicates whether, if this case passes for a message, the next case should also be tested. Unlike `fallthrough`, which skips the next case’s check, `continue` will evaluate the next case’s condition before executing. **Type**: `bool` **Default**: `false` ### [](#fallthrough)`fallthrough` Indicates whether, if this case passes for a message, the next case should also be executed without checking its condition. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to execute on a message. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#ignore-george)Ignore George We have a system where we’re counting a metric for all messages that pass through our system. However, occasionally we get messages from George that we don’t care about. For George’s messages we want to instead emit a metric that gauges how angry he is about being ignored and then we drop it. ```yaml pipeline: processors: - switch: - check: this.user.name.first != "George" processors: - metric: type: counter name: MessagesWeCareAbout - processors: - metric: type: gauge name: GeorgesAnger value: ${! json("user.anger") } - mapping: root = deleted() ``` ## [](#batching)Batching When a switch processor executes on a [batch of messages](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/) they are checked individually and can be matched independently against cases. During processing the messages matched against a case are processed as a batch, although the ordering of messages during case processing cannot be guaranteed to match the order as received. At the end of switch processing the resulting batch will follow the same ordering as the batch was received. If any child processors have split or otherwise grouped messages this grouping will be lost as the result of a switch is always a single batch. In order to perform conditional grouping and/or splitting use the [`group_by` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by/). --- # Page 449: sync_response **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sync_response.md --- # sync_response > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: sync_response latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/sync_response page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/sync_response.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/sync_response.adoc description: Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type. ```yml # Config fields, showing default values label: "" sync_response: {} ``` For most inputs this mechanism is ignored entirely, in which case the sync response is dropped without penalty. It is therefore safe to use this processor even when combining input types that might not have support for sync responses. --- # Page 450: text_chunker **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/text_chunker.md --- # text_chunker > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: text_chunker latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/text_chunker page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/text_chunker.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/text_chunker.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2026-05-26" --- Breaks down text-based message content into manageable chunks using a configurable strategy. This processor is ideal for creating vector embeddings of large text documents. #### Common ```yml processors: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes include_code_blocks: false keep_reference_links: false ``` #### Advanced ```yml processors: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes token_encoding: "" # No default (optional) allowed_special: [] disallowed_special: - "all" include_code_blocks: false keep_reference_links: false ``` ## [](#fields)Fields ### [](#allowed_special)`allowed_special[]` A list of special tokens to include in the output from this processor. **Type**: `array` **Default**: `[]` ### [](#chunk_overlap)`chunk_overlap` The number of characters duplicated in adjacent chunks of text. **Type**: `int` **Default**: `100` ### [](#chunk_size)`chunk_size` The maximum size of each chunk, using the selected [`length_measure`](#length_measure). **Type**: `int` **Default**: `512` ### [](#disallowed_special)`disallowed_special[]` A list of special tokens to exclude from the output of this processor. **Type**: `array` **Default**: ```yaml - "all" ``` ### [](#include_code_blocks)`include_code_blocks` When set to `true`, this processor includes code blocks in the output. **Type**: `bool` **Default**: `false` ### [](#keep_reference_links)`keep_reference_links` When set to `true`, this processor includes reference links in the output. **Type**: `bool` **Default**: `false` ### [](#length_measure)`length_measure` Choose a method to measure the length of a string. **Type**: `string` **Default**: `runes` | Option | Summary | | --- | --- | | graphemes | Use unicode graphemes to determine the length of a string. | | runes | Use the number of codepoints to determine the length of a string. | | token | Use the number of tokens (using the token_encoding tokenizer) to determine the length of a string. | | utf8 | Determine the length of text using the number of utf8 bytes. | ### [](#separators)`separators[]` A list of strings to use as separators between chunks when the [`recursive_character` strategy option](#strategy) is specified. By default, the following separators are tried in turn until one is successful: - Double newlines (\` `) - Single newlines (` ``) - Spaces (`" “,”"``) **Type**: `array` **Default**: ```yaml - "\n\n" - "\n" - " " - "" ``` ### [](#strategy)`strategy` Choose a strategy for breaking content down into chunks. **Type**: `string` | Option | Summary | | --- | --- | | markdown | Split text by markdown headers. | | recursive_character | Split text recursively by characters (defined in separators). | | token | Split text by tokens. | ### [](#token_encoding)`token_encoding` The type of encoding to use for tokenization. **Type**: `string` ```yaml # Examples: token_encoding: cl100k_base # --- token_encoding: r50k_base ``` --- # Page 451: try_catch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try_catch.md --- # try_catch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: try_catch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/try_catch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/try_catch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/try_catch.adoc description: Executes a list of child `processors` on each message and, if any of them fail, executes a separate list of `catch` processors to recover from or react to the error. page-git-created-date: "2026-07-09" page-git-modified-date: "2026-08-11" --- Executes a list of child `processors` on each message and, if any of them fail, executes a separate list of `catch` processors to recover from or react to the error. This processor combines the behavior of the [`try`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try/) and [`catch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch/) processors into a single block with an explicit recovery path. Because it contains both the fallible step and its recovery within a single processor, it is the recommended way to handle expected errors when strict error handling (`error_handling.strict`) is enabled. Each message of a batch is processed individually. The `processors` field is executed with "try" semantics: as soon as a processor fails for a given message the remaining `processors` are skipped for that message. Any message that failed is then routed to the `catch` processors. Before they run, the failure is moved off the message: it is stored as a structured object in a metadata field (see `error_metadata`, `error` by default) and the message’s failure flag is **cleared**. The error is therefore available to recovery logic as an ordinary variable rather than as a message property. The object contains: - `what`: the error message. - `name`: the name of the component that failed (when known). - `label`: the label of the component that failed (when set). - `path`: the dot-path of the component that failed (when known). So a recovery mapping reads the failure with, for example, `@error.what` (equivalent to `meta("error").what`). Because the flag is cleared, the `catch` processors run under the normal error semantics, including strict, so a _new_ failure raised while recovering is treated as a fresh error and is not silently tolerated. > 📝 **NOTE** > > Because the failure flag is cleared before the `catch` processors run, the [`error`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#error) and `error_source_*` functions do not report the original failure within the `catch` block; use the metadata object instead. An empty or omitted `catch` simply records the error in metadata and clears the flag (the failure is swallowed). ```yaml pipeline: processors: - try_catch: processors: - resource: foo - resource: bar catch: - mutation: 'root = "failed to process: " + @error.what' ``` In the example above, if either `foo` or `bar` fails for a message then the `mutation` is applied to that message, replacing its contents with a description of the error (read from the metadata object), and the message continues downstream without a failure flag. More information about error handling can be found in [Error Handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). #### Common ```yml processors: label: "" try_catch: processors: [] catch: [] error_metadata: error ``` #### Advanced ```yml processors: label: "" try_catch: processors: [] catch: [] error_metadata: error ``` ## [](#fields)Fields ### [](#catch)`catch[]` A list of processors to execute on each message that failed one of the `processors` above. The message is no longer flagged as failed when these run; the error is available as an object in the metadata field named by `error_metadata` (e.g. `@error.what`). When omitted or empty the error is recorded in metadata and the flag is cleared (the failure is swallowed). **Type**: `array` **Default**: `[]` ### [](#error_metadata)`error_metadata` The metadata key under which the caught error is stored, as an object with a `what` field (the error message) plus `name`, `label` and `path` fields describing the component that failed, before the `catch` processors are executed. **Type**: `string` **Default**: `error` ### [](#processors)`processors[]` A list of processors to execute on each message. If a processor fails for a given message the remaining processors in this list are skipped for that message, and the message is routed to the `catch` processors. **Type**: `array` **Default**: `[]` --- # Page 452: try **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try.md --- # try > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: try latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/try page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/try.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/try.adoc description: Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared). page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared). ```yml # Config fields, showing default values label: "" try: [] ``` This processor behaves similarly to the [`for_each`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, if a message has failed any prior processor (before or during the try block) then that message will skip all following processors. For example, with the following config: ```yaml pipeline: processors: - resource: foo - try: - resource: bar - resource: baz - resource: buz ``` If the processor `bar` fails for a particular message, that message will skip the processors `baz` and `buz`. Similarly, if `bar` succeeds but `baz` does not then `buz` will be skipped. If the processor `foo` fails for a message then none of `bar`, `baz` or `buz` are executed on that message. This processor is useful for when child processors depend on the successful output of previous processors. This processor can be followed with a [catch](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch/) processor for defining child processors to be applied only to failed messages. More information about error handing can be found in [Error Handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#nest-within-a-catch-block)Nest within a catch block In some cases it might be useful to nest a try block within a catch block, since the [`catch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch/) only clears errors _after_ executing its child processors this means a nested try processor will not execute unless the errors are explicitly cleared beforehand. This can be done by inserting an empty catch block before the try block like as follows: ```yaml pipeline: processors: - resource: foo - catch: - log: level: ERROR message: "Foo failed due to: ${! error() }" - catch: [] # Clear prior error - try: - resource: bar - resource: baz ``` --- # Page 453: unarchive **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/unarchive.md --- # unarchive > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: unarchive latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/unarchive page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/unarchive.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/unarchive.adoc description: Unarchives messages according to the selected archive format into multiple messages within a batch. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Unarchives messages according to the selected archive format into multiple messages within a [batch](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ```yml # Config fields, showing default values label: "" unarchive: format: "" # No default (required) ``` When a message is unarchived the new messages replace the original message in the batch. Messages that are selected but fail to unarchive (invalid format) will remain unchanged in the message batch but will be flagged as having failed, allowing you to [error handle them](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). ## [](#metadata)Metadata The metadata found on the messages handled by this processor will be copied into the resulting messages. For the unarchive formats that contain file information (tar, zip), a metadata field is also added to each message called `archive_filename` with the extracted filename. ## [](#fields)Fields ### [](#format)`format` The unarchiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Extract messages from a binary blob format. | | csv | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message. | | csv:x | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message using a custom delimiter. The custom delimiter must be a single character, e.g. the format "csv:\t" would consume a tab delimited file. | | json_array | Attempt to parse a message as a JSON array, and extract each element into its own message. | | json_documents | Attempt to parse a message as a stream of concatenated JSON documents. Each parsed document is expanded into a new message. | | json_map | Attempt to parse the message as a JSON map and for each element of the map expands its contents into a new message. A metadata field is added to each message called archive_key with the relevant key from the top-level map. | | lines | Extract the lines of a message each into their own message. | | tar | Extract messages from a unix standard tape archive. | | zip | Extract messages from a zip file. | --- # Page 454: while **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/while.md --- # while > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: while latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/while page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/while.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/while.adoc description: A processor that checks a Bloblang query against each batch of messages and executes child processors on them for as long as the query resolves to true. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A processor that checks a [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) against each batch of messages and executes child processors on them for as long as the query resolves to true. #### Common ```yml processors: label: "" while: at_least_once: false check: "" processors: [] # No default (required) ``` #### Advanced ```yml processors: label: "" while: at_least_once: false max_loops: 0 check: "" processors: [] # No default (required) ``` The field `at_least_once`, if true, ensures that the child processors are always executed at least one time (like a do .. while loop.) The field `max_loops`, if greater than zero, caps the number of loops for a message batch to this value. If following a loop execution the number of messages in a batch is reduced to zero the loop is exited regardless of the condition result. If following a loop execution there are more than 1 message batches the query is checked against the first batch only. The conditions of this processor are applied across entire message batches. You can find out more about batching [in this doc](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/). ## [](#fields)Fields ### [](#at_least_once)`at_least_once` Whether to always run the child processors at least one time. **Type**: `bool` **Default**: `false` ### [](#check)`check` A [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that should return a boolean value indicating whether the while loop should execute again. **Type**: `string` **Default**: `""` ```yaml # Examples: check: errored() # --- check: this.urls.unprocessed.length() > 0 ``` ### [](#max_loops)`max_loops` An optional maximum number of loops to execute. Helps protect against accidentally creating infinite loops. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to execute on each loop. **Type**: `array` --- # Page 455: workflow **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/workflow.md --- # workflow > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: workflow latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/workflow page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/workflow.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/workflow.adoc description: Executes a topology of branch processors, performing them in parallel where possible. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Executes a topology of [`branch` processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/), performing them in parallel where possible. ### Common ```yml processors: label: "" workflow: meta_path: meta.workflow order: [] branches: request_map: "" processors: [] # No default (required) result_map: "" ``` ### Advanced ```yml processors: label: "" workflow: meta_path: meta.workflow order: [] branch_resources: [] branches: request_map: "" processors: [] # No default (required) result_map: "" ``` ## [](#why-use-a-workflow)Why use a workflow ### [](#performance)Performance Most of the time the best way to compose processors is also the simplest, just configure them in series. This is because processors are often CPU bound, low-latency, and you can gain vertical scaling by increasing the number of processor pipeline threads, allowing Redpanda Connect to process [multiple messages in parallel](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/processing_pipelines/). However, some processors, such as [`aws_lambda`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_lambda/) and [`cache`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/cache/), interact with external services and therefore spend most of their time waiting for a response. These processors tend to be high-latency and low CPU activity, which causes messages to process slowly. When a processing pipeline contains multiple network processors that aren’t dependent on each other we can benefit from performing these processors in parallel for each individual message, reducing the overall message processing latency. ### [](#simplifying-processor-topology)Simplifying processor topology A workflow is often expressed as a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of processing stages, where each stage can result in N possible next stages, until finally the flow ends at an exit node. For example, if we had processing stages A, B, C and D, where stage A could result in either stage B or C being next, always followed by D, it might look something like this: ```text /--> B --\ A --| |--> D \--> C --/ ``` This flow would be easy to express in a standard Redpanda Connect config, we could simply use a [`switch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/switch/) to route to either B or C depending on a condition on the result of A. However, this method of flow control quickly becomes unfeasible as the DAG gets more complicated, imagine expressing this flow using switch processors: ```text /--> B -------------|--> D / / A --| /--> E --| \--> C --| \ \----------|--> F ``` And imagine doing so knowing that the diagram is subject to change over time. Yikes! Instead, with a workflow we can either trust it to automatically resolve the DAG or express it manually as simply as `order: [ [ A ], [ B, C ], [ E ], [ D, F ] ]`, and the conditional logic for determining if a stage is executed is defined as part of the branch itself. ## [](#examples)Examples ### [](#automatic-ordering)Automatic Ordering When the field `order` is omitted a best attempt is made to determine a dependency tree between branches based on their request and result mappings. In the following example the branches foo and bar will be executed first in parallel, and afterwards the branch baz will be executed. ```yaml pipeline: processors: - workflow: meta_path: meta.workflow branches: foo: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' baz: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ### [](#conditional-branches)Conditional Branches Branches of a workflow are skipped when the `request_map` assigns `deleted()` to the root. In this example the branch A is executed when the document type is "foo", and branch B otherwise. Branch C is executed afterwards and is skipped unless either A or B successfully provided a result at `tmp.result`. ```yaml pipeline: processors: - workflow: branches: A: request_map: | root = if this.document.type != "foo" { deleted() } processors: - http: url: TODO result_map: 'root.tmp.result = this' B: request_map: | root = if this.document.type == "foo" { deleted() } processors: - aws_lambda: function: TODO result_map: 'root.tmp.result = this' C: request_map: | root = if this.tmp.result != null { deleted() } processors: - http: url: TODO_SOMEWHERE_ELSE result_map: 'root.tmp.result = this' ``` ### [](#resources)Resources The `order` field can be used in order to refer to [branch processor resources](#resources), this can sometimes make your pipeline configuration cleaner, as well as allowing you to reuse branch configurations in order places. It’s also possible to mix and match branches configured within the workflow and configured as resources. ```yaml pipeline: processors: - workflow: order: [ [ foo, bar ], [ baz ] ] branches: bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' processor_resources: - label: foo branch: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' - label: baz branch: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ## [](#fields)Fields ### [](#branch_resources)`branch_resources[]` An optional list of [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) names that are configured as [Resources](#resources). These resources will be included in the workflow with any branches configured inline within the [`branches`](#branches) field. The order and parallelism in which branches are executed is automatically resolved based on the mappings of each branch. When using resources with an explicit order it is not necessary to list resources in this field. **Type**: `array` **Default**: `[]` ### [](#branches)`branches` An object of named [`branch` processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) that make up the workflow. The order and parallelism in which branches are executed can either be made explicit with the field `order`, or if omitted an attempt is made to automatically resolve an ordering based on the mappings of each branch. **Type**: `object` **Default**: `{}` ### [](#branches-processors)`branches.processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `array` ### [](#branches-request_map)`branches.request_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#branches-result_map)`branches.result_map` A [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ### [](#meta_path)`meta_path` A [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) indicating where to store and reference [structured metadata](#structured-metadata) about the workflow execution. **Type**: `string` **Default**: `meta.workflow` ### [](#order)`order` An explicit declaration of branch ordered tiers, which describes the order in which parallel tiers of branches should be executed. Branches should be identified by the name as they are configured in the field `branches`. It’s also possible to specify branch processors configured [as a resource](#resources). **Type**: `array>` **Default**: `[]` ```yaml # Examples: order: - - foo - bar - - baz # --- order: - - foo - - bar - - baz ``` ## [](#structured-metadata)Structured metadata When the field `meta_path` is non-empty the workflow processor creates an object describing which workflows were successful, skipped or failed for each message and stores the object within the message at the end. The object is of the following form: ```json { "succeeded": [ "foo" ], "skipped": [ "bar" ], "failed": { "baz": "the error message from the branch" } } ``` If a message already has a meta object at the given path when it is processed then the object is used in order to determine which branches have already been performed on the message (or skipped) and can therefore be skipped on this run. This is a useful pattern when replaying messages that have failed some branches previously. For example, given the above example object the branches foo and bar would automatically be skipped, and baz would be reattempted. The previous meta object will also be preserved in the field `.previous` when the new meta object is written, preserving a full record of all workflow executions. If a field `.apply` exists in the meta object for a message and is an array then it will be used as an explicit list of stages to apply, all other stages will be skipped. ## [](#error-handling)Error handling The recommended approach to handle failures within a workflow is to query against the [structured metadata](#structured-metadata) it provides, as it provides granular information about exactly which branches failed and which ones succeeded and therefore aren’t necessary to perform again. For example, if our meta object is stored at the path `meta.workflow` and we wanted to check whether a message has failed for any branch we can do that using a [Bloblang query](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) like `this.meta.workflow.failed.length() | 0 > 0`, or to check whether a specific branch failed we can use `this.exists("meta.workflow.failed.foo")`. However, if structured metadata is disabled by setting the field `meta_path` to empty then the workflow processor instead adds a general error flag to messages when any executed branch fails. In this case it’s possible to handle failures using [standard error handling patterns](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). --- # Page 456: xml **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/xml.md --- # xml > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: xml latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/processors/xml page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/processors/xml.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/processors/xml.adoc description: Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value. ```yml # Config fields, showing default values label: "" xml: operator: "" cast: false ``` ## [](#operators)Operators ### [](#to_json)`to_json` Converts an XML document into a JSON structure, where elements appear as keys of an object according to the following rules: - If an element contains attributes they are parsed by prefixing a hyphen, `-`, to the attribute label. - If the element is a simple element and has attributes, the element value is given the key `#text`. - XML comments, directives, and process instructions are ignored. - When elements are repeated the resulting JSON value is an array. - XML namespaces are stripped from element and attribute names, and namespace declarations (`xmlns`) are omitted. For example, given the following XML: ```xml This is a title This is a description foo1 foo2 foo3 ``` The resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":"1"}, {"#text":"foo2","-id":"2"}, "foo3" ] } } ``` With cast set to true, the resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":1}, {"#text":"foo2","-id":2}, "foo3" ] } } ``` ## [](#fields)Fields ### [](#cast)`cast` Whether to try to cast values that are numbers and booleans to the right type. Default: all values are strings. **Type**: `bool` **Default**: `false` ### [](#operator)`operator` An XML [operation](#operators) to apply to messages. **Type**: `string` **Default**: `""` **Options**: `to_json` --- # Page 457: Rate Limits **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/about.md --- # Rate Limits > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Rate Limits latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/rate_limits/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/rate_limits/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/rate_limits/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A rate limit is a strategy for limiting the usage of a shared resource across parallel components in a Redpanda Connect instance, or potentially across multiple instances. They are configured as a resource: ```yaml rate_limit_resources: - label: foobar local: count: 500 interval: 1s ``` And most components that hit external services have a field `rate_limit` for specifying a rate limit resource to use, identified by the `label` field. For example, if we wanted to use our `foobar` rate limit with a `http_client` input it would look like this: ```yaml input: http_client: url: TODO verb: GET rate_limit: foobar ``` By using a rate limit in this way we can guarantee that our input will only poll our HTTP source at the rate of 500 requests per second. Some components don’t have a `rate_limit` field but we might still wish to throttle them by a rate limit, in which case we can use the [`rate_limit` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/rate_limit/) that applies back pressure to a processing pipeline when the limit is reached. --- # Page 458: local **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/local.md --- # local > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: local latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/rate_limits/local page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/rate_limits/local.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/rate_limits/local.adoc description: A simple X every Y rate limit that can be shared across components within a pipeline. It does not support distributed rate limiting across instances. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- The local rate limit is a simple X every Y type rate limit that can be shared across any number of components within the pipeline but does not support distributed rate limits across multiple running instances of Benthos. ```yml # Config fields, showing default values label: "" local: count: 1000 interval: 1s ``` ## [](#fields)Fields ### [](#count)`count` The maximum number of requests to allow for a given period of time. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `"1s"` --- # Page 459: redis **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/rate_limits/redis.md --- # redis > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redis latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/rate_limits/redis page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/rate_limits/redis.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/rate_limits/redis.adoc description: A token bucket rate limit backed by Redis, shared across all Redpanda Connect instances that use the same Redis instance. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- A rate limit implementation using Redis. It works by using a simple token bucket algorithm to limit the number of requests to a given count within a given time period. The rate limit is shared across all instances of Redpanda Connect that use the same Redis instance, which must all have a consistent count and interval. #### Common ```yml # Common config fields, showing default values label: "" redis: url: redis://:6379 # No default (required) count: 1000 interval: 1s key: "" # No default (required) ``` #### Advanced ```yml # All config fields, showing default values label: "" redis: url: redis://:6379 # No default (required) kind: simple master: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] count: 1000 interval: 1s key: "" # No default (required) ``` ## [](#fields)Fields ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yml # Examples url: redis://:6379 url: redis://localhost:6379 url: redis://foousername:foopassword@redisplace:6379 url: redis://:foopassword@redisplace:6379 url: redis://localhost:6379/1 url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `"simple"` Options: `simple` , `cluster` , `failover` . ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yml # Examples master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yml # Examples root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yml # Examples root_cas_file: ./root_cas.pem ``` ### [](#tls-client_certs)`tls.client_certs` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yml # Examples client_certs: - cert: foo key: bar client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yml # Examples password: foo password: ${KEY_PASSWORD} ``` ### [](#count)`count` The maximum number of messages to allow for a given period of time. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `"1s"` ### [](#key)`key` The key to use for the rate limit. **Type**: `string` --- # Page 460: redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/redpanda/about.md --- # redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/redpanda/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/redpanda/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/redpanda/about.adoc page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- The Redpanda Connect configuration service allows you to: - Configure Redpanda cluster credentials in a single configuration block, which is referenced by multiple components in data pipeline. For more information, see the [Pipeline example](#pipeline-example). - Send logs and status updates to topics on a Redpanda cluster, in addition to the [default logger](https://docs.redpanda.com/connect/components/logger/about/). The `redpanda` namespace contains the configuration of this service. #### Common ```yml # Common configuration fields, showing default values redpanda: seed_brokers: [] # No default (optional) pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" ``` #### Advanced ```yml # All configuration fields, showing default values redpanda: seed_brokers: [] # No default (optional) client_id: benthos tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 5m request_timeout_overhead: 10s conn_idle_timeout: 20s pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) timeout: 10s max_message_bytes: 1MB broker_write_max_bytes: 100MB allow_auto_topic_creation: true ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#fields)Fields ### [](#seed_brokers)`seed_brokers` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yml # Examples seed_brokers: - localhost:9092 seed_brokers: - foo:9092 - bar:9092 seed_brokers: - foo:9092,bar:9092 ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#tls)`tls` Override system defaults with custom TLS settings. **Type**: `object` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent trusted root certificate, through possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yml # Example root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent trusted root certificate, through possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yml # Example root_cas_file: ./root_cas.pem ``` ### [](#tls-client_certs)`tls.client_certs` A list of client certificates to use. For each certificate, specify either the fields `cert` and `key` or `cert_file` and `key_file`. **Type**: `array` **Default**: `[]` ```yml # Examples client_certs: - cert: foo key: bar client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` The plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. > ⚠️ **WARNING** > > The `pbeWithMD5AndDES-CBC` algorithm does not authenticate ciphertext, and is vulnerable to padding oracle attacks which may allow an attacker to recover the plain text password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yml # Examples password: foo password: ${KEY_PASSWORD} ``` ### [](#sasl)`sasl` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yml # Example sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM-based authentication as specified by the aws-msk-iam-auth Java library. | | OAUTHBEARER | OAuth Bearer-based authentication. | | PLAIN | Plain text authentication. | | SCRAM-SHA-256 | SCRAM-based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM-based authentication as specified in RFC5802. | | none | Disable SASL authentication | ### [](#sasl-username)`sasl[].username` A username for `PLAIN` or `SCRAM-*` authentication. **Type**: `string` **Default**: `""` ### [](#sasl-password)`sasl[].password` A password for `PLAIN` or `SCRAM-*` authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s `OAUTHBEARER` authentication. **Type**: `string` **Default**: `""` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to `OAUTHBEARER` authentication requests. **Type**: `object` ### [](#sasl-aws)`sasl[].aws` AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` **Default**: `""` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Specify a custom endpoint for the AWS API. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](https://docs.redpanda.com/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume an [IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` **Default**: `false` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The role ARN to assume. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` **Default**: `""` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. **Type**: `string` **Default**: `5m` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#conn_idle_timeout)`conn_idle_timeout` Define how long connections can remain idle before they are closed. **Type**: `string` ### [](#pipeline_id)`pipeline_id` The ID of a Redpanda Connect data pipeline (optional). When specified, the pipeline ID is written to all logs and status updates sent to the configured topics. **Type**: `string` **Default**: `""` ### [](#logs_topic)`logs_topic` The topic that logs are sent to. **Type**: `string` **Default**: `""` ```yml # Example logs_topic: __redpanda.connect.logs ``` ### [](#logs_level)`logs_level` The logging level of logs sent to Redpanda. **Type**: `string` **Default**: `info` **Options**: `debug`, `info`, `warn`, `error` ### [](#status_topic)`status_topic` The topic that status updates are sent to. When configured, Redpanda Connect emits status events to this internal topic as pipelines start, run, and stop. **Type**: `string` **Default**: `""` ```yml # Example status_topic: __redpanda.connect.status ``` For monitoring and troubleshooting guidance, see [Monitor Pipeline Status](https://docs.redpanda.com/connect/guides/monitor-pipeline-status/). #### [](#status-event-types)Status event types Status events are emitted during the pipeline lifecycle. All events include `pipeline_id`, `instance_id`, and `timestamp` fields. **TYPE\_INITIALIZING** (value: 1) Emitted when a pipeline instance has successfully parsed its configuration and is attempting to start. This is the first event sent after pipeline startup. Example: ```json { "type": "TYPE_INITIALIZING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890000 } ``` **TYPE\_CONNECTION\_HEALTHY** (value: 2) Emitted every 30 seconds (heartbeat) when all pipeline connections (inputs and outputs) are active and functioning normally. Example: ```json { "type": "TYPE_CONNECTION_HEALTHY", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890030 } ``` **TYPE\_CONNECTION\_ERROR** (value: 3) Emitted every 30 seconds (during the same heartbeat cycle) when one or more connections are inactive or experiencing errors. The event includes detailed error information for each failing connection: - `path`: The configuration path of the connector (see [field paths](https://docs.redpanda.com/connect/configuration/field_paths/)) - `label`: Optional label assigned to the connector - `message`: The error message describing the connection failure Example: ```json { "type": "TYPE_CONNECTION_ERROR", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890060, "connection_errors": [ { "path": "input.kafka_franz", "label": "primary-input", "message": "kafka: connection refused" } ] } ``` **TYPE\_EXITING** (value: 4) Emitted when a pipeline instance is shutting down, either gracefully or due to an error. If shutdown was caused by an error, the event includes an `exit_error` object with the error message. Example (graceful shutdown): ```json { "type": "TYPE_EXITING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890090 } ``` Example (error shutdown): ```json { "type": "TYPE_EXITING", "pipeline_id": "my-pipeline", "instance_id": "abc123xyz", "timestamp": 1717890090, "exit_error": { "message": "failed to process message: invalid format" } } ``` #### [](#message-format)Message format Status events are written to the configured topic as Protocol Buffer JSON messages: - **Topic**: The value specified in `status_topic` - **Key**: The `pipeline_id` value (enables tracking all events for a specific pipeline) - **Value**: JSON-encoded status event - **Encoding**: Protocol Buffer JSON (protojson) For the full protobuf schema specification, see the [status.proto definition](https://github.com/redpanda-data/connect/blob/main/proto/redpanda/api/connect/v1alpha1/status.proto). ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition. The partition with the fewest buffered records. Partitions are selected per batch. | | manual | Manually select a partition for each message. You must also specify a value for the partition field. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute the partition for the record. | | round_robin | Does a round robin of messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but is useful if you want to ensure an even distribution of records to partitions. | ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is not available. **Type**: `bool` **Default**: `true` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` Options: `lz4` , `snappy` , `gzip` , `none` , `zstd` ### [](#timeout)`timeout` The maximum period of time allowed for sending log or status update messages before a request is abandoned and a retry attempted. **Type**: `string` **Default**: `10s` ### [](#max_message_bytes)`max_message_bytes` The maximum size of an individual message in bytes. Messages larger than this value are rejected. This field is equivalent to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MB` ```yml # Examples max_message_bytes: 100MB max_message_bytes: 50mib ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The upper bound for the number of bytes written to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `"100MB"` ```yml # Examples broker_write_max_bytes: 128MB broker_write_max_bytes: 50mib ``` --- # Page 461: Scanners **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/about.md --- # Scanners > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Scanners latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- For such inputs it’s necessary to define a mechanism by which the stream of source bytes can be chopped into smaller logical messages, processed and outputted as a continuous process whilst the stream is being read, as this dramatically reduces the memory usage of Redpanda Connect as a whole and results in a more fluid flow of data. The way in which we define this chopping mechanism is through scanners, configured as a field on each input that requires one. For example, if we wished to consume files line-by-line, which each individual line being processed as a discrete message, we could use the [`lines` scanner](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/lines/) with our `file` input: ## Common ```yaml input: file: paths: [ "./*.txt" ] scanner: lines: {} ``` ## Advanced ```yaml # Instead of newlines, use a custom delimiter: input: file: paths: [ "./*.txt" ] scanner: lines: custom_delimiter: "---END---" max_buffer_size: 100_000_000 # 100MB line buffer ``` A scanner is a plugin similar to any other core Redpanda Connect component (inputs, processors, outputs, etc), which means it’s possible to define your own scanners that can be utilized by inputs that need them. --- # Page 462: avro **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/avro.md --- # avro > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: avro latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/avro page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/avro.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/avro.adoc description: Consume a stream of Avro OCF datum. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consume a stream of Avro OCF datum. #### Common ```yml scanners: avro: {} ``` #### Advanced ```yml scanners: avro: raw_json: false ``` ## [](#avro-json-format)Avro JSON format This scanner creates documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The `"name"` is the type’s name and the `"value"` is the recursively encoded value. For Avro’s named types (record, fixed or enum), the user-specified name is used. For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - The `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`raw_json`](#raw_json) to `true`. ## [](#metadata)Metadata This scanner emits the following metadata for each message: - The `@avro_schema` field: The canonical Avro schema. - The `@avro_schema_fingerprint` field: The schema ID or fingerprint. ## [](#fields)Fields ### [](#raw_json)`raw_json` Whether to decode messages into normal JSON rather than [Avro JSON](https://avro.apache.org/docs/current/specification/_print/#json-encoding). When true, this unwraps union values (bare values instead of {"type": value} wrappers). **Type**: `bool` **Default**: `false` --- # Page 463: chunker **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/chunker.md --- # chunker > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: chunker latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/chunker page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/chunker.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/chunker.adoc description: Split an input stream into chunks of a given number of bytes. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Split an input stream into chunks of a given number of bytes. ```yml # Config fields, showing default values chunker: size: 0 # No default (required) ``` ## [](#fields)Fields ### [](#size)`size` The size of each chunk in bytes. **Type**: `int` --- # Page 464: csv **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/csv.md --- # csv > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: csv latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/csv page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/csv.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/csv.adoc description: Consume comma-separated values row by row, including support for custom delimiters. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consume comma-separated values row by row, including support for custom delimiters. ```yml # Config fields, showing default values csv: custom_delimiter: "" # No default (optional) parse_header_row: true lazy_quotes: false continue_on_error: false ``` ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `csv_row` The index of each row, beginning at 0. ## [](#fields)Fields ### [](#continue_on_error)`continue_on_error` If a row fails to parse due to any error emit an empty message marked with the error and then continue consuming subsequent rows when possible. This can sometimes be useful in situations where input data contains individual rows which are malformed. However, when a row encounters a parsing error it is impossible to guarantee that following rows are valid, as this indicates that the input data is unreliable and could potentially emit misaligned rows. **Type**: `bool` **Default**: `false` ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter instead of the default comma. **Type**: `string` ### [](#lazy_quotes)`lazy_quotes` If set to `true`, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. **Type**: `bool` **Default**: `false` ### [](#parse_header_row)`parse_header_row` Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, each message will consist of an array of values from the corresponding CSV row. **Type**: `bool` **Default**: `true` --- # Page 465: decompress **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/decompress.md --- # decompress > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: decompress latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/decompress page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/decompress.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/decompress.adoc description: Decompress the stream of bytes according to an algorithm, before feeding it into a child scanner. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Decompress the stream of bytes according to an algorithm, before feeding it into a child scanner. ```yml # Config fields, showing default values decompress: algorithm: "" # No default (required) into: to_the_end: {} ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` One of `gzip`, `pgzip`, `zlib`, `bzip2`, `flate`, `snappy`, `lz4`, `zstd`. **Type**: `string` ### [](#into)`into` The child scanner to feed the decompressed stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 466: json_array **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/json_array.md --- # json_array > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: json_array latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/json_array page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/json_array.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/json_array.adoc description: Consumes a stream of one or more JSON elements within a top level array. page-git-created-date: "2025-09-26" page-git-modified-date: "2026-08-11" --- Consumes a stream of one or more JSON elements within a top level array. This scanner is useful for: - Processing exports from systems that generate a JSON array as the top-level JSON structure (for example, logs, bulk exports, etc). - Efficiently breaking up large files with many objects into individual events/messages. Suppose you have a file `events.json`: `events.json` ```json [ {"event": "login", "user": "alice"}, {"event": "logout", "user": "bob"}, {"event": "purchase", "user": "carol", "amount": 42} ] ``` The configuration to process this file is: ```yaml input: file: paths: [ "./events.json" ] scanner: json_array: {} ``` Result: Each event in the array is processed as a separate message. ## [](#requirements)Requirements The `json_array` scanner expects the input to be a single JSON array, where each array element is a JSON object or value. ## [](#fields)Fields The `json_array` scanner has no required fields. You declare it as `{}` in your config. ```yaml json_array: {} ``` --- # Page 467: json_documents **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/json_documents.md --- # json_documents > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: json_documents latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/json_documents page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/json_documents.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/json_documents.adoc description: Consumes a stream of one or more JSON documents. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consumes a stream of one or more JSON documents. ```yml # Config fields, showing default values json_documents: {} ``` --- # Page 468: lines **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/lines.md --- # lines > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: lines latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/lines page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/lines.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/lines.adoc description: Split an input stream into a message per line of data. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Split an input stream into a message per line of data. ```yml # Config fields, showing default values lines: custom_delimiter: "" # No default (optional) max_buffer_size: 65536 omit_empty: false ``` ## [](#fields)Fields ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter for detecting the end of a line rather than a single line break. **Type**: `string` ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a line can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#omit_empty)`omit_empty` Omit empty lines. **Type**: `bool` **Default**: `false` --- # Page 469: re_match **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/re_match.md --- # re_match > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: re_match latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/re_match page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/re_match.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/re_match.adoc description: Split an input stream into segments matching against a regular expression. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Split an input stream into segments matching against a regular expression. ```yml # Config fields, showing default values re_match: pattern: (?m)^\d\d:\d\d:\d\d # No default (required) max_buffer_size: 65536 ``` ## [](#fields)Fields ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a message can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#pattern)`pattern` The pattern to match against. **Type**: `string` ```yaml # Examples: pattern: (?m)^\d\d:\d\d:\d\d ``` --- # Page 470: skip_bom **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/skip_bom.md --- # skip_bom > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: skip_bom latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/skip_bom page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/skip_bom.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/skip_bom.adoc description: Skip one or more byte order marks for each opened child scanner. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Skip one or more byte order marks for each opened child scanner. ```yml # Config fields, showing default values skip_bom: into: to_the_end: {} ``` ## [](#fields)Fields ### [](#into)`into` The child scanner to feed the resulting stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 471: switch **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/switch.md --- # switch > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: switch latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/switch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/switch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/switch.adoc description: Select a child scanner dynamically for source data based on factors such as the filename. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Select a child scanner dynamically for source data based on factors such as the filename. ```yml # Config fields, showing default values switch: [] # No default (required) ``` This scanner outlines a list of potential child scanner candidates to be chosen, and for each source of data the first candidate to pass will be selected. A candidate without any conditions acts as a catch-all and will pass for every source, it is recommended to always have a catch-all scanner at the end of your list. If a given source of data does not pass a candidate an error is returned and the data is rejected. ## [](#fields)Fields ### [](#re_match_name)`re_match_name` A regular expression to test against the name of each source of data fed into the scanner (filename or equivalent). If this pattern matches the child scanner is selected. **Type**: `string` ### [](#scanner)`scanner` The scanner to activate if this candidate passes. **Type**: `scanner` ## [](#examples)Examples ### [](#switch-based-on-file-name)Switch based on file name In this example a file input chooses a scanner based on the extension of each file ```yaml input: file: paths: [ ./data/* ] scanner: switch: - re_match_name: '\.avro$' scanner: { avro: {} } - re_match_name: '\.csv$' scanner: { csv: {} } - re_match_name: '\.csv.gz$' scanner: decompress: algorithm: gzip into: csv: {} - re_match_name: '\.tar$' scanner: { tar: {} } - re_match_name: '\.tar.gz$' scanner: decompress: algorithm: gzip into: tar: {} - scanner: { to_the_end: {} } ``` --- # Page 472: tar **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/tar.md --- # tar > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: tar latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/tar page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/tar.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/tar.adoc description: Consume a tar archive file by file. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Consume a tar archive file by file. ```yml # Config fields, showing default values tar: {} ``` ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `tar_name` --- # Page 473: to_the_end **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/scanners/to_the_end.md --- # to_the_end > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: to_the_end latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/scanners/to_the_end page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/scanners/to_the_end.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/scanners/to_the_end.adoc description: Read the input stream all the way until the end and deliver it as a single message. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Read the input stream all the way until the end and deliver it as a single message. ```yml # Config fields, showing default values to_the_end: {} ``` > ⚠️ **CAUTION** > > Some sources of data may not have a logical end, therefore caution should be made to exclusively use this scanner when the end of an input stream is clearly defined (and well within memory). --- # Page 474: Tracers **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/about.md --- # Tracers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Tracers latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/tracers/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/tracers/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/tracers/about.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- When a tracer is configured all messages will be allocated a root span during ingestion that represents their journey through a Redpanda Connect pipeline. Many Redpanda Connect processors create spans, and so tracing is a great way to analyse the pathways of individual messages as they progress through a Redpanda Connect instance. Some inputs, such as `http_server` and `http_client`, are capable of extracting a root span from the source of the message (HTTP headers). This is a work in progress and should eventually expand so that all inputs have a way of doing so. Other inputs, such as `kafka` can be configured to extract a root span by using the `extract_tracing_map` field. A tracer config section looks like this: ```yaml tracer: jaeger: agent_address: localhost:6831 sampler_type: const sampler_param: 1 ``` > ⚠️ **CAUTION** > > Although the configuration spec of this component is stable the format of spans, tags and logs created by Redpanda Connect is subject to change as it is tuned for improvement. --- # Page 475: gcp_cloudtrace **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/gcp_cloudtrace.md --- # gcp_cloudtrace > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: gcp_cloudtrace latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/tracers/gcp_cloudtrace page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/tracers/gcp_cloudtrace.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/tracers/gcp_cloudtrace.adoc description: Send tracing events to Google Cloud Trace. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Send tracing events to a [Google Cloud Trace](https://cloud.google.com/trace). #### Common ```yml tracers: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 flush_interval: "" # No default (optional) ``` #### Advanced ```yml tracers: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 tags: {} flush_interval: "" # No default (optional) ``` ## [](#fields)Fields ### [](#flush_interval)`flush_interval` The period of time between each flush of tracing spans. **Type**: `string` ### [](#project)`project` The google project with Cloud Trace API enabled. If this is omitted then the Google Cloud SDK will attempt auto-detect it from the environment. **Type**: `string` ### [](#sampling_ratio)`sampling_ratio` Sets the ratio of traces to sample. Tuning the sampling ratio is recommended for high-volume production workloads. **Type**: `float` **Default**: `1` ```yaml # Examples: sampling_ratio: 1 ``` ### [](#tags)`tags` A map of tags to add to tracing spans. **Type**: `object` **Default**: `{}` --- # Page 476: none **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/none.md --- # none > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: none latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/tracers/none page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/tracers/none.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/tracers/none.adoc description: Do not send tracing events anywhere. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Do not send tracing events anywhere. ```yml # Config fields, showing default values tracer: none: {} ``` --- # Page 477: open_telemetry_collector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/open_telemetry_collector.md --- # open_telemetry_collector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: open_telemetry_collector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/tracers/open_telemetry_collector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/tracers/open_telemetry_collector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/tracers/open_telemetry_collector.adoc description: Send tracing events to an Open Telemetry collector. page-git-created-date: "2026-05-28" page-git-modified-date: "2026-08-11" --- Send tracing events to an [Open Telemetry collector](https://opentelemetry.io/docs/collector/). #### Common ```yml tracers: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) sampling: enabled: false ratio: "" # No default (optional) ``` #### Advanced ```yml tracers: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) tags: {} sampling: enabled: false ratio: "" # No default (optional) ``` ## [](#fields)Fields ### [](#grpc)`grpc[]` A list of grpc collectors. **Type**: `array` ### [](#grpc-address)`grpc[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4317 ``` ### [](#grpc-secure)`grpc[].secure` Connect to the collector with client transport security **Type**: `bool` **Default**: `false` ### [](#http)`http[]` A list of http collectors. **Type**: `array` ### [](#http-address)`http[].address` The endpoint of a collector to send events to. **Type**: `string` ```yaml # Examples: address: localhost:4318 ``` ### [](#http-secure)`http[].secure` Connect to the collector over HTTPS **Type**: `bool` **Default**: `false` ### [](#sampling)`sampling` Settings for trace sampling. Sampling is recommended for high-volume production workloads. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable sampling. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` Sets the ratio of traces to sample. **Type**: `float` ```yaml # Examples: ratio: 0.85 # --- ratio: 0.5 ``` ### [](#service)`service` The name of the service in traces. **Type**: `string` **Default**: `benthos` ### [](#tags)`tags` A map of tags to add to all exported spans and metrics. **Type**: `object` **Default**: `{}` --- # Page 478: redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/redpanda.md --- # redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/components/tracers/redpanda page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/components/tracers/redpanda.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/components/tracers/redpanda.adoc description: Send tracing events to a Redpanda topic. page-git-created-date: "2025-12-03" page-git-modified-date: "2026-08-11" --- Export distributed tracing data to a Redpanda topic, enabling you to monitor and debug your Redpanda Connect pipelines. Traces are exported in OpenTelemetry format as JSON, allowing integration with observability platforms like Jaeger, Grafana Tempo, or custom trace consumers. #### Common ```yml tracers: redpanda: seed_brokers: [] # No default (required) topic: otel-traces format: json schema_registry: url: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} service: redpanda-connect sampling: enabled: false ratio: "" # No default (optional) ``` #### Advanced ```yml tracers: redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true acks: all compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB max_buffered_records: 10000 max_buffered_bytes: 0 max_in_flight_requests: 1 record_retries: 0 record_delivery_timeout: 0s topic: otel-traces format: json schema_registry: url: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} service: redpanda-connect tags: {} sampling: enabled: false ratio: "" # No default (optional) ``` This tracer automatically captures trace spans as messages flow through your pipeline, recording timing information, component metadata, and error details. Use this to: - **Track message flow** through complex pipelines with multiple processors. - **Identify performance bottlenecks** by analyzing span durations. - **Debug failures** by examining trace context and error details. - **Monitor pipeline health** across distributed Redpanda Connect instances. - **Correlate activity** across multiple services using trace IDs. The tracer writes to a dedicated Redpanda topic that can be consumed by trace analysis tools. Configure sampling to control trace volume in high-throughput environments. ## [](#fields)Fields ### [](#acks)`acks` The number of acknowledgements the leader broker must receive from ISR brokers before responding to the produce request. When `idempotent_write` is enabled this must be set to `all`. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Wait for all in-sync replicas to acknowledge (acks=-1). Required when idempotent_write is enabled. | | leader | Wait for the leader broker to acknowledge (acks=1). Messages are lost if the leader fails before replication. | | none | Do not wait for any acknowledgement (acks=0). Highest throughput but messages may be lost. | ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Whether to automatically create the trace topic if it doesn’t exist. If false, the topic must be created manually before starting the tracer. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This appears in broker logs and metrics to help identify which Redpanda Connect instance is sending traces. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Compression codec to use for trace messages. Options include `gzip`, `snappy`, `lz4`, `zstd`, or none. Compression can reduce network bandwidth and storage costs. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#format)`format` The format for trace data. Currently only `json` is supported, which exports OpenTelemetry spans as JSON messages. **Type**: `string` **Default**: `json` | Option | Summary | | --- | --- | | json | Emit in JSON Format | | protobuf | Emit in Protobuf Format | | schema-registry-json | Emit in JSON Format with Schema Registry encoding | | schema-registry-protobuf | Emit in Protobuf Format with Schema Registry encoding | ### [](#idempotent_write)`idempotent_write` Enable idempotent writes to prevent duplicate trace messages in case of retries. Recommended for production environments. **Type**: `bool` **Default**: `true` ### [](#max_buffered_bytes)`max_buffered_bytes` The maximum number of bytes the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered. Set to `0` to disable the byte-level limit (only `max_buffered_records` applies). This limit is checked after `max_buffered_records`. **Type**: `string` **Default**: `0` ```yaml # Examples: max_buffered_bytes: 256MB # --- max_buffered_bytes: 50mib ``` ### [](#max_buffered_records)`max_buffered_records` The maximum number of records the client will buffer in memory before blocking. When this limit is reached, `Produce()` calls will block until buffered records are delivered and space frees up. Increase this value for high-throughput pipelines to avoid back-pressure stalls. **Type**: `int` **Default**: `10000` ### [](#max_in_flight_requests)`max_in_flight_requests` The maximum number of produce requests in flight per broker connection. When `idempotent_write` is enabled, this is capped at 5 by the Kafka protocol (and at 1 for Kafka < v1.0.0). When `idempotent_write` is disabled, higher values improve throughput by pipelining requests but may cause out-of-order delivery. **Type**: `int` **Default**: `1` ### [](#max_message_bytes)`max_message_bytes` The maximum size of individual trace messages. Traces exceeding this size will be truncated or dropped. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum age of cached cluster metadata before it is refreshed. Reducing this value can help detect cluster changes faster but increases metadata requests. **Type**: `string` **Default**: `1m` ### [](#partitioner)`partitioner` Override the default partitioner for trace messages. By default, traces are distributed across partitions for load balancing. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#record_delivery_timeout)`record_delivery_timeout` The maximum time a record can sit in the producer buffer before it is failed, roughly equivalent to Kafka’s `delivery.timeout.ms`. This is evaluated before writing a request or after a produce response. When a record times out, all records in the same partition are also failed. Set to `0s` for no timeout (the default). With `idempotent_write` enabled, timeouts are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `string` **Default**: `0s` ### [](#record_retries)`record_retries` The maximum number of times a record produce is retried on failure before the record is failed. When a record fails, all records buffered in the same partition are also failed to preserve gapless ordering. Set to `0` for unlimited retries (the default). With `idempotent_write` enabled, retries are only enforced when safe to do so without creating invalid sequence numbers. **Type**: `int` **Default**: `0` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. This buffer helps prevent premature timeouts. **Type**: `string` **Default**: `10s` ### [](#sampling)`sampling` Configure trace sampling to control the volume of trace data. Sampling is essential for high-throughput pipelines to prevent trace data from overwhelming your observability infrastructure. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable trace sampling. When disabled, all traces are exported. When enabled, traces are sampled according to the configured ratio. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` The sampling ratio as a decimal between 0 and 1. For example, `0.1` samples 10% of traces, `0.01` samples 1%. Lower ratios reduce trace volume and overhead. For high-throughput production systems, start with 0.01-0.1 and adjust based on your needs. **Type**: `float` ```yaml # Examples: ratio: 0.05 # --- ratio: 0.85 # --- ratio: 0.5 ``` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `object` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Schema registry information to publish schemas for tracing data along with the data. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` (beta) Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#service)`service` The service name to identify this Redpanda Connect instance in traces. This appears in trace visualizations and helps correlate traces across distributed systems. Use descriptive names like `order-processor` or `analytics-pipeline`. **Type**: `string` **Default**: `redpanda-connect` ### [](#tags)`tags` Custom key-value tags to attach to all traces from this instance. Use tags to add metadata like environment (`production`, `staging`), region, version, or instance identifiers. Tags appear as resource attributes in OpenTelemetry traces. **Type**: `object` **Default**: `{}` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum time to wait for trace messages to be acknowledged by the broker before considering the write failed. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `array` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to use TLS for the connection to the Redpanda cluster. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Manage Secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) before adding it to your configuration. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The Redpanda topic where trace data is written. This topic should be dedicated to traces and configured with appropriate retention policies. Default: `otel-traces` **Type**: `string` **Default**: `otel-traces` --- # Page 479: Configuration **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/about.md --- # Configuration > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configuration latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/about.adoc description: Learn about different options for configuring Redpanda Connect. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Redpanda Connect pipelines are configured in a YAML file that consists of a number of root sections, arranged like so: #### Common ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` #### Full ```yaml http: address: 0.0.0.0:4195 debug_endpoints: false input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: none: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' input_resources: [] cache_resources: [] processor_resources: [] rate_limit_resources: [] output_resources: [] logger: level: INFO static_fields: '@service': benthos metrics: prometheus: {} tracer: none: {} shutdown_timeout: 20s shutdown_delay: "" ``` Most sections represent a component type, which you can read about in more detail in [this document](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/). These types are hierarchical. For example, an `input` can have a list of child `processor` types attached to it, which in turn can have their own `processor` children. This is powerful but can potentially lead to large and cumbersome configuration files. This document outlines tooling provided by Redpanda Connect to help with writing and managing these more complex configuration files. ## [](#testing)Testing For guidance on how to write and run unit tests for your configuration files read [this guide](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/unit_testing/). ## [](#customizing-your-configuration)Customizing your configuration Sometimes it’s useful to write a configuration where certain fields can be defined during deployment. For this purpose Redpanda Connect supports [environment variable interpolation](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/), allowing you to set fields in your config with environment variables like so: ```yaml input: kafka: addresses: - ${KAFKA_BROKER:localhost:9092} topics: - ${KAFKA_TOPIC:default-topic} ``` This is very useful for sharing configuration files across different deployment environments. ## [](#labels)Labels Labels are unique, user-defined identifiers used throughout Redpanda Connect configurations. They serve two purposes: - **Reference:** Allow different parts of your pipeline to refer to specific components or resources. - **Readability:** Make your configuration more understandable for humans, especially in complex deployments. You can assign labels to most pipeline components, including resources, inputs, outputs, processors, and entire pipelines. Using clear, descriptive labels improves both maintainability and clarity. Labels are commonly applied to the following components: ### [](#resources)Resources Labels identify [reusable resources](#reuse) such as processors, caches, and rate limiters, making them easy to reference elsewhere in your pipeline. ```yaml processor_resources: - label: my-transformer # Processor resource label mapping: 'root = content().uppercase()' cache_resources: - label: user-cache # Cache resource label memory: default_ttl: 300s rate_limit_resources: - label: api-limiter # Rate limiter resource label local: count: 100 interval: 1m ``` ### [](#component-labeling-for-clarity)Component labeling for clarity You can also use labels on inputs, outputs, processors, and other components to improve the human-readability of your configuration and make troubleshooting easier. For example: ```yaml input: label: ingest_api http_server: {} pipeline: label: user_data_ingest processors: - label: sanitize_fields mapping: 'root = this.trim()' - resource: my-transformer ``` ## [](#label-naming-requirements)Label naming requirements Labels must meet the following criteria: - **Length**: 3-128 characters - **Allowed characters**: Alphanumeric, hyphens, and underscores (`A-Za-z0-9-_`) - **Case sensitivity**: Labels are case-sensitive Example valid labels my-processor data\_transformer\_01 UserAnalytics-v2 Example invalid labels ab // Too short (less than 3 characters) my.processor // Invalid character: period my processor // Invalid character: space ## [](#reuse)Reusing configuration snippets Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component as a resource. In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ## [](#shutting-down)Shutting down Under normal operating conditions, the Redpanda Connect process will shut down when there are no more messages produced by inputs and the final message has been processed. The shutdown procedure can also be initiated by sending the process a interrupt (`SIGINT`) or termination (`SIGTERM`) signal. There are two top-level configuration options that control the shutdown behavior: `shutdown_timeout` and `shutdown_delay`. ### [](#shutdown-delay)Shutdown delay The `shutdown_delay` option can be used to delay the start of the shutdown procedure. This is useful for pipelines that need a short grace period to have their metrics and traces scraped. While the shutdown delay is in effect, the HTTP metrics endpoint continues to be available for scraping and any active tracers are free to flush remaining traces. The shutdown delay can be interrupted by sending the Redpanda Connect process a second OS interrupt or termination signal. ### [](#shutdown-timeout)Shutdown timeout The `shutdown_timeout` option sets a hard deadline for Redpanda Connect process to gracefully terminate. If this duration is exceeded then the process is forcefully terminated and any messages that were in-flight will be dropped. This option takes effect after the `shutdown_delay` duration has passed if that is enabled. --- # Page 480: Message Batching **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching.md --- # Message Batching > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Message Batching latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/batching page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/batching.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/batching.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect is able to join sources and sinks with sometimes conflicting batching behaviors without sacrificing its strong delivery guarantees. It’s also able to perform powerful [processing functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/windowed_processing/) across batches of messages such as grouping, archiving and reduction. Therefore, batching within Redpanda Connect is a mechanism that serves multiple purposes: 1. [Performance (throughput)](#performance) 2. [Grouped message processing](#grouped-message-processing) 3. [Compatibility (mixing multi and single part message protocols)](#compatibility) ## [](#performance)Performance For most users the only benefit of batching messages is improving throughput over your output protocol. For some protocols this can happen in the background and requires no configuration from you. However, if an output has a `batching` configuration block this means it benefits from batching and requires you to specify how you’d like your batches to be formed by configuring a [batching policy](#batch-policy): ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` However, a small number of inputs such as [`kafka`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka/) must be consumed sequentially (in this case by partition) and therefore benefit from specifying your batch policy at the input level instead: ```yaml input: kafka: addresses: [ todo:9092 ] topics: [ benthos_input_stream ] batching: count: 10 period: 100ms output: kafka: addresses: [ todo:9092 ] topic: benthos_stream ``` Inputs that behave this way are documented as such and have a `batching` configuration block. Sometimes you may prefer to create your batches before processing in order to benefit from [batch wide processing](#grouped-message-processing), in which case if your input doesn’t already support [a batch policy](#batch-policy) you can instead use a [`broker`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/broker/), which also allows you to combine inputs with a single batch policy: ```yaml input: broker: inputs: - resource: foo - resource: bar batching: count: 50 period: 500ms ``` This also works the same with [output brokers](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/broker/). ## [](#grouped-message-processing)Grouped message processing And some processors such as [`while`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/while/) are executed once across a whole batch, you can avoid this behavior with the [`for_each` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/for_each/): ```yaml pipeline: processors: - for_each: - while: at_least_once: true max_loops: 0 check: errored() processors: - catch: [] # Wipe any previous error - resource: foo # Attempt this processor until success ``` There’s a vast number of processors that specialise in operations across batches such as [grouping](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by/) and [archiving](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/). For example, the following processors group a batch of messages according to a metadata field and compresses them into separate `.tar.gz` archives: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_partition") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_partition") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` For more examples of batched (or windowed) processing check out [this document](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/windowed_processing/). ## [](#compatibility)Compatibility Redpanda Connect is able to read and write over protocols that support multiple part messages, and all payloads travelling through Redpanda Connect are represented as a multiple part message. Therefore, all components within Redpanda Connect are able to work with multiple parts in a message as standard. When messages reach an output that _doesn’t_ support multiple parts the message is broken down into an individual message per part, and then one of two behaviors happen depending on the output. If the output supports batch sending messages then the collection of messages are sent as a single batch. Otherwise, Redpanda Connect falls back to sending the messages sequentially in multiple, individual requests. This behavior means that not only can multiple part message protocols be easily matched with single part protocols, but also the concept of multiple part messages and message batches are interchangeable within Redpanda Connect. ### [](#shrinking-batches)Shrinking batches A message batch (or multiple part message) can be broken down into smaller batches using the [`split`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/split/) processor: ```yaml input: # Consume messages that arrive in three parts. resource: foo processors: # Drop the third part - select_parts: parts: [ 0, 1 ] # Then break our message parts into individual messages - split: size: 1 ``` This is also useful when your input source creates batches that are too large for your output protocol: ```yaml input: aws_s3: bucket: todo pipeline: processors: - decompress: algorithm: gzip - unarchive: format: tar # Limit batch sizes to 5MB - split: byte_size: 5_000_000 ``` ## [](#batch-policy)Batch policy When an input or output component has a config field `batching` that means it supports a batch policy. This is a mechanism that allows you to configure exactly how your batching should work on messages before they are routed to the input or output it’s associated with. Batches are considered complete and will be flushed downstream when either of the following conditions are met: - The `byte_size` field is non-zero and the total size of the batch in bytes matches or exceeds it (disregarding metadata.) - The `count` field is non-zero and the total number of messages in the batch matches or exceeds it. - A message added to the batch causes the [`check`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) to return to `true`. - The `period` field is non-empty and the time since the last batch exceeds its value. This allows you to combine conditions: ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` > ⚠️ **CAUTION** > > A batch policy has the capability to _create_ batches, but not to break them down. If your configured pipeline is processing messages that are batched _before_ they reach the batch policy then they may circumvent the conditions you’ve specified here, resulting in sizes you aren’t expecting. If you are affected by this limitation then consider breaking the batches down with a [`split` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/split/) before they reach the batch policy. ### [](#post-batch-processing)Post-batch processing A batch policy also has a field `processors` which allows you to define an optional list of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) to apply to each batch before it is flushed. This is a good place to aggregate or archive the batch into a compatible format for an output: ```yaml output: http_client: url: http://localhost:4195/post batching: count: 10 processors: - archive: format: lines ``` The above config will batch up messages and then merge them into a line delimited format before sending it over HTTP. This is an easier format to parse than the default which would have been [rfc1342](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). During shutdown any remaining messages waiting for a batch to complete will be flushed down the pipeline. --- # Page 481: Contextual Variables **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/contextual-variables.md --- # Contextual Variables > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Contextual Variables latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/contextual-variables page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/contextual-variables.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/contextual-variables.adoc description: Learn about the advantages of using contextual variables, and how to add them to your data pipelines. page-git-created-date: "2025-01-09" page-git-modified-date: "2025-08-08" --- Learn about the advantages of using contextual variables, and how to add them to your data pipelines. ## [](#understanding-contextual-variables)Understanding contextual variables Contextual variables provide an easy way to access information about the environment in which a data pipeline is running and the pipeline itself. You can add any of the following contextual variables to your pipeline configurations: | Contextual variable name | Description | | --- | --- | | ${REDPANDA_BROKERS} | The bootstrap server address of the cluster on which the data pipeline is running. | | ${REDPANDA_ID} | The ID of the cluster on which the data pipeline is running. | | ${REDPANDA_REGION} | The cloud region where the data pipeline is deployed. | | ${REDPANDA_PIPELINE_ID} | The ID of the data pipeline that is currently running. | | ${REDPANDA_PIPELINE_NAME} | The display name of the data pipeline that is currently running. | | ${REDPANDA_SCHEMA_REGISTRY_URL} | The URL of the Schema Registry associated with the cluster on which the data pipeline is running. | Contextual variables are automatically set at runtime, which means that you can reuse them across multiple pipelines and development environments. For example, if you add the contextual variable `${REDPANDA_ID}` to a pipeline configuration, it’s always set to the ID of the cluster on which the data pipeline is running, whether the pipeline is in your development, user acceptance testing, or production environment. This increases the portability of pipeline configurations and reduces maintenance overheads. You can also use contextual variables to improve data traceability. See the [Example pipeline configuration](#example-pipeline-configuration) for full details. ## [](#add-contextual-variable-to-a-data-pipeline)Add contextual variable to a data pipeline Add a contextual variable to any pipeline configuration using the notation `${CONTEXTUAL_VARIABLE_NAME}`, for example: ```yaml output: kafka_franz: seed_brokers: - ${REDPANDA_BROKERS} ``` ### [](#example-pipeline-configuration)Example pipeline configuration For improved data traceability, the following pipeline configuration adds the data pipeline display name (`${REDPANDA_PIPELINE_NAME}`) and ID (`${REDPANDA_PIPELINE_ID}`) to all messages that are processed. The configuration also uses the `$REDPANDA_BROKERS` contextual variable to automatically populate the bootstrap server address of the cluster on which the pipeline is run, which allows Redpanda Connect to write updated messages to the `data` topic defined in the `kafka_franz` output. ```yaml input: generate: mapping: | root.data = "test message" interval: 10s pipeline: processors: - bloblang: | root = this root.source = "${REDPANDA_PIPELINE_NAME}" root.source_id = "${REDPANDA_PIPELINE_ID}" output: kafka_franz: seed_brokers: - ${REDPANDA_BROKERS} topic: data tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: cluster-username password: cluster-password ``` ## [](#suggested-reading)Suggested reading - Learn how to [add secrets to your pipeline](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/). - Try one of our [Redpanda Connect cookbooks](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/). - Choose [connectors for your use case](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/). --- # Page 482: Error Handling **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling.md --- # Error Handling > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Error Handling latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/error_handling page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/error_handling.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/error_handling.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Redpanda Connect supports a range of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/), such as `http` and `aws_lambda`, that may fail when retry attempts are exhausted. By default, when a processor fails, the message data continues through the pipeline mostly unchanged, except for the addition of a metadata flag, which you can use for handling errors. To make processing errors terminal instead, see [Strict error handling](#strict-error-handling). This topic explains some common error-handling patterns, including dropping messages, recovering them with more processing, and routing them to a dead-letter queue. It also shows how to combine these approaches, where appropriate. ## [](#strict-error-handling)Strict error handling To make processing errors terminal instead of relying on error flags, enable strict error handling with the top-level `error_handling` configuration block: ```yaml error_handling: strict: true ``` When `error_handling.strict` is enabled: - A processing error is terminal for the affected message. The message skips the remaining processors in its pipeline. - The failed message is rejected (nacked) at the output rather than written. - A standalone `catch` processor does not recover failed messages, because a failed message short-circuits past it. To recover from an expected error under strict mode, wrap the fallible step and its recovery logic in a [`try_catch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try_catch/). To retry transient failures until they succeed, use a [`retry` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/retry/). > 📝 **NOTE** > > Strict error handling will become the default and only behavior in the next major version of Redpanda Connect. The rest of this topic describes error-handling patterns for the default behavior. ## [](#abandon-on-failure)Abandon on failure You can use the [`try` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try/) to define a list of processors that are executed in sequence. If a processor fails for a particular message, that message skips the remaining processors. For example: - If `processor_1` fails to process a message, that message skips `processor_2` and `processor_3`. - If a message is processed by `processor_1`, but `processor_2` fails, that message skips `processor_3`, and so on. ```yaml pipeline: processors: - try: - resource: processor_1 - resource: processor_2 # Skip if processor_1 fails - resource: processor_3 # Skip if processor_1 or processor_2 fails ``` ## [](#recover-failed-messages)Recover failed messages You can also route failed messages through defined processing steps using a [`catch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch/). For example, if `processor_1` fails to process a message, it is rerouted to `processor_2`. ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - catch: - resource: processor_2 # Processes rerouted messages ``` After messages complete all processing steps defined in the `catch` block, failure flags are removed and they are treated like regular messages. To keep failure flags in messages, you can simulate a `catch` block using a [`switch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/switch/): ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - switch: - check: errored() processors: - resource: processor_2 # Processes rerouted messages ``` ## [](#logging-errors)Logging errors When an error occurs, there may be useful information stored in the error flag. You can use [`error`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#error) Bloblang function interpolations to write this information to logs. You can also add the following Bloblang functions to expose additional details about the processor that triggered the error. - [`error_source_label`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#error_source_label) - [`error_source_name`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#error_source_name) - [`error_source_path`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#error_source_path) For example, this configuration catches processor failures and writes the following information to logs: - The label of the processor (`${!error_source_label()}`) that failed - The cause of the failure (`${!error()}`) ```yaml pipeline: processors: - try: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - log: message: "Processor ${!error_source_label()} failed due to: ${!error()}" ``` You could also add an error message to the message payload: ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - mapping: | root = this root.meta.error = error() ``` ## [](#attempt-until-success)Attempt until success To process a particular message until it is successful, try using a [`retry`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/retry/) processor: ```yaml pipeline: processors: - retry: backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s processors: # Retries this processor until the message is processed, or the maximum elapsed time is reached. - resource: processor_1 ``` ## [](#drop-failed-messages)Drop failed messages To filter out any failed messages from your pipeline, you can use a [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/): ```yaml pipeline: processors: - mapping: root = if errored() { deleted() } ``` The mapping uses the error flag to identify any failed messages in a batch and drops the messages, which propagates acknowledgements (also known as "acks") upstream to the pipeline’s input. ## [](#reject-messages)Reject messages Some inputs, such as `nats`, `gcp_pubsub`, and `amqp_1`, support nacking (rejecting) messages. Rather than delivering unprocessed messages to your output, you can use the [`reject_errored` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/reject_errored/) to perform a nack (or rejection) on them: ```yaml output: reject_errored: resource: processor_1 # Only non-errored messages go here ``` ## [](#route-to-a-dead-letter-queue)Route to a dead-letter queue You can also route failed messages to a different output by nesting the [`reject_errored` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/reject_errored/) within a [`fallback` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/fallback/) ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` If you want to route data differently based on the type of error message, you can use a [`switch` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/switch/): ```yaml output: switch: cases: # Capture specifically cat-related errors - check: errored() && error().contains("meow") output: resource: processor_1 # Capture all other errors - check: errored() output: resource: processor_2 # Finally, route all successfully processed messages here - output: resource: processor_3 ``` Finally, you can attach additional metadata when routing messages to the dead-letter queue, such as the error message. This can be done by running a series of [processors](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/about/) before sending the data to the final [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/). ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - processors: - mutation: | root.error = @fallback_error # Adds the error message before sending the message to the dead-letter queue output resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` --- # Page 483: Field Paths **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths.md --- # Field Paths > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Field Paths latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/field_paths page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/field_paths.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/field_paths.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- Many components within Redpanda Connect allow you to target certain fields using a JSON dot path. The syntax of a path within Redpanda Connect is similar to [JSON Pointers](https://tools.ietf.org/html/rfc6901), except with dot separators instead of slashes (and no leading dot.) When a path is used to set a value any path segment that does not yet exist in the structure is created as an object. For example, if we had the following JSON structure: ```json { "foo": { "bar": 21 } } ``` The query path `foo.bar` would return `21`. The characters `~` (%x7E) and `.` (%x2E) have special meaning in Redpanda Connect paths. Therefore `~` needs to be encoded as `~0` and `.` needs to be encoded as `~1` when these characters appear within a key. For example, if we had the following JSON structure: ```json { "foo.foo": { "bar~bo": { "": { "baz": 22 } } } } ``` The query path `foo~1foo.bar~0bo..baz` would return `22`. ## [](#arrays)Arrays When Redpanda Connect encounters an array while traversing a JSON structure it requires the next path segment to be either an integer of an existing index or, depending on whether the path is used to query or set the target value, the character `*` or `-` respectively. For example, if we had the following JSON structure: ```json { "foo": [ 0, 1, { "bar": 23 } ] } ``` The query path `foo.2.bar` would return `23`. ### [](#querying)Querying When a query reaches an array the character `*` indicates that the query should return the value of the remaining path from each array element (within an array.) ### [](#setting)Setting When an array is reached the character `-` indicates that a new element should be appended to the end of the existing elements, if this character is not the final segment of the path then an object is created. --- # Page 484: Interpolation **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation.md --- # Interpolation > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Interpolation latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/interpolation page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/interpolation.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/interpolation.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Environment variables are not currently supported in Redpanda Connect in Redpanda Cloud, but you can use [contextual variables](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/contextual-variables/) to access information about the environment in which a data pipeline is running, and the pipeline itself. Redpanda Connect allows you to dynamically set config fields with environment variables anywhere within a config file using the syntax `${}` (or `${:}` in order to specify a default value). This is useful for setting environment specific fields such as addresses: ```yaml input: kafka: addresses: [ "${BROKERS}" ] consumer_group: redpanda_connect_consumer topics: [ "haha_business" ] ``` ```sh BROKERS="foo:9092,bar:9092" rpk connect run ./config.yaml ``` If a literal string is required that matches this pattern (`${foo}`) you can escape it with double brackets. For example, the string `${{foo}}` is read as the literal `${foo}`. ## [](#undefined-variables)Undefined variables When an environment variable interpolation is found within a config, does not have a default value specified, and the environment variable is not defined a linting error will be reported. In order to avoid this it is possible to specify environment variable interpolations with an explicit empty default value by adding the colon without a following value, i.e. `${FOO:}` would be equivalent to `${FOO}` and would not trigger a linting error should `FOO` not be defined. ## [](#yaml-tags)YAML tags By default, Redpanda Connect interpolates environment variables as strings. You can use [YAML tags](https://yaml.org/spec/1.2.2/#24-tags) to interpret values as another scalar type, such as integers. ```yaml output: redpanda: # ... batching: count: !!int ${BATCHING_COUNT:500} period: "${BATCHING_PERIOD:1s}" ``` Redpanda Connect supports the [core schema tags](https://yaml.org/spec/1.2.2/#103-core-schema) for scalar types: - `null` - `bool` - `int` - `float` - `str` (default) ## [](#bloblang-queries)Bloblang queries Some Redpanda Connect fields also support [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) function interpolations, which are much more powerful expressions that allow you to query the contents of messages and perform arithmetic. The syntax of a function interpolation is `${!}`, where the contents are a bloblang query (the right-hand-side of a bloblang map) including a range of [functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#functions). For example, with the following config: ```yaml output: kafka: addresses: [ "TODO:6379" ] topic: 'dope-${! json("topic") }' ``` A message with the contents `{"topic":"foo","message":"hello world"}` would be routed to the Kafka topic `dope-foo`. If a literal string is required that matches this pattern (`${!foo}`) then, similar to environment variables, you can escape it with double brackets. For example, the string `${{!foo}}` would be read as the literal `${!foo}`. Bloblang supports arithmetic, boolean operators, coalesce and mapping expressions. For more in-depth details about the language [check out the docs](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/). ## [](#examples)Examples ### [](#reference-metadata)Reference metadata A common usecase for interpolated functions is dynamic routing at the output level using metadata: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("output_topic") } key: ${! meta("key") } ``` ### [](#coalesce-and-mapping)Coalesce and mapping Bloblang supports coalesce and mapping, which makes it easy to extract values from slightly varying data structures: ```yaml pipeline: processors: - cache: resource: foocache operator: set key: '${! json().message.(foo | bar).id }' value: '${! content() }' ``` Here’s a map of inputs to resulting values: {"foo":{"a":{"baz":"from\_a"},"c":{"baz":"from\_c"}}} -> from\_a {"foo":{"b":{"baz":"from\_b"},"c":{"baz":"from\_c"}}} -> from\_b {"foo":{"b":null,"c":{"baz":"from\_c"}}} -> from\_c ### [](#delayed-processing)Delayed processing We have a stream of JSON documents each with a unix timestamp field `doc.received_at` which is set when our platform receives it. We wish to only process messages an hour _after_ they were received. We can achieve this by running the `sleep` processor using an interpolation function to calculate the seconds needed to wait for: ```yaml pipeline: processors: - sleep: duration: '${! 3600 - ( timestamp_unix() - json("doc.created_at").number() ) }s' ``` If the calculated result is less than or equal to zero the processor does not sleep at all. If the value of `doc.created_at` is a string then our method `.number()` will attempt to parse it into a number. --- # Page 485: Metadata **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/metadata.md --- # Metadata > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Metadata latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/metadata page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/metadata.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/metadata.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- In Redpanda Connect each message has raw contents and metadata, which is a map of key/value pairs representing an arbitrary amount of complementary data. When an input protocol supports attributes or metadata they will automatically be added to your messages, refer to the respective input documentation for a list of metadata keys. When an output supports attributes or metadata any metadata key/value pairs in a message will be sent (subject to service limits). ## [](#editing-metadata)Editing metadata Redpanda Connect allows you to add and remove metadata using the [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/). For example, you can do something like this in your pipeline: ```yaml pipeline: processors: - mapping: | # Remove all existing metadata from messages meta = deleted() # Add a new metadata field `time` from the contents of a JSON # field `event.timestamp` meta time = event.timestamp ``` You can also use [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) to delete individual metadata keys with: ```bloblang meta foo = deleted() ``` Or do more interesting things like remove all metadata keys with a certain prefix: ```bloblang meta = @.filter(kv -> !kv.key.has_prefix("kafka_")) ``` ## [](#using-metadata)Using metadata Metadata values can be referenced in any field that supports [interpolation functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/). For example, you can route messages to Kafka topics using interpolation of metadata keys: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } ``` Redpanda Connect also allows you to conditionally process messages based on their metadata with the [`switch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/switch/): ```yaml pipeline: processors: - switch: - check: '@doc_type == "nested"' processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ foo, bar, baz ] args_mapping: | root = [ this.document.foo, this.document.bar, @kafka_topic, ] # In: {"document":{"foo":"value1","bar":"value2"}} ``` ## [](#restricting-metadata)Restricting metadata Outputs that support metadata, headers or some other variant of enriched fields on messages will attempt to send all metadata key/value pairs by default. However, sometimes it’s useful to refer to metadata fields at the output level even though we do not wish to send them with our data. In this case it’s possible to restrict the metadata keys that are sent with the field `metadata.exclude_prefixes` within the respective output config. For example, if we were sending messages to kafka using a metadata key `target_topic` to determine the topic but we wished to prevent that metadata key from being sent as a header we could use the following configuration: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } metadata: exclude_prefixes: - target_topic ``` And when the list of metadata keys that we do _not_ want to send is large it can be helpful to use a [Bloblang mapping](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) in order to give all of these "private" keys a common prefix: ```yaml pipeline: processors: # Has an explicit list of public metadata keys, and everything else is given # an underscore prefix. - mapping: | let allowed_meta = [ "foo", "bar", "baz", ] meta = @.map_each_key(key -> if !$allowed_meta.contains(key) { "_" + key }) output: kafka: addresses: [ TODO ] topic: ${! meta("_target_topic") } metadata: exclude_prefixes: [ "_" ] ``` --- # Page 486: Monitor Data Pipelines on BYOC and Dedicated Clusters **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/monitor-connect.md --- # Monitor Data Pipelines on BYOC and Dedicated Clusters > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Monitor Data Pipelines on BYOC and Dedicated Clusters latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/monitor-connect page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/monitor-connect.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/monitor-connect.adoc description: Configure Prometheus monitoring of your data pipelines on BYOC clusters. page-git-created-date: "2024-09-09" page-git-modified-date: "2024-12-03" --- You can configure monitoring on BYOC and Dedicated clusters to understand the behavior, health, and performance of your data pipelines. Redpanda Connect automatically exports [detailed metrics for each component of your data pipeline](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/) to a Prometheus endpoint, along with metrics for all other cluster services. You don’t need to update the configuration of your pipeline. ## [](#configure-prometheus)Configure Prometheus To monitor a BYOC cluster in [Prometheus](https://prometheus.io/): 1. On the Redpanda Cloud **Overview** page for your cluster, under **How to connect**, click the **Prometheus** tab. 2. Click the copy icon next to **Prometheus YAML** to copy the contents to your clipboard. The YAML contains the Prometheus scrape target configuration, as well as authentication, for the cluster. ```yaml - job_name: redpandaCloud-sample static_configs: - targets: - console-..fmc.cloud.redpanda.com metrics_path: /api/cloud/prometheus/public_metrics basic_auth: username: prometheus password: "" scheme: https ``` 3. Save the YAML configuration to Prometheus replacing the following placeholders: - `.`: ID and identifier from the **HTTPS endpoint**. - ``: Copy and paste the onscreen Prometheus password. Metrics from Redpanda endpoints are scraped into Prometheus. The metrics for each data pipeline are labelled by pipeline ID. ## [](#use-redpanda-monitoring-examples)Use Redpanda monitoring examples For hands-on learning, Redpanda provides a repository with examples of monitoring Redpanda with Prometheus and Grafana: [redpanda-data/observability](https://github.com/redpanda-data/observability/tree/main/cloud). ![Example Redpanda Connect Dashboard^](https://docs.redpanda.com/cloud-data-platform/shared/_images/redpanda_connect_dashboard.png) It includes [an example Grafana dashboard for Redpanda Connect](https://github.com/redpanda-data/observability/blob/main/grafana-dashboards/Redpanda-Connect-Dashboard.json) and a [sandbox environment](https://github.com/redpanda-data/observability#sandbox-environment) in which you launch a Dockerized Redpanda cluster and create a custom workload to monitor with dashboards. --- # Page 487: Process Pipelines **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/processing_pipelines.md --- # Process Pipelines > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Process Pipelines latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/processing_pipelines page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/processing_pipelines.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/processing_pipelines.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- If you have processors that are heavy on CPU and aren’t specific to a certain input or output they are best suited for the pipeline section. It is advantageous to use the pipeline section as it allows you to set an explicit number of parallel threads of execution: ```yaml input: resource: foo pipeline: threads: 4 processors: - mapping: | root = this fans = fans.map_each(match { this.obsession > 0.5 => this _ => deleted() }) output: resource: bar ``` If the field `threads` is set to `-1` (the default) it will automatically match the number of logical CPUs available. By default almost all Redpanda Connect sources will utilize as many processing threads as have been configured, which makes horizontal scaling easy. --- # Page 488: Manage Pipeline Resources on Clusters **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/resource-management.md --- # Manage Pipeline Resources on Clusters > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Pipeline Resources on Clusters latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/resource-management page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/resource-management.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/resource-management.adoc description: Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. page-git-created-date: "2024-12-18" page-git-modified-date: "2026-05-26" --- Learn how to set an initial resource limit for a standard data pipeline (excluding Ollama AI components) and how to manually scale the pipeline’s resources to improve performance. ## [](#prerequisites)Prerequisites - A running Redpanda Cloud cluster. - An estimate of the throughput of your data pipeline. You can get some basic statistics by running your data pipeline locally using the [`benchmark` processor](https://docs.redpanda.com/connect/components/processors/benchmark/). ### [](#understanding-compute-units)Understanding compute units A compute unit allocates a specific amount of server resources (CPU and memory) to a data pipeline to handle message throughput. By default, each pipeline is allocated one compute unit, which includes 0.1 CPU (100 milliCPU or `100m`) and 400 MB (`400M`) of memory. For sizing purposes, one compute unit supports an estimated message throughput of 1 MB/s. However, actual performance depends on the complexity of a pipeline, including the components it contains and the processing it does. You can allocate a maximum of 72 compute units per pipeline. You can add compute units in increments of one up to 15 compute units. Beyond this, scaling options increase to 33 and then to 72 compute units. This scaling strategy is based on the number of machine cores required to provision resources, which scale from two to four, and then to eight cores. Server resources are charged at an [hourly rate in compute unit hours (compute/hour)](https://docs.redpanda.com/cloud-data-platform/billing/billing/#redpanda-connect-pipeline-metrics). | Number of compute units | CPU | Memory | | --- | --- | --- | | 1 | 0.1 CPU (100m) | 400 MB (400M) | | 2 | 0.2 CPU (200m) | 800 MB (800M) | | 3 | 0.3 CPU (300m) | 1.2 GB (1200M) | | 4 | 0.4 CPU (400m) | 1.6 GB (1600M) | | 5 | 0.5 CPU (500m) | 2.0 GB (2000M) | | 6 | 0.6 CPU (600m) | 2.4 GB (2400M) | | 7 | 0.7 CPU (700m) | 2.8 GB (2800M) | | 8 | 0.8 CPU (800m) | 3.2 GB (3200M) | | 9 | 0.9 CPU (900m) | 3.6 GB (3600M) | | 10 | 1.0 CPU (1000m) | 4.0 GB (4000M) | | 11 | 1.1 CPU (1100m) | 4.4 GB (4400M) | | 12 | 1.2 CPU (1200m) | 4.8 GB (4800M) | | 13 | 1.3 CPU (1300m) | 5.2 GB (5200M) | | 14 | 1.4 CPU (1400m) | 5.6 GB (5600M) | | 15 | 1.5 CPU (1500m) | 6.0 GB (6000M) | | 33 | 3.3 CPU (3300m) | 13.2 GB (13200M) | | 72 | 7.2 CPU (7200m) | 28.8 GB (28800M) | > 📝 **NOTE** > > A GPU machine is automatically assigned to each pipeline that contains embedded Ollama AI components. By default, GPU-enabled pipelines are allocated eight compute units. For larger workloads, you can scale them up to a maximum of 30 compute units. ### [](#set-an-initial-resource-limit)Set an initial resource limit When you create a data pipeline, you can allocate a fixed amount of server resources to it using compute units. > 📝 **NOTE** > > If your pipeline reaches the CPU limit, it becomes throttled, which reduces the data processing rate. If it reaches the memory limit, the pipeline restarts. To set an initial resource limit: 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. On the **Clusters** page, select the cluster where you want to add a pipeline. 3. Go to the **Connect** page. 4. Select the **Redpanda Connect** tab. 5. Click **Create pipeline**. 6. Enter details for your pipeline, including a short name and description. 7. For **Compute units**, leave the default **1** compute unit to experiment with pipelines that create low message volumes. For higher throughputs, you can allocate a maximum of 72 compute units. 8. For **Configuration**, paste your pipeline configuration and click **Create** to run it. ### [](#scale-resources)Scale resources View the server resources allocated to a data pipeline, and manually scale those resources to improve performance or decrease resource consumption. To view resources already allocated to a data pipeline: #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the cluster where the pipeline is set up. 3. On the **Connect** page, select your pipeline and look at the value for **Resources**. - CPU resources are displayed first, in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. - Memory is displayed next in megabytes. For example, `1` compute unit is `400M` or 400 MB. #### Data Plane API 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API. 2. Make a request to [`GET /v1/redpanda-connect/pipelines`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_listpipelines), which lists details of all pipelines on your cluster by ID. - Memory (`memory_shares`) is displayed in megabytes. For example, `1` compute unit is `400M` or 400 MB. - CPU resources (`cpu_shares`) are displayed in milliCPU. For example, `1` compute unit is `100m` or 0.1 CPU. To scale the resources for a pipeline: #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the cluster where the pipeline is set up. 3. On the **Connect** page, select your pipeline and click **Edit**. 4. For **Compute units**, update the number of compute units. You can allocate a maximum of 72 compute units per pipeline. 5. Click **Update** to apply your changes. The specified resources are available immediately. #### Data Plane API You can only update CPU resources using the Data Plane API. For every 0.1 CPU that you allocate, Redpanda Cloud automatically reserves 400 MB of memory for the exclusive use of the pipeline. 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API, if you haven’t already. 2. Make a request to [`GET /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_getpipeline), including the ID of the pipeline you want to update. You’ll use the returned values in the next step. 3. Now make a request to [`PUT /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_updatepipeline), to update the pipeline resources: - Reuse the values returned by your `GET` request to populate the request body. - Replace the `cpu_shares` value with the resources you want to allocate, and enter any valid value for `memory_shares`. This example allocates 0.2 CPU or 200 milliCPU to a data pipeline. For `cpu_shares`, `0.1` CPU is the minimum allocation. ```bash curl -X PUT "https:///v1/redpanda-connect/pipelines/xxx..." \ -H 'accept: application/json'\ -H 'authorization: Bearer xxx...' \ -H "content-type: application/json" \ -d '{ "config_yaml": "input:\n generate:\n interval: 1s\n mapping: |\n root.id = uuid_v4()\n root.user.name = fake(\"name\")\n root.user.email = fake(\"email\")\n root.content = fake(\"paragraph\")\n\npipeline:\n processors:\n - mutation: |\n root.title = \"PRIVATE AND CONFIDENTIAL\"\n\noutput:\n kafka_franz:\n seed_brokers:\n - seed-j888.byoc.prd.cloud.redpanda.com:9092\n sasl:\n mechanism: SCRAM-SHA-256\n password: password\n username: connect\n topic: processed-emails\n tls:\n enabled: true\n", "description": "Email processor", "display_name": "emailprocessor-pipeline", "resources": { "memory_shares": "800M", "cpu_shares": "200m" } }' ``` A successful response shows the updated resource allocations with the `cpu_shares` value returned in milliCPU. 4. Make a request to [`GET /v1/redpanda-connect/pipelines`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_listpipelines) to verify your pipeline resource updates. --- # Page 489: Manage Secrets **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management.md --- # Manage Secrets > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Secrets latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/secret-management page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/secret-management.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/secret-management.adoc description: Learn how to manage secrets in Redpanda Connect using the Cloud Console, Data Plane API, or Terraform, and learn how to add them to your data pipelines. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-08-07" --- Learn how to manage secrets in Redpanda Connect, and how to add them to your data pipelines without exposing them. Secrets are stored in the secret management solution of your cloud provider and are retrieved when you run a pipeline configuration that references them. ## [](#manage-secrets)Manage secrets You can manage secrets from the Cloud UI or the Data Plane API. If you manage your Redpanda Cloud resources with Terraform, you can also create secrets with the `redpanda_secret` resource. See [Manage cluster secrets](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#manage-cluster-secrets). ### [](#create-a-secret)Create a secret You can create a secret and reference it in multiple data pipelines on the same cluster. #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the **Secrets Store** page. 3. Click **Create secret**. 4. For **ID**, enter a name for the secret. You cannot rename the secret once it is created. 5. For **Value**, enter the secret you need to add. 6. For **Scopes**, select Redpanda Connect. 7. Optionally, add labels to help organize your secrets. 8. Click **Create**. You can now [add the secret to your data pipeline](#add-a-secret-to-a-data-pipeline). #### Data Plane API You must use a Base64-encoded secret. 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API. 2. Make a request to [`POST /v1/secrets`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_createsecret). ```bash curl -X POST "https:///v1/secrets" \ -H 'accept: application/json'\ -H 'authorization: Bearer '\ -H 'content-type: application/json' \ -d '{"id":"","scopes":["SCOPE_REDPANDA_CONNECT"],"secret_data":""}' ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The API key you generated during authentication. - ``: The ID or name of the secret you want to add. Use only the following characters: `^[A-Z][A-Z0-9_]*$`. - ``: The Base64-encoded secret. - This scope: `"SCOPE_REDPANDA_CONNECT"`. The response returns the name of the secret and the scope `"SCOPE_REDPANDA_CONNECT"`. You can now [add the secret to your data pipeline](#add-a-secret-to-a-data-pipeline). ### [](#update-a-secret)Update a secret You can only update the secret value, not its name. > 📝 **NOTE** > > Changes to secret values do not take effect until a pipeline is restarted. #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the **Secrets Store** page. 3. Find the secret you want to update, and click the edit icon. 4. Enter the new secret value or labels, and click **Update**. 5. Start and stop any pipelines that reference the secret. #### Data Plane API You must use a Base64-encoded secret. 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API. 2. Make a request to [`PUT /v1/secrets/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_updatesecret). ```bash curl -X PUT "https:///v1/secrets/" \ -H 'accept: application/json'\ -H 'authorization: Bearer '\ -H 'content-type: application/json' \ -d '{"scopes":["SCOPE_REDPANDA_CONNECT"],"secret_data":""}' ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The name of the secret you want to update. - ``: The API key you generated during authentication. - This scope: `"SCOPE_REDPANDA_CONNECT"`. - ``: Your new Base64-encoded secret. The response returns the name of the secret and the scope `"SCOPE_REDPANDA_CONNECT"`. ### [](#delete-a-secret)Delete a secret Before you delete a secret, make sure that you remove references to it from your data pipelines. > 📝 **NOTE** > > Changes do not affect pipelines that are already running. #### Cloud UI 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the **Secrets Store** page. 3. Find the secret you want to remove, and click the delete icon. 4. Confirm your deletion. #### Data Plane API 1. [Authenticate and get the base URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart) for the Data Plane API. 2. Make a request to [`DELETE /v1/secrets/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_deletesecret). ```bash curl -X DELETE "https:///v1/secrets/" \ -H 'accept: application/json'\ -H 'authorization: Bearer '\ ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The name of the secret you want to delete. - ``: The API key you generated during authentication. ## [](#add-a-secret-to-a-data-pipeline)Add a secret to a data pipeline ### Cloud UI 1. Go to the **Connect** page, and create a pipeline (or open an existing pipeline to edit). 2. Click the **Secret** button to add a new or existing secret to the pipeline. ### Data Plane API You can add a secret to any pipeline in your cluster using the notation `${secrets.SECRET_NAME}`. For example: ```yml sasl: - mechanism: SCRAM-SHA-256 username: "user" password: "${secrets.PASSWORD}" ``` --- # Page 490: Unit Testing **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/unit_testing.md --- # Unit Testing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Unit Testing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/unit_testing page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/unit_testing.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/unit_testing.adoc page-git-created-date: "2024-09-09" page-git-modified-date: "2026-05-26" --- The Redpanda Connect service offers a command `rpk connect test` for running unit tests on sections of a configuration file. This makes it easy to protect your config files from regressions over time. ## [](#writing-a-test)Writing a test Let’s imagine we have a configuration file `foo.yaml` containing some processors: ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: '"%vend".format(content().uppercase().string())' output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` One way to write our unit tests for this config is to accompany it with a file of the same name and extension but suffixed with `_benthos_test`, which in this case would be `foo_benthos_test.yaml`. ```yml tests: - name: example test target_processors: '/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` Under `tests` we have a list of any number of unit tests to execute for the config file. Each test is run in complete isolation, including any resources defined by the config file. Tests should be allocated a unique `name` that identifies the feature being tested. The field `target_processors` is either the label of a processor to test, or a [JSON Pointer](https://tools.ietf.org/html/rfc6901) that identifies the position of a processor, or list of processors, within the file which should be executed by the test. For example a value of `foo` would target a processor with the label `foo`, and a value of `/input/processors` would target all processors within the input section of the config. The field `environment` allows you to define an object of key/value pairs that set environment variables to be evaluated during the parsing of the target config file. These are unique to each test, allowing you to test different environment variable interpolation combinations. The field `input_batch` lists one or more messages to be fed into the targeted processors as a batch. Each message of the batch may have its raw content defined as well as metadata key/value pairs. For the common case where the messages are in JSON format, you can use `json_content` instead of `content` to specify the message structurally rather than verbatim. The field `output_batches` lists any number of batches of messages which are expected to result from the target processors. Each batch lists any number of messages, each one defining [`conditions`](#output-conditions) to describe the expected contents of the message. If the number of batches defined does not match the resulting number of batches the test will fail. If the number of messages defined in each batch does not match the number in the resulting batches the test will fail. If any condition of a message fails then the test fails. ### [](#inline-tests)Inline tests Sometimes it’s more convenient to define your tests within the config being tested. This is fine, simply add the `tests` field to the end of the config being tested. ### [](#bloblang-tests)Bloblang tests Sometimes when working with large [Bloblang mappings](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) it’s preferred to have the full mapping in a separate file to your Redpanda Connect configuration. In this case it’s possible to write unit tests that target and execute the mapping directly with the field `target_mapping`, which when specified is interpreted as either an absolute path or a path relative to the test definition file that points to a file containing only a Bloblang mapping. For example, if we were to have a file `cities.blobl` containing a mapping: ```bloblang root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` We can accompany it with a test file `cities_test.yaml` containing a regular test definition: ```yml tests: - name: test cities mapping target_mapping: './cities.blobl' environment: {} input_batch: - content: | { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } output_batches: - - json_equals: {"Cities": "Bellevue, Olympia, Seattle"} ``` And execute this test the same way we execute other Redpanda Connect tests (`rpk connect test ./dir/cities_test.yaml`, `rpk connect test ./dir/…​`, etc). ### [](#fragmented-tests)Fragmented tests Sometimes the number of tests you need to define in order to cover a config file is so vast that it’s necessary to split them across multiple test definition files. This is possible but Redpanda Connect still requires a way to detect the configuration file being targeted by these fragmented test definition files. In order to do this we must prefix our `target_processors` field with the path of the target relative to the definition file. The syntax of `target_processors` in this case is a full [JSON Pointer](https://tools.ietf.org/html/rfc6901) that should look something like `target.yaml#/pipeline/processors`. For example, if we saved our test definition above in an arbitrary location like `./tests/first.yaml` and wanted to target our original `foo.yaml` config file, we could do that with the following: ```yml tests: - name: example test target_processors: '../foo.yaml#/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` ## [](#input-definitions)Input Definitions ### [](#content)`content` Sets the raw content of the message. ### [](#json_content)`json_content` ```yml json_content: foo: foo value bar: [ element1, 10 ] ``` Sets the raw content of the message to a JSON document matching the structure of the value. ### [](#file_content)`file_content` ```yml file_content: ./foo/bar.txt ``` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. ### [](#metadata)`metadata` A map of key/value pairs that sets the metadata values of the message. ## [](#output-conditions)Output Conditions ### [](#bloblang)`bloblang` ```yml bloblang: 'this.age > 10 && @foo.length() > 0' ``` Executes a [Bloblang expression](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) on a message, if the result is anything other than a boolean equalling `true` the test fails. ### [](#content_equals)`content_equals` ```yml content_equals: example content ``` Checks the full raw contents of a message against a value. ### [](#content_matches)`content_matches` ```yml content_matches: "^foo [a-z]+ bar$" ``` Checks whether the full raw contents of a message matches a regular expression (re2). ### [](#metadata_equals)`metadata_equals` ```yml metadata_equals: example_key: example metadata value ``` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. ### [](#file_equals)`file_equals` ```yml file_equals: ./foo/bar.txt ``` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. ### [](#file_json_equals)`file_json_equals` ```yml file_json_equals: ./foo/bar.json ``` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. ### [](#json_equals)`json_equals` ```yml json_equals: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. You can also structure the condition content as YAML and it will be converted to the equivalent JSON document for testing: ```yml json_equals: key: value ``` ### [](#json_contains)`json_contains` ```yml json_contains: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. ## [](#running-tests)Running tests Executing tests for a specific config can be done by pointing the subcommand `test` at either the config to be tested or its test definition, e.g. `rpk connect test ./config.yaml` and `rpk connect test ./config_benthos_test.yaml` are equivalent. The `test` subcommand also supports wildcard patterns e.g. `rpk connect test ./foo/*.yaml` will execute all tests within matching files. In order to walk a directory tree and execute all tests found you can use the shortcut `./…​`, e.g. `rpk connect test ./…​` will execute all tests found in the current directory, any child directories, and so on. If you want to allow components to write logs at a provided level to stdout when running the tests, you can use `rpk connect test --log `. Please consult the [logger docs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/logger/about/) for further details. ## [](#mocking-processors)Mocking processors BETA: This feature is currently in a BETA phase, which means breaking changes could be made if a fundamental issue with the feature is found. Sometimes you’ll want to write tests for a series of processors, where one or more of them are networked (or otherwise stateful). Rather than creating and managing mocked services you can define mock versions of those processors in the test definition. For example, if we have a config with the following processors: ```yaml pipeline: processors: - mapping: 'root = "simon says: " + content()' - label: get_foobar_api http: url: http://example.com/foobar verb: GET - mapping: 'root = content().uppercase()' ``` Rather than create a fake service for the `http` processor to interact with we can define a mock in our test definition that replaces it with a [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/). Mocks are configured as a map of labels that identify a processor to replace and the config to replace it with: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: get_foobar_api: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` With the above test definition the `http` processor will be swapped out for `mapping: 'root = content().string() + " this is some mock content"'`. For the purposes of mocking it is recommended that you use a [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) that simply mutates the message in a way that you would expect the mocked processor to. > 📝 **NOTE** > > It’s not currently possible to mock components that are imported as separate resource files (using `--resource`/`-r`). It is recommended that you mock these by maintaining separate definitions for test purposes (`-r "./test/*.yaml"`). ### [](#more-granular-mocking)More granular mocking It is also possible to target specific fields within the test config by [JSON pointers](https://tools.ietf.org/html/rfc6901) as an alternative to labels. The following test definition would create the same mock as the previous: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: /pipeline/processors/1: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` ## [](#fields)Fields The schema of a template file is as follows: ### [](#tests)`tests` A list of one or more unit tests to execute. **Type**: `array` ### [](#tests-name)`tests[].name` The name of the test, this should be unique and give a rough indication of what behavior is being tested. **Type**: `string` ### [](#tests-environment)`tests[].environment` An optional map of environment variables to set for the duration of the test. **Type**: `object` ### [](#tests-target_processors)`tests[].target_processors` A \[JSON Pointer\]\[json-pointer\] that identifies the specific processors which should be executed by the test. The target can either be a single processor or an array of processors. Alternatively a resource label can be used to identify a processor. It is also possible to target processors in a separate file by prefixing the target with a path relative to the test file followed by a # symbol. **Type**: `string` **Default**: `"/pipeline/processors"` ```yml # Examples target_processors: foo_processor target_processors: /pipeline/processors/0 target_processors: target.yaml#/pipeline/processors target_processors: target.yaml#/pipeline/processors ``` ### [](#tests-target_mapping)`tests[].target_mapping` A file path relative to the test definition path of a Bloblang file to execute as an alternative to testing processors with the `target_processors` field. This allows you to define unit tests for Bloblang mappings directly. **Type**: `string` **Default**: `""` ### [](#tests-mocks)`tests[].mocks` An optional map of processors to mock. Keys should contain either a label or a JSON pointer of a processor that should be mocked. Values should contain a processor definition, which will replace the mocked processor. Most of the time you’ll want to use a \[`mapping` processor\]\[processors.mapping\] here, and use it to create a result that emulates the target processor. **Type**: `object` ```yml # Examples mocks: get_foobar_api: mapping: root = content().string() + " this is some mock content" mocks: /pipeline/processors/1: mapping: root = content().string() + " this is some mock content" ``` ### [](#tests-input_batch)`tests[].input_batch` Define a batch of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `array` ### [](#tests-input_batch-content)`tests[].input_batch[].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batch-json_content)`tests[].input_batch[].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batch-file_content)`tests[].input_batch[].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batch-metadata)`tests[].input_batch[].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-input_batches)`tests[].input_batches` Define a series of batches of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `two-dimensional array` ### [](#tests-input_batches-content)`tests[].input_batches[][].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batches-json_content)`tests[].input_batches[][].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batches-file_content)`tests[].input_batches[][].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batches-metadata)`tests[].input_batches[][].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-output_batches)`tests[].output_batches` List of output batches. **Type**: `two-dimensional array` ### [](#tests-output_batches-bloblang)`tests[].output_batches[][].bloblang` Executes a Bloblang mapping on the output message, if the result is anything other than a boolean equalling `true` the test fails. **Type**: `string` ```yml # Examples bloblang: this.age > 10 && @foo.length() > 0 ``` ### [](#tests-output_batches-content_equals)`tests[].output_batches[][].content_equals` Checks the full raw contents of a message against a value. **Type**: `string` ### [](#tests-output_batches-content_matches)`tests[].output_batches[][].content_matches` Checks whether the full raw contents of a message matches a regular expression (re2). **Type**: `string` ```yml # Examples content_matches: ^foo [a-z]+ bar$ ``` ### [](#tests-output_batches-metadata_equals)`tests[].output_batches[][].metadata_equals` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. **Type**: `object` ```yml # Examples metadata_equals: example_key: example metadata value ``` ### [](#tests-output_batches-file_equals)`tests[].output_batches[][].file_equals` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_equals: ./foo/bar.txt ``` ### [](#tests-output_batches-file_json_equals)`tests[].output_batches[][].file_json_equals` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_equals: ./foo/bar.json ``` ### [](#tests-output_batches-json_equals)`tests[].output_batches[][].json_equals` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. **Type**: `object` ```yml # Examples json_equals: key: value ``` ### [](#tests-output_batches-json_contains)`tests[].output_batches[][].json_contains` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. **Type**: `object` ```yml # Examples json_contains: key: value ``` ### [](#tests-output_batches-file_json_contains)`tests[].output_batches[][].file_json_contains` Checks that both the message and the file contents are valid JSON documents, and that the message is a superset of the condition. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_contains: ./foo/bar.json ``` --- # Page 491: Windowed Processing **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/windowed_processing.md --- # Windowed Processing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Windowed Processing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/configuration/windowed_processing page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/configuration/windowed_processing.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/configuration/windowed_processing.adoc description: Learn how to process periodic windows of messages with Redpanda Connect. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- A window is a batch of messages made with respect to time, with which we are able to perform processing that can analyze or aggregate the messages of the window. This is useful in stream processing as the dataset is never "complete", and therefore in order to perform analysis against a collection of messages we must do so by creating a continuous feed of windows (collections), where our analysis is made against each window. For example, given a stream of messages relating to cars passing through various traffic lights: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` Windowing allows us to produce a stream of messages representing the total traffic for each light every hour: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "unique_cars": 15, "passengers": 43 } ``` ## [](#creating-windows)Creating windows The first step in processing windows is producing the windows themselves, this can be done by configuring a window producing buffer after your input: ### System A `system_window` buffer creates windows by following the system clock of the running machine. Windows will be created and emitted at predictable times, but this also means windows for historic data will not be emitted and therefore prevents backfills of traffic data: ```yaml input: kafka: addresses: [ TODO ] topics: [ traffic_data ] consumer_group: traffic_consumer checkpoint_limit: 1000 buffer: system_window: timestamp_mapping: root = this.created_at size: 1h allowed_lateness: 3m ``` For more information about this buffer refer to the `system_window` buffer docs. ## [](#grouping)Grouping With a window buffer chosen our stream of messages will be emitted periodically as batches of all messages that fit within each window. Since we want to analyse the window separately for each traffic light we need to expand this single batch out into one for each traffic light identifier within the window. For that purpose we have two processor options: [`group_by`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by/) and [`group_by_value`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/group_by_value/). In our case we want to group by the value of the field `traffic_light` of each message, which we can do with the following: ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } ``` ## [](#aggregating)Aggregating Once our window has been grouped the next step is to calculate the aggregated passenger and unique cars counts. For this purpose the Redpanda Connect [mapping language Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) comes in handy as the method [`from_all`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#from_all) executes the target function against the entire batch and returns an array of the values, allowing us to mutate the result with chained methods such as [`sum`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#sum): ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } - mapping: | let is_first_message = batch_index() == 0 root.traffic_light = this.traffic_light root.created_at = @window_end_timestamp root.total_cars = if $is_first_message { json("registration_plate").from_all().unique().length() } root.passengers = if $is_first_message { json("passengers").from_all().sum() } # Only keep the first batch message containing the aggregated results. root = if ! $is_first_message { deleted() } ``` [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) is very powerful, and by using [`from`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#from) and [`from_all`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#from_all) it’s possible to perform a wide range of batch-wide processing. If you fancy a challenge try updating the above mapping to only count passengers from the first journey of each registration plate in the window (hint: the [`fold` method](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#fold) might come in handy). --- # Page 492: Redpanda Connect Quickstart **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart.md --- # Redpanda Connect Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Connect Quickstart latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/connect-quickstart page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/connect-quickstart.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/connect-quickstart.adoc description: Learn how to quickly start building data pipelines with Redpanda Connect. page-topic-type: tutorial personas: streaming_developer learning-objective-1: Build a producer pipeline that generates and publishes data to a topic learning-objective-2: Build a consumer pipeline that reads, transforms, and logs data from a topic page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-12" --- In this quickstart, you build data pipelines to generate, transform, and handle streaming data end-to-end. You create two pipelines: one that generates dad jokes and writes them to a topic in your cluster, and another that reads those jokes and gives each one a random "cringe rating". After completing this quickstart, you will be able to: - Build a producer pipeline that generates and publishes data to a topic - Build a consumer pipeline that reads, transforms, and logs data from a topic ## [](#prerequisites)Prerequisites You must have a Redpanda Cloud account with a Serverless, Dedicated, or standard BYOC cluster. If you don’t already have an account, [sign up for a free trial](https://redpanda.com/try-redpanda/cloud-trial). > 📝 **NOTE** > > Clusters can create up to 100 pipelines. For additional pipelines, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). ## [](#quickstart-pipelines)Quickstart pipelines This quickstart creates the following pipelines: - The first pipeline produces dad jokes and writes them to a topic in your cluster. - The second pipeline consumes those dad jokes and gives each one a random "cringe rating" from 1-10. The **producer pipeline** uses the following Redpanda Connect components: | Component type | Component | Purpose | | --- | --- | --- | | Input | generate | Creates jokes | | Output | redpanda | Writes messages to your topic | | Processor | log | Logs generated messages | | Processor | catch | Catches errors | The **consumer pipeline** uses the following Redpanda Connect components: | Component type | Component | Purpose | | --- | --- | --- | | Input | redpanda | Reads messages from your topic | | Output | drop | Drops the processed messages | | Processor | bloblang | Processes ratings | | Processor | log | Logs processed messages | | Processor | catch | Catches errors | > 💡 **TIP** > > The pipeline editor provides an IDE-like experience for creating pipelines. After a component has been added, you can click the leaf icon in the left sidebar to open its documentation. ![Redpanda Connect user interface](https://docs.redpanda.com/cloud-data-platform/shared/_images/connect_ui.png) ## [](#build-a-producer-pipeline)Build a producer pipeline Every pipeline requires an input and an output in a configuration file. You can select components in the left sidebar and customize the YAML in the editor. To create the producer pipeline: 1. Go to the **Connect** page for your cluster and click **Create a pipeline**. 2. Enter this name for the pipeline: `joke-generator-producer`. 3. In the left sidebar, click **Add input +** and search for and select the `generate` input connector. The YAML for this connector appears in the editor. 4. Click **Add output +** and search for and select the `redpanda` output connector. The YAML for this connector also appears in the editor. 5. The `redpanda` connector requires a Redpanda topic and user: 1. In the `redpanda` output connector, click **Topic +** to create a new topic. Toggle to **New** and enter `dad-jokes` for the topic name. Click **Add**. 2. In the `redpanda` output connector, click **User +** to create a new user. Toggle to **New** and enter `connect` for the username. Click **Add**. 6. Replace the generated YAML in the editor with the following. This configuration includes the `log` and `catch` processors and the `mapping` for joke generation. Bloblang is Redpanda Connect’s scripting language used to add logic. ```yaml input: generate: interval: 5s count: 0 mapping: | let jokes = [ "Why don't scientists trust atoms? Because they make up everything!", "I'm reading a book about anti-gravity. It's impossible to put down!", "Why did the scarecrow win an award? He was outstanding in his field!", "What do you call a fake noodle? An impasta!", "Why don't eggs tell jokes? They'd crack each other up!", "I used to play piano by ear, but now I use my hands.", "What do you call a bear with no teeth? A gummy bear!", "Why did the bicycle fall over? It was two tired!", "What do you call a fish wearing a crown? A king fish!", "Why don't skeletons fight each other? They don't have the guts!", "What do you call cheese that isn't yours? Nacho cheese!", "Why can't you hear a pterodactyl using the bathroom? Because the 'p' is silent!", "What did the ocean say to the beach? Nothing, it just waved!", "Why did the math book look sad? It had too many problems!", "What do you call a sleeping bull? A bulldozer!", "How do you organize a space party? You planet!", "What's orange and sounds like a parrot? A carrot!", "Why did the coffee file a police report? It got mugged!", "What do you call a can opener that doesn't work? A can't opener!", "Why don't oysters donate to charity? Because they're shellfish!" ] let joke_index = random_int() % $jokes.length() root.joke = $jokes.index($joke_index) root.id = uuid_v4() root.timestamp = now() root.source = "dad-joke-generator" root.joke_length = root.joke.length() pipeline: processors: - log: level: INFO message: "📝 Generating joke: ${! json(\"joke\") }" - catch: - log: level: ERROR message: "❌ Error generating joke: ${! error() }" output: redpanda: seed_brokers: # Optional - ${REDPANDA_BROKERS} tls: enabled: true # Optional (default: false) client_certs: [] sasl: - mechanism: SCRAM-SHA-256 username: ${secrets.KAFKA_USER_CONNECT} password: ${secrets.KAFKA_PASSWORD_CONNECT} topic: dad-jokes # Optional ``` > 📝 **NOTE** > > - Notice the `${REDPANDA_BROKERS}` [contextual variable](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/contextual-variables/) in the configuration. This references your cluster’s bootstrap server address, so you can use it in any pipeline without hardcoding connection details. Use the slash command menu in the YAML editor or use the command palette to insert the Redpanda broker’s contextual variable. > > - Notice `${secrets.KAFKA_USER_CONNECT}` and `${secrets.KAFKA_PASSWORD_CONNECT}`. These reference secrets that you can create using the slash command menu in the YAML editor or on the **Security** page. > > - The Brave browser does not fully support code snippets. 7. Click **Save**. Your pipeline details display, and after a few seconds, the pipeline starts running. The pipeline generates jokes and writes the jokes to your Redpanda topic. ### [](#review-the-pipeline-logs)Review the pipeline logs The page loads new log messages as they come in. When Live mode is disabled, you can filter logs, for example, by level, message content, or path. The log shows activity from the past five hours. Click through the log messages to see the startup sequence. For example, you’ll see when the output becomes active: ```json { "instance_id": "d73c39bp7l8c73d7lll0", "label": "", "level": "INFO", "message": "Output type redpanda is now active", "path": "root.output", "pipeline_id": "d73a55ptub9s73agpthg", "time": "2026-03-27T17:43:02.36416142Z" } ``` ### [](#view-the-processed-messages)View the processed messages 1. Go to the **Topics** page for your cluster and select the `dad-jokes` topic. 2. Click any message to see the structure. For example: ```json { "id": "d242c355-4cee-4382-817a-190c7a115a19", "joke": "I used to play piano by ear, but now I use my hands.", "joke_length": 52, "source": "dad-joke-generator", "timestamp": "2026-03-27T15:30:38.963227997Z" } ``` ## [](#build-a-consumer-pipeline)Build a consumer pipeline This next pipeline rates the jokes that you generated in the first pipeline. To create the consumer pipeline: 1. Go back to the **Connect** page for your cluster, and click **Create a pipeline**. 2. Enter this name for the pipeline: `joke-generator-consumer`. 3. In the left sidebar, click **Add input +**, and search for and select the `redpanda` input connector. 4. The `redpanda` connector requires a Redpanda topic and user: 1. In the `redpanda` input connector, click **Topic +** and select the existing topic `dad-jokes`. Click **Add**. 2. In the `redpanda` input connector, click **User +** and select the existing user `connect`. For consumer group, enter `dad-joke-raters`. This allows the user `connect` to be granted READ and DESCRIBE permissions for the `dad-joke-raters` consumer group. Click **Add**. 5. Click **Add output +**, and search for and select the `drop` output connector. (For testing purposes, this output drops messages instead of forwarding them. In a real scenario you would replace the `drop` connector with your real destination.) 6. Replace the generated YAML in the editor with the following configuration, which includes the `bloblang`, `log`, and `catch` processors. > 📝 **NOTE** > > This example explicitly includes several optional configuration fields for the `redpanda` input. They’re shown here for demonstration purposes, so you can see a range of available settings. ```yaml input: redpanda: seed_brokers: # Optional - ${REDPANDA_BROKERS} client_id: benthos # Optional (default: "benthos") tls: enabled: true # Optional (default: false) client_certs: [] sasl: - mechanism: SCRAM-SHA-256 username: ${secrets.KAFKA_USER_CONNECT} password: ${secrets.KAFKA_PASSWORD_CONNECT} metadata_max_age: 5m # Optional (default: "5m") request_timeout_overhead: 10s # Optional (default: "10s") conn_idle_timeout: 20s # Optional (default: "20s") topics: # Required (mutually exclusive with regexp_topics) - dad-jokes regexp_topics: false # Optional (default: false). Mutually exclusive with topics. rebalance_timeout: 45s # Optional (default: "45s") session_timeout: 1m # Optional (default: "1m") heartbeat_interval: 3s # Optional (default: "3s") start_from_oldest: true # Optional (default: true) start_offset: earliest # Optional (default: "earliest") fetch_max_bytes: 50MiB # Optional (default: "50MiB") fetch_max_wait: 5s # Optional (default: "5s") fetch_min_bytes: 1B # Optional (default: "1B") fetch_max_partition_bytes: 1MiB # Optional (default: "1MiB") transaction_isolation_level: read_uncommitted # Optional (default: "read_uncommitted") consumer_group: dad-joke-raters # Optional commit_period: 5s # Optional (default: "5s") partition_buffer_bytes: 1MB # Optional (default: "1MB") topic_lag_refresh_period: 5s # Optional (default: "5s") max_yield_batch_bytes: 32KB # Optional (default: "32KB") auto_replay_nacks: true # Optional (default: true) pipeline: processors: - bloblang: | root = this let rating = random_int(min: 1, max: 11) root.cringe_rating = $rating root.cringe_level = if $rating <= 3 { "Mild - Almost acceptable" } else if $rating <= 6 { "Medium - Classic dad joke territory" } else if $rating <= 8 { "High - Eye-roll inducing" } else { "EXTREME - Peak dad joke achievement" } root.processed_at = now() root.rating_emoji = match { $rating <= 3 => "😐", $rating <= 6 => "😬", $rating <= 8 => "🤦", _ => "💀" } let age_seconds = (timestamp_unix() - this.timestamp.ts_parse("2006-01-02T15:04:05Z07:00").ts_unix()) root.age_seconds = $age_seconds - log: level: INFO message: | 🎭 JOKE RATED! ${! json("rating_emoji") } Joke: "${! json("joke") }" Cringe Rating: ${! json("cringe_rating") }/10 - ${! json("cringe_level") } Age: ${! json("age_seconds") } seconds old Processed at: ${! json("processed_at") } - catch: - log: level: ERROR message: "❌ Failed to process joke: ${! error() }" output: drop: {} ``` 7. Click **Save** to start your pipeline. 8. Your pipeline details display, and after a few seconds, the pipeline starts running. Check the logs to see a rated joke. For example: ```json { "custom_source": "true", "instance_id": "d454dkn4u2is73ava480", "label": "", "level": "INFO", "message": "🎭 JOKE RATED! 💀\nJoke: \"I used to play piano by ear, but now I use my hands.\"\nCringe Rating: 9/10 - EXTREME - Peak dad joke achievement\nAge: 659 seconds old\nProcessed at: 2026-03-27T17:54:13.340229297Z\n", "path": "root.pipeline.processors.1", "pipeline_id": "d454djahlips73dmcll0", "time": "2026-03-27T17:54:13.341137527Z" } ``` ## [](#enable-egress-for-pipelines-on-byoc-and-dedicated-clusters)Enable egress for pipelines on BYOC and Dedicated clusters The consumer pipeline in this quickstart drops messages instead of writing them to a real destination. When you replace the `drop` output with a connector that writes to your own systems, pipelines on BYOC and Dedicated clusters may need additional egress access. On BYOC and Dedicated clusters, pipelines run inside the Redpanda data plane VPC. By default, pipelines can connect to your Redpanda cluster and to publicly routable endpoints. Outbound connections to other destinations, such as a database with a private address in a peered VPC, are blocked. To allow pipelines to reach these destinations, add an egress allowlist to your cluster using the [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) (version `>= 2.1.1`). The `redpanda_connect.allowed_destination_cidr_ports` attribute on the `redpanda_cluster` resource accepts up to 16 rules, each allowing egress to a destination CIDR block on a port or port range: ```hcl resource "redpanda_cluster" "example" { # ... other cluster arguments ... redpanda_connect = { allowed_destination_cidr_ports = [ { cidr = "10.62.0.0/16" # CIDR of the VPC that hosts your database port_start = 5432 # PostgreSQL } ] } } ``` The allowlist permits outbound traffic from pipelines, but it does not create a network path to the destination. For private destinations, you must also establish routing, for example by peering the Redpanda data plane VPC with the VPC that hosts your database. To set the allowlist with the Cloud API, and for the full rule reference and troubleshooting guidance, see [Configure Egress for Redpanda Connect Pipelines](https://docs.redpanda.com/cloud-data-platform/networking/connect-egress-allowlist/). For a pipeline example that writes to a private PostgreSQL database, see [Enable egress to custom destinations](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#enable-egress-to-custom-destinations). ## [](#clean-up)Clean up When you’ve finished experimenting with your data pipeline, you can delete the pipelines and the topic you created for this quickstart. 1. On the **Connect** page, click the **…​** icon next to the `joke-generator-producer` pipeline and select **Delete**. Repeat for the `joke-generator-consumer` pipeline. 2. Confirm your deletion to remove the pipelines and associated logs. 3. On the **Topics** page, delete the `dad-jokes` topic. ## [](#next-steps)Next steps - Try one of the [Redpanda Connect cookbooks](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/). - Choose [connectors for your use case](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/). - [Add secrets to your pipeline](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/). - [Monitor a data pipeline on a BYOC or Dedicated cluster](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/monitor-connect/). - [Manually scale resources for a pipeline](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/resource-management/). - [Configure, test, and run a data pipeline locally](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/). --- # Page 493: Cookbooks **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks.md --- # Cookbooks > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Cookbooks latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/index.adoc description: Step-by-step Redpanda Connect cookbooks that walk through complete, real-world pipeline builds in Redpanda Cloud. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- - [DynamoDB CDC Patterns](dynamodb_cdc/) Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. - [Enrichment Workflows](enrichments/) How to configure Redpanda Connect to process a workflow of enrichment services. - [Filtering and Sampling](filtering/) Configure Redpanda Connect to conditionally drop messages. - [Ingest data into Snowflake](snowflake_ingestion/) Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. - [Joining Streams](joining_streams/) How to hydrate documents by joining multiple streams. - [Redpanda Migrator](redpanda_migrator/) Move your workloads from any Kafka system to Redpanda Cloud using a single command. - [Retrieval-Augmented Generation (RAG)](rag/) How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. - [Work with Jira Issues](jira/) Learn how to query, filter, and create Jira issues using Redpanda Connect pipelines. --- # Page 494: DynamoDB CDC Patterns **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/dynamodb_cdc.md --- # DynamoDB CDC Patterns > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: DynamoDB CDC Patterns latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/dynamodb_cdc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/dynamodb_cdc.adoc description: Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Apply reusable patterns for capturing DynamoDB CDC events learning-objective-2: Adapt integration patterns to route CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-03-04" page-git-modified-date: "2026-08-11" --- The DynamoDB CDC input enables capturing item-level changes from DynamoDB tables with streams enabled. This cookbook provides reusable patterns for filtering, transforming, and routing DynamoDB CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Apply reusable patterns for capturing DynamoDB CDC events - Adapt integration patterns to route CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, configure the following. ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) for installation instructions. ### [](#dynamodb-streams)DynamoDB Streams The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled with an appropriate view type: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images (recommended for detecting changes) To enable streams on an existing table using the AWS CLI: ```bash aws dynamodb update-table \ --table-name orders \ --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES ``` ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for AWS configuration. Environment variables keep credentials out of your pipeline configuration files. ```bash export DYNAMODB_TABLE=orders (1) export AWS_REGION=us-east-1 (2) export REDPANDA_BROKERS=localhost:9092 (3) export S3_BUCKET=cdc-archive (4) ``` | 1 | The name of the DynamoDB table with streams enabled. | | --- | --- | | 2 | The AWS region where your DynamoDB table is located. | | 3 | The Redpanda broker addresses (for Redpanda output examples). | | 4 | The S3 bucket name (for S3 output examples). | Redpanda Connect loads AWS credentials from the standard [credential chain](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (environment variables, `~/.aws/credentials`, or IAM roles). ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from a DynamoDB table and outputs them with metadata: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Extract the change event details - mapping: | root.event_type = this.eventName root.table = this.tableName root.event_id = this.eventID root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage root.sequence_number = this.dynamodb.sequenceNumber root.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available fields for Bloblang mappings, see the [message structure](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_dynamodb_cdc/#_message_structure) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events Filter events to process only specific change types: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} start_from: latest pipeline: processors: # Filter to only process INSERT and MODIFY events (ignore REMOVE) - mapping: | root = if this.eventName == "REMOVE" { deleted() } else { this } # Transform to a simplified format - mapping: | root.event_type = this.eventName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage output: stdout: codec: lines ``` This example: - Filters out `REMOVE` events using `deleted()` - Transforms the event to a simplified format ## [](#route-to-redpanda)Route to Redpanda Stream DynamoDB changes to Redpanda for real-time processing: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints batch_size: 100 poll_interval: 500ms pipeline: processors: # Transform to a Kafka-friendly format with a composite key - mapping: | let keys = this.dynamodb.keys meta kafka_key = [$keys.pk, $keys.sk].filter(v -> v != null).join("#") root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-cdc-events key: ${! @kafka_key } partitioner: murmur2_hash compression: snappy batching: count: 100 period: 1s ``` This example: - Creates a composite message key from the DynamoDB primary key - Transforms the DynamoDB format to plain JSON - Batches messages for efficient delivery ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Add partitioning metadata for S3 organization - mapping: | let event_time = now() meta s3_path = "year=%s/month=%s/day=%s/hour=%s".format( $event_time.ts_format("2006"), $event_time.ts_format("01"), $event_time.ts_format("02"), $event_time.ts_format("15") ) root.event_type = this.eventName root.table = this.tableName root.sequence_number = this.dynamodb.sequenceNumber root.event_time = $event_time root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: aws_s3: bucket: ${S3_BUCKET} path: dynamodb-cdc/${DYNAMODB_TABLE}/${! @s3_path }/${! uuid_v4() }.json region: ${AWS_REGION} object_canned_acl: private batching: count: 1000 period: 1m processors: - archive: format: lines ``` This example: - Organizes files by time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Transform to a common format - mapping: | root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.data = if this.dynamodb.exists("newImage") { this.dynamodb.newImage } else { this.dynamodb.oldImage } output: switch: cases: # Route INSERT events to a topic for new records - check: this.event_type == "INSERT" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-inserts # Route MODIFY events to a topic for updates - check: this.event_type == "MODIFY" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-updates # Route REMOVE events to a topic for deletes - check: this.event_type == "REMOVE" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-deletes # Fallback for any unexpected event types - output: drop: {} ``` This pattern: - Separates processing pipelines for inserts, updates, and deletes - Applies different retention policies per event type - Supports specialized downstream consumers ## [](#detect-changed-fields)Detect changed fields Compare old and new images to identify which fields changed: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Only process MODIFY events - mapping: | root = if this.eventName != "MODIFY" { deleted() } else { this } # Compare old and new images to find changed fields - mapping: | let old_data = this.dynamodb.oldImage let new_data = this.dynamodb.newImage root.table = this.tableName root.keys = this.dynamodb.keys root.timestamp = now() # Find fields that changed by comparing key-value pairs root.changes = $new_data.key_values().filter(kv -> !$old_data.exists(kv.key) || $old_data.get(kv.key) != kv.value).map_each(kv -> {"field": kv.key, "old_value": if $old_data.exists(kv.key) { $old_data.get(kv.key) } else { null }, "new_value": kv.value}) # Find fields that were removed root.removed_fields = $old_data.keys().filter(k -> !$new_data.exists(k)) output: stdout: codec: lines ``` This pattern: - Filters to only MODIFY events - Compares old and new images to find differences - Outputs a list of changed fields with their old and new values > 📝 **NOTE** > > This pattern requires the `NEW_AND_OLD_IMAGES` stream view type. The `.key_values()` method converts an object to an array of key-value pairs that can be filtered and mapped. ## [](#checkpointing)Checkpointing The DynamoDB CDC input automatically manages checkpoints in a separate DynamoDB table: ```yaml input: aws_dynamodb_cdc: tables: - orders checkpoint_table: cdc-checkpoints (1) checkpoint_limit: 500 (2) start_from: trim_horizon (3) ``` | 1 | Custom checkpoint table name (default: redpanda_dynamodb_checkpoints). | | --- | --- | | 2 | Checkpoint after every 500 messages (lower = better recovery, higher = fewer writes). | | 3 | Start from the oldest available record when no checkpoint exists. | If a checkpoint table doesn’t exist, it’s created automatically with the required schema. ## [](#performance-tuning)Performance tuning Optimize throughput and latency with these settings: ```yaml input: aws_dynamodb_cdc: tables: - orders batch_size: 1000 (1) poll_interval: 100ms (2) max_tracked_shards: 10000 (3) throttle_backoff: 50ms (4) ``` | 1 | Maximum records per shard per request (1-1000). | | --- | --- | | 2 | Time between polls when no records are available. | | 3 | Maximum shards to track (for very large tables). | | 4 | Backpressure delay when too many messages are in-flight. | ### [](#throughput-considerations)Throughput considerations - DynamoDB Streams allows 5 `GetRecords` calls per second per shard - Higher `batch_size` improves throughput but increases memory usage - Shorter `poll_interval` reduces latency but increases API calls ## [](#troubleshoot-common-issues)Troubleshoot common issues ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify streams are enabled on the table: ```bash aws dynamodb describe-table --table-name orders \ --query 'Table.StreamSpecification' ``` 2. Check that changes are being made to the table 3. Verify `start_from` is set to `trim_horizon` to capture existing stream data ### [](#duplicate-events)Duplicate events Each stream record appears exactly once in DynamoDB Streams. However, if your pipeline fails before checkpointing, records may be re-read on restart, resulting in at-least-once processing semantics. To handle potential duplicates: - Use idempotent processing in downstream systems - Deduplicate using the `dynamodb_sequence_number` metadata - Lower `checkpoint_limit` to reduce the window of possible duplicates ### [](#stream-retention)Stream retention DynamoDB Streams retains data for 24 hours. If your pipeline is offline longer than that: - Consider using [Kinesis Data Streams for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html) with the [`aws_kinesis` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_kinesis/) instead (up to 1 year retention) - Implement a full-table scan fallback for disaster recovery ## [](#next-steps)Next steps - [DynamoDB CDC Input Reference](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_dynamodb_cdc/) - [AWS Configuration Guide](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws/) - [Kinesis Input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_kinesis/) (for Kinesis Data Streams for DynamoDB) - [DynamoDB Streams Documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) --- # Page 495: Enrichment Workflows **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/enrichments.md --- # Enrichment Workflows > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Enrichment Workflows latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/enrichments page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/enrichments.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/enrichments.adoc description: How to configure Redpanda Connect to process a workflow of enrichment services. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- This cookbook demonstrates how to enrich a stream of JSON documents with HTTP services. This method also works with [AWS Lambda functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/aws_lambda/). We will start off by configuring a single enrichment, then we will move onto a workflow of enrichments with a network of dependencies using the [`workflow` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/workflow/). Each enrichment will be performed in parallel across a [pre-batched](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/batching/) stream of documents. Workflow enrichments that do not depend on each other will also be performed in parallel, making this orchestration method very efficient. The imaginary problem we are going to solve is applying a set of NLP based enrichments to a feed of articles in order to detect fake news. We will be consuming and writing to Kafka, but the example works with any [input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/) and [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } } ``` ## [](#meet-the-enrichments)Meet the enrichments ### [](#claims-detector)Claims detector To start us off we will configure a single enrichment, which is an imaginary 'claims detector' service. This is an HTTP service that wraps a trained machine learning model to extract claims that are made within a body of text. The service expects a `POST` request with JSON payload of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ``` And returns a JSON payload of the form: ```json { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` Since each request only applies to a single document we will make this enrichment scale by deploying multiple HTTP services and hitting those instances in parallel across our document batches. In order to send a mapped request and map the response back into the original document we will use the [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/), with a child `http` processor. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` With this pipeline our documents will come out looking something like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." }, "tmp": { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` ### [](#hyperbole-detector)Hyperbole detector Next up is a 'hyperbole detector' that takes a `POST` request containing the article contents and returns a hyperbole score between 0 and 1. This time the format is array-based and therefore supports calculating multiple documents in a single request, making better use of the host machines GPU. A request should take the following form: ```json [ { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ] ``` And the response looks like this: ```json [ { "hyperbole_rank": 0.73 } ] ``` In order to create a single request from a batch of documents, and subsequently map the result back into our batch, we will use the [`archive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/archive/) and [`unarchive`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/unarchive/) processors in our [`branch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) flow, like this: ```yaml pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' ``` The purpose of the `json_array` format `archive` processor is to take a batch of JSON documents and place them into a single document as an array. Subsequently, we then send one single request for each batch. After the request is made we do the opposite with the `unarchive` processor in order to convert it back into a batch of the original size. ### [](#fake-news-detector)Fake news detector Finally, we are going to use a 'fake news detector' that takes the article contents as well as the output of the previous two enrichments and calculates a fake news rank between 0 and 1. This service behaves similarly to the claims detector service and takes a document of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking.", "hyperbole_rank": 0.73, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` And returns an object of the form: ```json { "fake_news_rank": 0.893 } ``` We then wish to map the field `fake_news_rank` from that result into the original document at the path `article.fake_news_score`. Our [`branch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) block for this enrichment would look like this: ```yaml pipeline: processors: - branch: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' ``` Note that in our `request_map` we are targeting fields that are populated from the previous two enrichments. If we were to execute all three enrichments in a sequence we’ll end up with a document looking like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "fake_news_score": 0.76 }, "tmp": { "hyperbole_rank": 0.34, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` Great! However, as a streaming pipeline this set up isn’t ideal as our first two enrichments are independent and could potentially be executed in parallel in order to reduce processing latency. ## [](#combining-into-a-workflow)Combining into a workflow If we configure our enrichments within a [`workflow` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/workflow/) we can use Redpanda Connect to automatically detect our dependency graph, giving us two key benefits: 1. Enrichments at the same level of a dependency graph (claims and hyperbole) will be executed in parallel. 2. When introducing more enrichments to our pipeline the added complexity of resolving the dependency graph is handled automatically by Redpanda Connect. Placing our branches within a [`workflow` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/workflow/) makes our final pipeline configuration look like this: ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - workflow: meta_path: '' # Don't bother storing branch metadata. branches: claims: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' hyperbole: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' fake_news: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' - catch: - log: fields_mapping: 'root.content = content().string()' message: "Enrichments failed due to: ${!error()}" - mapping: | root = this root.tmp = deleted() output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` Since the contents of `tmp` won’t be required downstream we remove it after our enrichments using a [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/). A [`catch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/catch/) processor was added at the end of the pipeline which catches documents that failed enrichment. You can replace the log event with a wide range of recovery actions such as sending to a dead-letter/retry queue, dropping the message entirely, etc. You can read more about error handling [in this article](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/). --- # Page 496: Filtering and Sampling **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/filtering.md --- # Filtering and Sampling > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Filtering and Sampling latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/filtering page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/filtering.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/filtering.adoc description: Configure Redpanda Connect to conditionally drop messages. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Filtering events in Redpanda Connect is both easy and flexible, this cookbook demonstrates a few different types of filtering you can do. All of these examples make use of the [`mapping` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) but shouldn’t require any prior knowledge. ## [](#the-basic-filter)The basic filter Dropping events with [Bloblang](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/) is done by mapping the function `deleted()` to the `root` of the mapped document. To remove all events indiscriminately you can simply do: ```yaml pipeline: processors: - mapping: root = deleted() ``` But that’s most likely not what you want. We can instead only delete an event under certain conditions with a [`match`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#pattern-matching) or [`if`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#conditional-mapping) expression: ```yaml pipeline: processors: - mapping: | root = if @topic.or("") == "foo" || this.doc.type == "bar" || this.doc.urls.contains("https://www.benthos.dev/").catch(false) { deleted() } ``` The above config removes any events where: - The metadata field `topic` is equal to `foo` - The event field `doc.type` (a string) is equal to `bar` - The event field `doc.urls` (an array) contains the string `https://www.benthos.dev/` Events that do not match any of these conditions will remain unchanged. ## [](#sample-events)Sample events Another type of filter we might want is a sampling filter, we can do that with a random number generator: ```yaml pipeline: processors: - mapping: | # Drop 50% of documents randomly root = if random_int() % 2 == 0 { deleted() } ``` We can also do this in a deterministic way by hashing events and filtering by that hash value: ```yaml pipeline: processors: - mapping: | # Drop ~10% of documents deterministically (same docs filtered each run) root = if content().hash("xxhash64").slice(-8).number() % 10 == 0 { deleted() } ``` --- # Page 497: Work with Jira Issues **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/jira.md --- # Work with Jira Issues > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Work with Jira Issues latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/jira page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/jira.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/jira.adoc description: Learn how to query, filter, and create Jira issues using Redpanda Connect pipelines. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Query Jira issues using JQL patterns with the Jira processor learning-objective-2: Combine generate input with Jira processor for scheduled queries learning-objective-3: Create Jira issues using the HTTP processor and REST API page-git-created-date: "2026-02-18" page-git-modified-date: "2026-02-18" --- The Jira processor enables querying Jira issues using JQL (Jira Query Language) and returning structured data. It’s a processor, so you can use it in pipelines for input-style flows (pair with `generate`) or output-style flows (pair with `drop`). Use this cookbook to: - Query Jira issues on a schedule or on-demand - Filter issues using JQL patterns - Create Jira issues using the HTTP processor ## [](#prerequisites)Prerequisites The examples in this cookbook use the Secrets Store for Jira credentials. This keeps sensitive credentials secure and separate from your pipeline configuration. 1. [Generate a Jira API token](https://id.atlassian.com/manage-profile/security/api-tokens). 2. Add your Jira credentials to the [Secrets Store](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/): - `JIRA_BASE_URL`: Your Jira instance URL (for example, `https://your-domain.atlassian.net`) - `JIRA_USERNAME`: Your Jira account email address - `JIRA_API_TOKEN`: The API token generated from your Atlassian account - `JIRA_AUTH_TOKEN` (optional, for creating issues): Base64-encoded `username:api_token` string ## [](#use-jira-as-an-input)Use Jira as an input To use Jira as an input, combine the `generate` input with the Jira processor. This pattern triggers Jira queries at regular intervals or on-demand. > 💡 **TIP** > > Replace `MYPROJECT` in the examples with your actual Jira project key. ### [](#query-jira-periodically)Query Jira periodically This example queries Jira every 30 seconds for recent issues: ```yaml input: generate: interval: 30s mapping: | root.jql = "project = MYPROJECT AND updated >= -1h ORDER BY updated DESC" root.maxResults = 50 root.fields = ["key", "summary", "status", "assignee", "priority"] pipeline: processors: - jira: base_url: "${secrets.JIRA_BASE_URL}" username: "${secrets.JIRA_USERNAME}" api_token: "${secrets.JIRA_API_TOKEN}" output: stdout: {} ``` ### [](#one-time-query)One-time query For a single query, use `count` instead of `interval`: ```yaml input: generate: count: 1 mapping: | root.jql = "project = MYPROJECT AND status = Open" root.maxResults = 100 pipeline: processors: - jira: base_url: "${secrets.JIRA_BASE_URL}" username: "${secrets.JIRA_USERNAME}" api_token: "${secrets.JIRA_API_TOKEN}" output: stdout: {} ``` ## [](#input-message-format)Input message format The Jira processor expects input messages containing valid Jira queries in JSON format: ```json { "jql": "project = MYPROJECT AND status = Open", "maxResults": 50, "fields": ["key", "summary", "status", "assignee"] } ``` ### [](#required-fields)Required fields - `jql`: The JQL (Jira Query Language) query string ### [](#optional-fields)Optional fields - `maxResults`: Maximum number of results to return (default: 50) - `fields`: Array of field names to include in the response ## [](#jql-query-patterns)JQL query patterns Here are common JQL patterns for filtering issues: ### [](#recent-issues-by-project)Recent issues by project ```jql project = AND created >= -7d ORDER BY created DESC ``` ### [](#issues-assigned-to-current-user)Issues assigned to current user ```jql assignee = currentUser() AND status != Done ``` ### [](#issues-by-status)Issues by status ```jql project = AND status IN (Open, 'In Progress', 'To Do') ``` ### [](#issues-by-priority)Issues by priority ```jql project = AND priority = High ORDER BY created DESC ``` ## [](#output-message-format)Output message format The Jira processor returns individual issue messages, rather than a response object with an `issues` array. Each message output by the Jira processor represents a single issue: ```json { "id": "12345", "key": "DOC-123", "fields": { "summary": "Example issue", "status": { "name": "In Progress" }, "assignee": { "displayName": "John Doe" } } } ``` The Jira processor automatically handles pagination internally. The processor: 1. Makes the initial request with `startAt=0`. 2. Checks if more results are available. 3. Automatically fetches subsequent pages until all results are retrieved. 4. Outputs each issue as an individual message. You don’t need to handle pagination manually. ## [](#create-and-update-jira-issues)Create and update Jira issues The Jira processor is read-only and only supports querying. To create or update Jira issues, use the [`http` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/http/) with the Jira REST API. ### [](#create-a-jira-issue)Create a Jira issue ```yaml input: generate: count: 1 mapping: | root.fields = { "project": {"key": "MYPROJECT"}, "summary": "Issue created from Redpanda Connect", "description": { "type": "doc", "version": 1, "content": [{"type": "paragraph", "content": [{"type": "text", "text": "Created via API"}]}] }, "issuetype": {"name": "Task"} } pipeline: processors: - http: url: "${secrets.JIRA_BASE_URL}/rest/api/3/issue" verb: POST headers: Content-Type: application/json Authorization: "Basic ${secrets.JIRA_AUTH_TOKEN}" output: stdout: {} ``` ## [](#see-also)See also - [Jira processor reference](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/jira/) - [Jira REST API documentation](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/) - [JQL query guide](https://www.atlassian.com/software/jira/guides/jql) --- # Page 498: Joining Streams **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/joining_streams.md --- # Joining Streams > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Joining Streams latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/joining_streams page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/joining_streams.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/joining_streams.adoc description: How to hydrate documents by joining multiple streams. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- This cookbook demonstrates how to merge JSON events from parallel streams using content based rules and a [cache](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/) of your choice. The imaginary problem we are going to solve is hydrating a feed of article comments with information from their parent articles. We will be consuming and writing to Kafka, but the example works with any [input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/about/) and [output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user1" } } ``` Comments can either be posted on an article or a parent comment, are received over the topic `comments`, and look like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "user": { "id": "user2" } } ``` Our goal is to end up with a single stream of comments, where information about the root article of the comment is attached to the event. The above comment should exit our pipeline looking like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "article": { "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user2" } } ``` In order to achieve this we will need to cache articles as they pass through our pipelines and then retrieve them for each comment passing through. Since the parent of a comment might be another comment we will also need to cache and retrieve comments in the same way. ## [](#caching-articles)Caching articles Our first pipeline is very simple, we just consume articles, reduce them to only the fields we wish to cache, and then cache them. If we receive the same article multiple times we’re going to assume it’s okay to overwrite the old article in the cache. In this example I’m targeting Redis, but you can choose any of the supported [cache targets](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/about/). The TTL of cached articles is set to one week. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group pipeline: processors: # Reduce document into only fields we wish to cache. - mapping: 'article = article' # Store reduced articles into our cache. - cache: operator: set resource: hydration_cache key: '${!json("article.id")}' value: '${!content()}' # Drop all articles after they are cached. output: drop: {} cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` ## [](#hydrating-comments)Hydrating comments Our second pipeline consumes comments, caches them in case a subsequent comment references them, obtains its parent (article or comment), and attaches the root article to the event before sending it to our output topic `comments_hydrated`. In this config we make use of the [`branch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) processor as it allows us to reduce documents into smaller maps for caching and gives us greater control over how results are mapped back into the document. ```yaml input: kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group pipeline: processors: # Perform both hydration and caching within a for_each block as this ensures # that a given message of a batch is cached before the next message is # hydrated, ensuring that when a message of the batch has a parent within # the same batch hydration can still work. - for_each: # Attempt to obtain parent event from cache (if the ID exists). - branch: request_map: 'root = this.comment.parent_id | deleted()' processors: - cache: operator: get resource: hydration_cache key: '${!content()}' # And if successful copy it into the field `article`. result_map: 'root.article = this.article' # Reduce comment into only fields we wish to cache. - branch: request_map: | root.comment.id = this.comment.id root.article = this.article processors: # Store reduced comment into our cache. - cache: operator: set resource: hydration_cache key: '${!json("comment.id")}' value: '${!content()}' # No `result_map` since we don't need to map into the original message. # Send resulting documents to our hydrated topic. output: kafka: addresses: [ TODO ] topic: comments_hydrated cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` This pipeline satisfies our basic needs but errors aren’t handled at all, meaning intermittent cache connectivity problems that span beyond our cache retries will result in failed documents entering our `comments_hydrated` topic. This is also the case if a comment arrives in our pipeline before its parent. There are [many patterns for error handling](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/) to choose from in Redpanda Connect. In this example we’re going to introduce a delayed retry queue as it enables us to reprocess failed documents after a grace period, which is isolated from our main pipeline. ## [](#adding-a-retry-queue)Adding a retry queue Our retry queue is going to be another topic called `comments_retried`. Since most errors are related to time we will delay retry attempts by storing the current timestamp after a failed request as a metadata field. We will use an input [`broker`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/broker/) so that we can consume both the `comments` and `comments_retry` topics in the same pipeline. Our config (omitting the caching sections for brevity) now looks like this: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group - kafka: addresses: [ TODO ] topics: [ comments_retry ] consumer_group: benthos_comments_group processors: - for_each: # Calculate time until next retry attempt and sleep for that duration. # This sleep blocks the topic 'comments_retry' but NOT 'comments', # because both topics are consumed independently and these processors # only apply to the 'comments_retry' input. - sleep: duration: '${! 3600 - ( timestamp_unix() - meta("last_attempted").number() ) }s' pipeline: processors: - try: - for_each: # Attempt to obtain parent event from cache. - branch: {} # Omitted # Reduce document into only fields we wish to cache. - branch: {} # Omitted # If we've reached this point then both processors succeeded. - mapping: 'meta output_topic = "comments_hydrated"' - catch: # If we reach here then a processing stage failed. - mapping: | meta output_topic = "comments_retry" meta last_attempted = timestamp_unix() # Send resulting documents either to our hydrated topic or the retry topic. output: kafka: addresses: [ TODO ] topic: '${!meta("output_topic")}' cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` You can find a full example [in the project repo](https://github.com/redpanda-data/connect/blob/master/config/examples/joining_streams.yaml), and with this config we can deploy as many instances of Redpanda Connect as we need as the partitions will be balanced across the consumers. --- # Page 499: Retrieval-Augmented Generation (RAG) **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/rag.md --- # Retrieval-Augmented Generation (RAG) > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Retrieval-Augmented Generation (RAG) latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/rag page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/rag.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/rag.adoc description: How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. page-git-created-date: "2024-09-12" page-git-modified-date: "2026-05-26" --- This cookbook shows you how to create a vector embeddings indexing pipeline for Retrieval-Augmented Generation (RAG), using PostgreSQL and [PGVector](https://github.com/pgvector/pgvector). Follow the cookbook to: - Take textual data from a Redpanda topic and compute vector embeddings for it using [Ollama](https://ollama.ai) - Write the pipeline output into a PostgreSQL table with a [PGVector](https://github.com/pgvector/pgvector) index on the embeddings column. ## [](#compute-the-embeddings)Compute the embeddings Start by creating a Redpanda topic, which you can use as an input for an indexing data pipeline. ```bash rpk topic create articles echo '{ "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } }' | rpk topic produce articles -f '%v' ``` Your indexing pipeline can read from the Redpanda topic, using the [`kafka`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka/) input: ```yaml input: kafka: addresses: [ "TODO" ] topics: [ articles ] consumer_group: rp_connect_articles_group tls: enabled: true sasl: mechanism: SCRAM-SHA-256 user: "TODO" password: "TODO" ``` Use [Nomic Embed](https://ollama.com/library/nomic-embed-text) to compute embeddings. Since each request only applies to a single document, you can scale this by making requests in parallel across document batches. To send a mapped request and map the response back into the original document, use the [`branch` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/branch/) with a child [`ollama_embeddings`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/ollama_embeddings/) processor. ```yaml pipeline: threads: -1 processors: - branch: request_map: 'root = "search_document: %s\n%s".format(this.article.title, this.article.content)' processors: - ollama_embeddings: model: nomic-embed-text result_map: 'root.article.embeddings = this' ``` With this pipeline, your processed documents should look something like this: ```yaml { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "embeddings": [0.754, 0.19283, 0.231, 0.834], # This vector will actually have 768 dimensions } } ``` Now, try sending this transformed data to PostgreSQL using the [`sql_insert`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sql_insert/) output. You can take advantage of the `init_statement` functionality to set up `pgvector` and a table to write the data to. ```yaml output: sql_insert: driver: postgres dsn: "TODO" init_statement: | CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS searchable_text ( id varchar(128) PRIMARY KEY, title text NOT NULL, body text NOT NULL, embeddings vector(768) NOT NULL ); CREATE INDEX IF NOT EXISTS text_hnsw_index ON searchable_text USING hnsw (embeddings vector_l2_ops); table: searchable_text columns: ["id", "title", "body", "embeddings"] args_mapping: "[this.article.id, this.article.title, this.article.content, this.article.embeddings.vector()]" ``` After deploying this pipeline using the Redpanda Console, you can verify data is being written into PostgreSQL using `psql` to execute `SELECT count(*) FROM searchable_text;`. --- # Page 500: Redpanda Migrator **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator.md --- # Redpanda Migrator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Migrator latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/redpanda_migrator page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/redpanda_migrator.adoc description: Move your workloads from any Kafka system to Redpanda Cloud using a single command. page-git-created-date: "2024-10-02" page-git-modified-date: "2026-05-26" --- With Redpanda Migrator, you can move your workloads from any Apache Kafka system to Redpanda using a single command. It lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. Redpanda Migrator is Redpanda’s alternative to Kafka MirrorMaker 2 for continuous replication into Redpanda, including timestamp-based consumer group offset translation. Redpanda Connect’s Redpanda Migrator uses the unified migrator components (available in Redpanda Connect 4.67.5+): - [`redpanda_migrator` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator/) connects to the source Kafka cluster and Schema Registry. - [`redpanda_migrator` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) handles all migration logic including topic creation, schema synchronization, and consumer group offset translation. > 📝 **NOTE** > > If you’re currently using the legacy `redpanda_migrator_bundle` components, see [Migrate to the Unified Redpanda Migrator](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/migrate-unified-redpanda-migrator/) for migration instructions. ## [](#create-a-kafka-cluster-and-a-redpanda-cloud-cluster)Create a Kafka cluster and a Redpanda Cloud cluster First, you need to provision two clusters, a Kafka one called `source` and a Redpanda Cloud one called `destination`. This cookbook uses the following sample connection details throughout the rest of this cookbook: Source broker: source.cloud.kafka.com:9092 schema registry: https://schema-registry-source.cloud.kafka.com:30081 username: kafka password: testpass Destination broker: destination.cloud.redpanda.com:9092 schema registry: https://schema-registry-destination.cloud.redpanda.com:30081 username: redpanda password: testpass Then you create two topics in the `source` Kafka cluster, `foo` and `bar`, and an ACL for each topic: ```bash cat > ./config.properties < 📝 **NOTE** > > The Brave browser does not fully support code snippets. `generate_data.yaml` ```yaml http: enabled: false input: sequence: inputs: - generate: mapping: | let msg = counter() root.data = $msg meta kafka_topic = match $msg % 2 { 0 => "foo" 1 => "bar" } interval: 1s count: 0 batch_size: 1 processors: - schema_registry_encode: url: "https://schema-registry-source.cloud.kafka.com:30081" subject: ${! metadata("kafka_topic") } avro_raw_json: true basic_auth: enabled: true username: kafka password: testpass output: kafka_franz: seed_brokers: [ "source.cloud.kafka.com:9092" ] topic: ${! @kafka_topic } partitioner: manual partition: ${! random_int(min:0, max:1) } tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: kafka password: testpass ``` > 📝 **NOTE** > > The Brave browser does not fully support code snippets. 5. Click **Create**. Your pipeline details are displayed and the pipeline state changes from **Starting** to **Running**, which may take a few minutes. If you don’t see this state change, refresh your page. Next, add a Redpanda Connect consumer, which reads messages from the `source` cluster topics, and leave it running. This consumer uses the `foobar` consumer group, which is reused in a later step when consuming from the `destination` cluster. 1. Go to the **Connect** page on your cluster and click **Create pipeline**. 2. In **Pipeline name**, enter a name and add a short description. 3. For **Compute units**, leave the default value of **1**. 4. For **Configuration**, paste the following configuration. `read_data_source.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "source.cloud.kafka.com:9092" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: kafka password: testpass processors: - schema_registry_decode: url: "https://schema-registry-source.cloud.kafka.com:30081" avro_raw_json: true basic_auth: enabled: true username: kafka password: testpass output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` > 📝 **NOTE** > > The Brave browser does not fully support code snippets. 5. Click **Create**. Your pipeline details are displayed and the pipeline state changes from **Starting** to **Running**, which may take a few minutes. If you don’t see this state change, refresh your page. At this point, the `source` cluster has some data in both `foo` and `bar` topics, and the consumer prints the messages it reads from these topics to `stdout`. ## [](#required-permissions)Required permissions This cookbook authenticates as a superuser for simplicity. In production, when the source and destination clusters enforce authorization (ACLs), a basic consumer or producer ACL is not sufficient for Redpanda Migrator. The migrator authenticates to the source cluster with the credentials in the `redpanda_migrator` input, and to the destination cluster with the credentials in the `redpanda_migrator` output, so grant ACLs to each principal as follows. > ❗ **IMPORTANT** > > To recreate each topic on the destination with matching settings, Redpanda Migrator reads the source topic’s configuration with a `DescribeConfigs` request, which requires the `DESCRIBE_CONFIGS` operation on the topic. > > A consumer ACL (`READ`) implicitly grants `DESCRIBE`, but it does **not** grant `DESCRIBE_CONFIGS`. If the source principal has only `READ`, the migrator consumes messages successfully but fails to create topics, logging an error such as: > > ```text > level=error msg="Failed to send message to redpanda_migrator: creating records: sync topics: > create topic : get topic details : TOPIC_AUTHORIZATION_FAILED: > Not authorized to access topics: [Topic authorization failed.]" > ``` > > Despite the `create topic` wording, this failure is the `DescribeConfigs` read against the **source** cluster, not the topic creation on the destination. ### [](#source-cluster)Source cluster Grant the source principal (the `redpanda_migrator` input credentials) these ACLs: | Resource | Operations | Purpose | | --- | --- | --- | | Topic (migrated topics) | READ, DESCRIBE_CONFIGS | Consume records (READ, which also grants DESCRIBE for metadata and offsets) and read topic configurations to replicate them (DESCRIBE_CONFIGS). | | Group (the input’s consumer_group) | READ | Join the migrator’s own consumer group and track progress. | | Group (migrated groups) | DESCRIBE | Read source consumer group offsets. Required when consumer group migration is enabled (consumer_groups.enabled, the default). | | Cluster | DESCRIBE | List source consumer groups. Also required to read source ACLs when sync_topic_acls is enabled on the output. | For example, to grant the least-privilege source ACLs to `User:migrator` with `rpk`: ```bash # Consume records and read topic configs (READ also grants DESCRIBE; DESCRIBE_CONFIGS does not come with READ) rpk security acl create --allow-principal User:migrator \ --operation read,describe_configs \ --topic # READ on the migrator's own consumer group rpk security acl create --allow-principal User:migrator \ --operation read \ --group # DESCRIBE on the groups being migrated (omit if consumer_groups.enabled is false) rpk security acl create --allow-principal User:migrator \ --operation describe \ --group # List consumer groups (and describe ACLs if sync_topic_acls is enabled) rpk security acl create --allow-principal User:migrator \ --operation describe \ --cluster ``` ### [](#destination-cluster)Destination cluster Grant the destination principal (the `redpanda_migrator` output credentials) these ACLs: | Resource | Operations | Purpose | | --- | --- | --- | | Topic (migrated topics) | CREATE, WRITE, ALTER, DESCRIBE_CONFIGS | Create topics (CREATE and DESCRIBE_CONFIGS), produce migrated records (WRITE), and add partitions to match the source (ALTER). These operations also grant DESCRIBE. | | Cluster | CREATE | Allow creating destination topics whose names are not known in advance. Use instead of per-topic CREATE. | | Group (migrated groups) | READ | Commit translated consumer group offsets. Required when consumer group migration is enabled (consumer_groups.enabled, the default). | | Cluster | ALTER | Create migrated ACLs on the destination. Required only when sync_topic_acls is enabled. | > 💡 **TIP** > > Run `rpk security acl --help-operations` to see which ACL operation each Kafka request requires. ## [](#configure-and-start-redpanda-migrator)Configure and start Redpanda Migrator The unified Redpanda Migrator does the following: - The `redpanda_migrator` input connects to the source Kafka cluster and Schema Registry to consume messages and schema information. - The `redpanda_migrator` output handles all migration logic: - Schema migration: reads schemas from the source Schema Registry and synchronizes them to the destination. - Topic creation: automatically creates destination topics that don’t exist with proper configurations. - ACL migration: migrates access control lists according to the migration rules. - Message streaming: processes and routes messages from source to destination topics. - Consumer group offset translation: maps source consumer group offsets to equivalent destination positions. - If new topics are created in the source cluster while the migrator is running, they are migrated when messages are written to them. ACL migration for topics adheres to the following principles: - `ALLOW WRITE` ACLs for topics are not migrated - `ALLOW ALL` ACLs for topics are downgraded to `ALLOW READ` - Group ACLs are not migrated > 📝 **NOTE** > > Changing topic configurations, such as partition count, isn’t currently supported. Now, use the following unified Redpanda Migrator configuration. See the [`redpanda_migrator` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator/) and [`redpanda_migrator` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) docs for details. 1. Go to the **Connect** page on your cluster and click **Create pipeline**. 2. In **Pipeline name**, enter a name and add a short description. 3. For **Compute units**, leave the default value of **1**. 4. For **Configuration**, paste the following configuration. `redpanda_migrator.yaml` ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "source.cloud.kafka.com:9092" ] topics: - '^[^_]' # Skip internal topics which start with `_` regexp_topics: true consumer_group: migrator tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: kafka password: testpass # Source Schema Registry settings schema_registry: url: "https://schema-registry-source.cloud.kafka.com:30081" basic_auth: enabled: true username: kafka password: testpass output: label: "migration_pipeline" (2) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "destination.cloud.redpanda.com:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: redpanda password: testpass # Destination Schema Registry and migration settings schema_registry: url: https://schema-registry-destination.cloud.redpanda.com:30081 include_deleted: true translate_ids: true basic_auth: enabled: true username: redpanda password: testpass # Consumer group migration settings consumer_groups: enabled: true interval: 30s serverless: false (3) ``` > 💡 **TIP** > > Label names must be between 3 and 128 characters and can only contain alphanumeric characters, hyphens, and underscores (`A-Za-z0-9-_`). ## [](#check-the-status-of-migrated-topics)Check the status of migrated topics You can use the Redpanda [`rpk` CLI tool](https://docs.redpanda.com/streaming/current/get-started/rpk/) to check which topics and ACLs have been migrated to the `destination` cluster. You can quickly [install `rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) if you don’t already have it. > 📝 **NOTE** > > For now, users require manual migration. However, this step is not required for the current demo. Similarly, roles are specific to Redpanda and, for now, also require manual migration if the `source` cluster is based on Redpanda. ```bash rpk -X brokers=destination.cloud.redpanda.com:9092 -X tls.enabled=true -X sasl.mechanism=SCRAM-SHA-256 -X user=redpanda -X pass=testpass topic list NAME PARTITIONS REPLICAS _schemas 1 1 bar 2 1 foo 2 1 rpk -X brokers=destination.cloud.redpanda.com:9092 -X tls.enabled=true -X sasl.mechanism=SCRAM-SHA-256 -X user=redpanda -X pass=testpass security acl list PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME RESOURCE-PATTERN-TYPE OPERATION PERMISSION ERROR User:redpanda * TOPIC bar LITERAL READ DENY User:redpanda * TOPIC foo LITERAL READ ALLOW ``` ## [](#check-metrics-to-monitor-progress)Check metrics to monitor progress Redpanda Connect provides a comprehensive suite of metrics in various formats, such as Prometheus, which you can use to monitor its performance in your observability stack. Besides the [standard Redpanda Connect metrics](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/about/#metric-names), the `redpanda_migrator` input also emits an `input_redpanda_migrator_lag` metric for monitoring the migration progress of each topic and partition. To monitor the migration progress, use the Redpanda Cloud OpenMetrics endpoint, which exposes all Redpanda and connector metrics for your cluster. You can integrate this endpoint with Prometheus, Datadog, or other observability platforms. For step-by-step instructions on configuring monitoring and connecting your observability tool, see [Monitor Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/). After ingesting the metrics, search for the `input_redpanda_migrator_lag` metric in your monitoring tool and filter by `topic` and `partition` as needed to track migration lag for each topic and partition. ## [](#read-from-the-migrated-topics)Read from the migrated topics Stop the `read_data_source.yaml` consumer you started earlier and, afterwards, start a similar consumer for the `destination` cluster. Before starting the consumer up on the `destination` cluster, make sure you give the migrator bundle some time to replicate the translated offset. 1. On the **Connect** page, stop the `read_data_source` pipeline you created earlier. 2. Go to the **Connect** page on your cluster and click **Create pipeline**. 3. In **Pipeline name**, enter a name and add a short description. 4. For **Compute units**, leave the default value of **1**. 5. For **Configuration**, paste the following configuration. `read_data_destination.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "destination.cloud.redpanda.com:9092" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar sasl: - mechanism: SCRAM-SHA-256 username: redpanda password: testpass processors: - schema_registry_decode: url: "https://schema-registry-destination.cloud.redpanda.com:30081" avro_raw_json: true basic_auth: enabled: true username: redpanda password: testpass output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` > 📝 **NOTE** > > The Brave browser does not fully support code snippets. 6. Click **Create**. Your pipeline details are displayed and the pipeline state changes from **Starting** to **Running**, which may take a few minutes. If you don’t see this state change, refresh your page. The `source` cluster consumer uses the same `foobar` consumer group. This consumer resumes reading messages from where the `source` consumer left off. Redpanda Migrator performs offset remapping when migrating consumer group offsets to the `destination` cluster. While more sophisticated approaches are possible, Redpanda chose to use a simple timestamp-based approach. So, for each migrated offset, the `destination` cluster is queried to find the latest offset before the received offset timestamp. Redpanda Migrator then writes this offset as the `destination` consumer group offset for the corresponding topic and partition pair. Although the timestamp-based approach doesn’t guarantee exactly-once delivery, it minimizes the likelihood of message duplication and avoids the need for complex and error-prone offset remapping logic. --- # Page 501: Ingest data into Snowflake **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/snowflake_ingestion.md --- # Ingest data into Snowflake > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Ingest data into Snowflake latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/cookbooks/snowflake_ingestion page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/cookbooks/snowflake_ingestion.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/cookbooks/snowflake_ingestion.adoc description: Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. page-git-created-date: "2025-01-28" page-git-modified-date: "2026-05-26" --- Configure a Redpanda Connect pipeline to generate and write data into a Redpanda Serverless topic, and then ingest that data into [Snowflake](https://www.snowflake.com/en/) using [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview). ## [](#prerequisites)Prerequisites - A [Redpanda Cloud account](https://cloud.redpanda.com/sign-up) - [`rpk` installed](https://docs.redpanda.com/current/get-started/rpk-install/) and [signed into your Cloud account](https://docs.redpanda.com/redpanda-cloud/reference/rpk/rpk-cloud/rpk-cloud-login/) - A [Snowflake account](https://trial.snowflake.com/) - `openssl` command-line tool ## [](#set-up-your-redpanda-cluster)Set up your Redpanda cluster In [Redpanda Cloud](https://cloud.redpanda.com/), create a new Serverless Standard cluster. When the cluster is ready, run `rpk cloud cluster select` to select the cluster and set it to be your current [rpk profile](https://docs.redpanda.com/current/get-started/config-rpk-profile/). Next, create a `demo_topic` to use as the data source for ingesting data into Snowflake: ```bash rpk topic create demo_topic ``` Create a user with minimal [ACLs](https://docs.redpanda.com/current/manage/security/authorization/acl/) to run the ingestion pipeline into Snowflake: ```bash rpk security user create ingestion_user --password Testing1234 ``` Now that the user exists, give them read permissions to `demo_topic`, as well as full control over any consumer group with the prefix `redpanda_connect`: ```bash rpk security acl create --allow-principal ingestion_user --operation read --topic demo_topic rpk security acl create --allow-principal ingestion_user --resource-pattern-type prefixed --operation all --group redpanda_connect ``` ## [](#set-up-your-snowflake-account)Set up your Snowflake account Log in to your Snowflake account with a user who has the ACCOUNTADMIN role. Then, run the following SQL commands in a worksheet. They set up another user with minimal permissions to write data into a specified database and schema, ready for streaming data to Snowflake. ```sql -- Set default values for multiple variables SET PWD = 'Test1234567'; SET USER = 'STREAMING_USER'; SET DB = 'STREAMING_DB'; SET ROLE = 'REDPANDA_CONNECT'; SET WH = 'STREAMING_WH'; USE ROLE ACCOUNTADMIN; -- Create users CREATE USER IF NOT EXISTS IDENTIFIER($USER) PASSWORD=$PWD COMMENT='STREAMING USER FOR REDPANDA CONNECT'; -- Create roles CREATE OR REPLACE ROLE IDENTIFIER($ROLE); -- Create the destination database and virtual warehouse CREATE DATABASE IF NOT EXISTS IDENTIFIER($DB); USE IDENTIFIER($DB); CREATE OR REPLACE WAREHOUSE IDENTIFIER($WH) WITH WAREHOUSE_SIZE = 'SMALL'; -- Grant privileges GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE IDENTIFIER($ROLE); GRANT ROLE IDENTIFIER($ROLE) TO USER IDENTIFIER($USER); GRANT OWNERSHIP ON DATABASE IDENTIFIER($DB) TO ROLE IDENTIFIER($ROLE); GRANT USAGE ON WAREHOUSE IDENTIFIER($WH) TO ROLE IDENTIFIER($ROLE); -- Set defaults ALTER USER IDENTIFIER($USER) SET DEFAULT_ROLE=$ROLE; ALTER USER IDENTIFIER($USER) SET DEFAULT_WAREHOUSE=$WH; -- Run the following commands to find your account identifier. Copy it down for later use. -- It will be something like `organization_name-account_name` -- e.g. ykmxgak-wyb52636 WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` ### [](#create-an-rsa-key-pair)Create an RSA key pair Create an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth) using `openssl` to authenticate Redpanda Connect to Snowflake. When you’re prompted to give an encryption password, record it for later. ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -passout pass:Testing123 -out rsa_key.p8 ``` Create a public key. You’re prompted to enter your encryption password. ```bash openssl rsa -in rsa_key.p8 -pubout -passout pass:Testing123 -out rsa_key.pub ``` To register the public key in Snowflake, remove the public key delimiters and output only the base64-encoded portion of the PEM file. Run the following bash command to print it: ```bash cat rsa_key.pub | sed -e '1d' -e '$d' | tr -d '\n' ``` In the Snowflake worksheet, add the output of the bash command you just ran to the following SQL command and execute it: ```sql use role accountadmin; alter user streaming_user set rsa_public_key='< PubKeyWithoutDelimiters >'; ``` ### [](#create-a-schema-using-streaming_user)Create a schema using `streaming_user` Log out of Snowflake and sign back in as the default user (`streaming_user`) with the associated password (default: `Test1234567`). You created these credentials in [Set up your Snowflake account](#set-up-your-snowflake-account). Run the following SQL commands in a worksheet to create a schema (e.g. `STREAMING_SCHEMA`) in the default database (e.g. `STREAMING_DB`): ```sql SET DB = 'STREAMING_DB'; SET SCHEMA = 'STREAMING_SCHEMA'; USE IDENTIFIER($DB); CREATE OR REPLACE SCHEMA IDENTIFIER($SCHEMA); ``` ## [](#create-a-pipeline-from-your-redpanda-cluster-to-snowflake)Create a pipeline from your Redpanda cluster to Snowflake You can now create the pipeline. First create [secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) for the passwords and keys you created during setup. On your Serverless cluster, go to the **Connect** page, select the **Secrets** tab and then create three secrets: - `REDPANDA_PASS` with the value `Testing1234` - `SNOWFLAKE_KEY` with the output value of `awk '{printf "%s\\n", $0}' rsa_key.p8` - `SNOWFLAKE_KEY_PASS` with the value `Testing123` Select the **Pipelines** tab and create a pipeline called **RedpandaToSnowflake**. Use the following YAML configuration: ```yaml input: # Reads data from our `demo_topic` kafka_franz: seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["demo_topic"] consumer_group: "redpanda_connect_to_snowflake" tls: {enabled: true} checkpoint_limit: 4096 sasl: - mechanism: SCRAM-SHA-256 username: ingestion_user password: ${secrets.REDPANDA_PASS} # Define the batching policy. This cookbook creates small batches, # but in a production environment use the largest file size you can. batching: count: 100 # Collect 10 messages before flushing period: 10s # or after 10 seconds, whichever comes first output: snowflake_streaming: # Replace this placeholder with your account identifier account: "< OrgName-AccountName >" user: STREAMING_USER role: REDPANDA_CONNECT database: STREAMING_DB schema: STREAMING_SCHEMA table: STREAMING_DATA # Inject your private key and password private_key_file: "${secrets.SNOWFLAKE_KEY}" private_key_pass: "${secrets.SNOWFLAKE_KEY_PASS}" schema_evolution: enabled: true max_in_flight: 1 ``` You now can produce some data using `rpk` to test that everything works: ```bash echo '{"animal":"redpanda","attributes":"cute","age":6}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"polar bear","attributes":"cool","age":13}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"unicorn","attributes":"rare","age":999}' | rpk topic produce demo_topic -f '%v\n' ``` The data produced into the `demo_topic` is consumed and streamed into Snowflake in seconds. Go back to the Snowflake worksheet and run the following query to see data arrive in Snowflake with the schema from the JSON data you produced. ```sql SELECT * FROM STREAMING_DB.STREAMING_SCHEMA.STREAMING_DATA LIMIT 50; ``` See also: - The [`kafka_franz` input](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/kafka_franz/) - The [`snowflake_streaming`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/snowflake_streaming/) output --- # Page 502: Guides **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides.md --- # Guides > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Guides latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/index.adoc description: Guides for building and operating Redpanda Connect pipelines in Redpanda Cloud, from Bloblang mappings to deployment patterns. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- - [Bloblang](bloblang/about/) Learn what Bloblang is and how to use the native mapping language. - Cloud Credentials - [Amazon Web Services](cloud/aws/) Find out about AWS components in Redpanda Connect. - [Google Cloud Platform](cloud/gcp/) Find out about GCP components in Redpanda Connect. - [Ingest Real-Time Sensor Telemetry with the HTTP Gateway](cloud/gateway/) Learn how to stream sensor telemetry data into Redpanda Cloud using the gateway input in Redpanda Connect. - [Synchronous Responses](sync_responses/) Understand synchronous response handling in Redpanda Connect, ensuring reliable and efficient data processing. - [Migrate to the Unified Redpanda Migrator](migrate-unified-redpanda-migrator/) Learn how to migrate from legacy migrator components to the unified \`redpanda\_migrator\` input/output pair in Redpanda Connect 4.67.5+. --- # Page 503: Bloblang **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about.md --- # Bloblang > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Bloblang latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/bloblang/about page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/bloblang/about.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/bloblang/about.adoc description: Learn what Bloblang is and how to use the native mapping language. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Bloblang, or blobl for short, is a language designed for mapping data of a wide variety of forms. It’s a safe, fast, and powerful way to perform document mapping within Redpanda Connect. It also has a [Go API for writing your own functions and methods](https://pkg.go.dev/github.com/redpanda-data/connect/v4/public/bloblang) as plugins. Bloblang is available as a [processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/mapping/) and it’s also possible to use blobl queries in [function interpolations](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/#bloblang-queries). This document outlines the core features of the Bloblang language, but if you’re totally new to Bloblang then it’s worth following [the walkthrough first](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/walkthrough/). ## [](#learn-bloblang)Learn Bloblang [learnbloblang.com](https://www.learnbloblang.com) is an interactive resource for learning Bloblang with hands-on exercises. ## [](#assignment)Assignment A Bloblang mapping expresses how to create a new document by extracting data from an existing input document. Assignments consist of a dot separated path segments on the left-hand side describing a field to be created within the new document, and a right-hand side query describing what the content of the new field should be. The keyword `root` on the left-hand side refers to the root of the new document, the keyword `this` on the right-hand side refers to the current context of the query, which is the read-only input document when querying from the root of a mapping: ```bloblang root.id = this.thing.id root.type = "yo" # Both `root` and `this` are optional, and will be inferred in their absence. content = thing.doc.message # In: {"thing":{"id":"wat1","doc":{"title":"wut","message":"hello world"}}} ``` Since the document being created starts off empty it is sometimes useful to begin a mapping by copying the entire contents of the input document, which can be expressed by assigning `this` to `root`. ```bloblang root = this root.foo = "added value" # In: {"id":"wat1","message":"hello world"} ``` If the new document `root` is never assigned to or otherwise mutated then the original document remains unchanged. ### [](#special-characters-in-paths)Special characters in paths Quotes can be used to describe sections of a field path that contain whitespace, dots or other special characters: ```bloblang # Use quotes around a path segment in order to include whitespace or dots within # the path root."foo.bar".baz = this."buz bev".fub # In: {"buz bev":{"fub":"hello world"}} ``` ### [](#non-structured-data)Non-structured data Bloblang is able to map data that is unstructured, whether it’s a log line or a binary blob, by referencing it with the [`content` function](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#content), which returns the raw bytes of the input document: ```bloblang # Parse a base64 encoded JSON document root = content().decode("base64").parse_json() # In: eyJmb28iOiJiYXIifQ== ``` And your newly mapped document can also be unstructured, simply assign a value type to the `root` of your document: ```bloblang root = this.foo # In: {"foo":"hello world"} ``` And the resulting message payload will be the raw value you’ve assigned. ### [](#deleting)Deleting It’s possible to selectively delete fields from an object by assigning the function `deleted()` to the field path: ```bloblang root = this root.bar = deleted() # In: {"id":"wat1","message":"hello world","bar":"remove me"} ``` ### [](#variables)Variables Another type of assignment is a `let` statement, which creates a variable that can be referenced elsewhere within a mapping. Variables are discarded at the end of the mapping and are mostly useful for query reuse. Variables are referenced within queries with `$`: ```bloblang # Set a temporary variable let foo = "yo" root.new_doc.type = $foo ``` ### [](#metadata)Metadata Redpanda Connect messages contain metadata that is separate from the main payload, in Bloblang you can modify the metadata of the resulting message with the `meta` assignment keyword. Metadata values of the resulting message are referenced within queries with the `@` operator or the [`metadata()` function](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#metadata): ```bloblang # Reference a metadata value root.new_doc.bar = @kafka_topic # Or `@.kafka_topic` or `metadata("kafka_topic")` # Delete all metadata meta = deleted() # Set metadata values meta bar = "hello world" meta baz = { "something": "structured" } # Get an object of key/values for all metadata root.meta_obj = @ # Or `metadata()` ``` ## [](#coalesce)Coalesce The pipe operator (`|`) used within brackets allows you to coalesce multiple candidates for a path segment. The first field that exists and has a non-null value will be selected: ```bloblang root.new_doc.type = this.thing.(article | comment | this).type # In: {"thing":{"article":{"type":"foo"}}} # In: {"thing":{"comment":{"type":"bar"}}} # In: {"thing":{"type":"baz"}} ``` Opening brackets on a field begins a query where the context of `this` changes to value of the path it is opened upon, therefore in the above example `this` within the brackets refers to the contents of `this.thing`. ## [](#literals)Literals Bloblang supports number, boolean, string, null, array and object literals: ```bloblang root = [ 7, false, "string", null, { "first": 11, "second": {"foo":"bar"}, "third": """multiple lines on this string""" } ] # In: {} ``` The values within literal arrays and objects can be dynamic query expressions, as well as the keys of object literals. ## [](#comments)Comments You might’ve already spotted, comments are started with a hash (`#`) and end with a line break: ```bloblang root = this.some.value # And now this is a comment ``` ## [](#boolean-logic-and-arithmetic)Boolean logic and arithmetic Bloblang supports a range of boolean operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`: ```bloblang root.is_big = this.number > 100 root.multiplied = this.number * 7 # In: {"number":50} # In: {"number":150} ``` For more information about these operators and how they work check out [the arithmetic page](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/arithmetic/). ## [](#conditional-mapping)Conditional mapping Use `if` as either a statement or an expression in order to perform maps conditionally: ```bloblang root = this root.sorted_foo = if this.foo.type() == "array" { this.foo.sort() } if this.foo.type() == "string" { root.upper_foo = this.foo.uppercase() root.lower_foo = this.foo.lowercase() } # In: {"foo":"FooBar"} # In: {"foo":["foo","bar"]} ``` And add as many `else if` queries as you like, followed by an optional final fallback `else`: ```bloblang root.sound = if this.type == "cat" { this.cat.meow } else if this.type == "dog" { this.dog.woof.uppercase() } else { "sweet sweet silence" } # In: {"type":"cat","cat":{"meow":"meeeeooooow!"}} # In: {"type":"dog","dog":{"woof":"guurrrr woof woof!"}} # In: {"type":"caterpillar","caterpillar":{"name":"oleg"}} ``` ## [](#pattern-matching)Pattern matching A `match` expression allows you to perform conditional mappings on a value, each case should be either a boolean expression, a literal value to compare against the target value, or an underscore (`_`) which captures values that have not matched a prior case: ```bloblang root.new_doc = match this.doc { this.type == "article" => this.article this.type == "comment" => this.comment _ => this } # In: {"doc":{"type":"article","article":{"id":"foo","content":"qux"}}} # In: {"doc":{"type":"comment","comment":{"id":"bar","content":"quz"}}} # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` Within a match block the context of `this` changes to the pattern matched expression, therefore `this` within the match expression above refers to `this.doc`. Match cases can specify a literal value for simple comparison: ```bloblang root = this root.type = match this.type { "doc" => "document", "art" => "article", _ => this } # In: {"type":"doc","foo":"bar"} ``` The match expression can also be left unset which means the context remains unchanged, and the catch-all case can also be omitted: ```bloblang root.new_doc = match { this.doc.type == "article" => this.doc.article this.doc.type == "comment" => this.doc.comment } # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` If no case matches then the mapping is skipped entirely, hence we would end up with the original document in this case. ## [](#functions)Functions Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` You can find a full list of functions and their parameters in [the functions page](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/). ## [](#methods)Methods Methods are similar to functions but enact upon a target value, these provide most of the power in Bloblang as they allow you to augment query values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,8,15,20]},"user":{"fav_pokemon":"pikachu"}} ``` Methods also support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` You can find a full list of methods and their parameters in [the methods page](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/). ## [](#maps)Maps Defining named maps allows you to reuse common mappings on values with the [`apply` method](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#apply): ```bloblang map things { root.first = this.thing_one root.second = this.thing_two } root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Within a map the keyword `root` refers to a newly created document that will replace the target of the map, and `this` refers to the original value of the target. The argument of `apply` is a string, which allows you to dynamically resolve the mapping to apply. ## [](#import-maps)Import maps It’s possible to import maps defined in a file with an `import` statement: ```bloblang import "./common_maps.blobl" root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Imports from a Bloblang mapping within a Redpanda Connect config are relative to the process running the config. Imports from an imported file are relative to the file that is importing it. ## [](#filtering)Filtering By assigning the root of a mapped document to the `deleted()` function you can delete a message entirely: ```bloblang # Filter all messages that have fewer than 10 URLs. root = if this.doc.urls.length() < 10 { deleted() } # In: {"doc":{"urls":["a","b","c"]}} # In: {"doc":{"urls":["a","b","c","d","e","f","g","h","i","j"]}} ``` ## [](#error-handling)Error handling Functions and methods can fail under certain circumstances, such as when they receive types they aren’t able to act upon. These failures, when not caught, will cause the entire mapping to fail. However, the [method `catch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#catch) can be used in order to return a value when a failure occurs instead: ```bloblang # Map an empty array to `foo` if the field `bar` is not a string. root.foo = this.bar.split(",").catch([]) # In: {"bar":"a,b,c"} # In: {"bar":123} ``` Since `catch` is a method it can also be attached to bracketed map expressions: ```bloblang # Map `false` if any of the operations in this boolean query fail. root.thing = ( this.foo > this.bar && this.baz.contains("wut") ).catch(false) # In: {"foo":10,"bar":5,"baz":"wut wut"} # In: {"foo":"not a number","bar":5,"baz":"wut wut"} ``` And one of the more powerful features of Bloblang is that a single `catch` method at the end of a chain of methods can recover errors from any method in the chain: ```bloblang # Catch errors caused by: # - foo not existing # - foo not being a string # - an element from split foo not being a valid JSON string root.things = this.foo.split(",").map_each( ele -> ele.parse_json() ).catch([]) # Specifically catch a JSON parse error root.things = this.foo.split(",").map_each( ele -> ele.parse_json().catch({}) ) # In: {"foo":"{\"a\":1},{\"b\":2}"} # In: {"foo":"not valid json"} ``` However, the `catch` method only acts on errors, sometimes it’s also useful to set a fall back value when a query returns `null` in which case the [method `or`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#or) can be used the same way: ```bloblang # Map "default" if either the element index 5 does not exist, or the underlying # element is `null`. root.foo = this.bar.index(5).or("default") # In: {"bar":["a","b","c"]} # In: {"bar":["a","b","c","d","e","f","g"]} ``` ## [](#unit-testing)Unit testing It’s possible to execute unit tests for your Bloblang mappings using the standard Redpanda Connect unit test capabilities outlined [in this document](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/unit_testing/). ## [](#troubleshooting)Troubleshooting 1. I’m seeing `unable to reference message as structured (with 'this')` when I try to run mappings with `rpk connect blobl`. That particular error message means the mapping is failing to parse what’s being fed in as a JSON document. Make sure that the data you are feeding in is valid JSON, and also that the documents _do not_ contain line breaks as `rpk connect blobl` will parse each line individually. Why? That’s a good question. Bloblang supports non-JSON formats too, so it can’t delimit documents with a streaming JSON parser like tools such as `jq`, so instead it uses line breaks to determine the boundaries of each message. --- # Page 504: Bloblang Arithmetic **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/arithmetic.md --- # Bloblang Arithmetic > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Bloblang Arithmetic latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/bloblang/arithmetic page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/bloblang/arithmetic.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/bloblang/arithmetic.adoc description: How arithmetic works within Bloblang page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Bloblang supports a range of comparison operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`. How these operators behave is dependent on the type of the values they’re used with, and therefore it’s worth fully understanding these behaviors if you intend to use them heavily in your mappings. ## [](#mathematical)Mathematical All mathematical operators (`+`, `-`, `*`, `/`, `%`) are valid against number values, and addition (`+`) is also supported when both the left and right hand side arguments are strings. If a mathematical operator is used with an argument that is non-numeric (with the aforementioned string exception) then a [recoverable mapping error will be thrown](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). ### [](#number-degradation)Number degradation In Bloblang any number resulting from a method, function or arithmetic is either a 64-bit signed integer or a 64-bit floating point value. Numbers from input documents can be any combination of size and be signed or unsigned. When a mathematical operation is performed with two or more integer values Bloblang will create an integer result, with the exception of division. However, if any number within a mathematical operation is a floating point then the result will be a floating point value. In order to explicitly coerce numbers into integer types you can use the [`.ceil()`, `.floor()`, or `.round()` methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#number-manipulation). ## [](#comparison)Comparison The not (`!`) operator reverses the boolean value of the expression immediately following it, and is valid to place before any query that yields a boolean value. If the following expression yields a non-boolean value then a [recoverable mapping error will be thrown](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). If you wish to reverse the boolean result of a complex query then simply place the query within brackets (`!(this.foo > this.bar)`). ### [](#equality)Equality The equality operators (`==` and `!=`) are valid to use against any value type. In order for arguments to be considered equal they must match in both their basic type (`string`, `number`, `null`, `bool`, etc) as well as their value. If you wish to compare mismatched value types then use [coercion methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#type-coercion). Number arguments are considered equal if their value is the same when represented the same way, which means their underlying representations (integer, float, etc) do not need to match in order for them to be considered equal. ### [](#numerical)Numerical Numerical comparisons (`>`, `>=`, `<`, `<=`) are valid to use against number values only. If a non-number value is used as an argument then a [recoverable mapping error will be thrown](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). ### [](#boolean)Boolean Boolean comparison operators (`||`, `&&`) are valid to use against boolean values only (`true` or `false`). If a non-boolean value is used as an argument then a [recoverable mapping error will be thrown](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#error-handling). --- # Page 505: Bloblang Functions **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions.md --- # Bloblang Functions > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Bloblang Functions latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/bloblang/functions page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/bloblang/functions.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/bloblang/functions.adoc description: A list of Bloblang functions page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` ## [](#batch_index)batch_index Returns the zero-based index of the current message within its batch. Use this to conditionally process messages based on their position, or to create sequential identifiers within a batch. ### [](#examples)Examples ```bloblang root = if batch_index() > 0 { deleted() } ``` Create a unique identifier combining batch position with timestamp: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#batch_size)batch_size Returns the total number of messages in the current batch. Use this to determine batch boundaries or compute relative positions. ### [](#examples-2)Examples ```bloblang root.total = batch_size() ``` Check if processing the last message in a batch: ```bloblang root.is_last = batch_index() == batch_size() - 1 ``` ## [](#bytes)bytes Creates a zero-initialized byte array of specified length. Use this to allocate fixed-size byte buffers for binary data manipulation or to generate padding. ### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | length | integer | The size of the resulting byte array. | ### [](#examples-3)Examples ```bloblang root.data = bytes(5) ``` Create a buffer for binary operations: ```bloblang root.header = bytes(16) root.payload = content() ``` ## [](#content)content Returns the raw message payload as bytes, regardless of the current mapping context. Use this to access the original message when working within nested contexts, or to store the entire message as a field. ### [](#examples-4)Examples ```bloblang root.doc = content().string() # In: {"foo":"bar"} # Out: {"doc":"{\"foo\":\"bar\"}"} ``` Preserve original message while adding metadata: ```bloblang root.original = content().string() root.processed_by = "ai" # In: {"foo":"bar"} # Out: {"original":"{\"foo\":\"bar\"}","processed_by":"ai"} ``` ## [](#count)count > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. The `count` function is a counter starting at 1 which increments after each time it is called. Count takes an argument which is an identifier for the counter, allowing you to specify multiple unique counters in your configuration. ### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | An identifier for the counter. | ### [](#examples-5)Examples ```bloblang root = this root.id = count("bloblang_function_example") # In: {"message":"foo"} # Out: {"id":1,"message":"foo"} # In: {"message":"bar"} # Out: {"id":2,"message":"bar"} ``` ## [](#counter)counter Generates an incrementing sequence of integers starting from a minimum value (default 1). Each counter instance maintains its own independent state across message processing. When the maximum value is reached, the counter automatically resets to the minimum. ### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | min | query expression | The starting value of the counter. This is the first value yielded. Evaluated once when the mapping is initialized. | | max | query expression | The maximum value before the counter resets to min. Evaluated once when the mapping is initialized. | | set (optional) | query expression | An optional query that controls counter behavior: when it resolves to a non-negative integer, the counter is set to that value; when it resolves to null, the counter is read without incrementing; when it resolves to a deletion, the counter resets to min; otherwise the counter increments normally. | ### [](#examples-6)Examples Generate sequential IDs for each message: ```bloblang root.id = counter() # In: {} # Out: {"id":1} # In: {} # Out: {"id":2} ``` Use a custom range for the counter: ```bloblang root.batch_num = counter(min: 100, max: 200) # In: {} # Out: {"batch_num":100} # In: {} # Out: {"batch_num":101} ``` Increment a counter multiple times within a single mapping using a named map: ```bloblang map increment { root = counter() } root.first_id = null.apply("increment") root.second_id = null.apply("increment") # In: {} # Out: {"first_id":1,"second_id":2} # In: {} # Out: {"first_id":3,"second_id":4} ``` Conditionally reset a counter based on input data: ```bloblang root.streak = counter(set: if this.status != "success" { 0 }) # In: {"status":"success"} # Out: {"streak":1} # In: {"status":"success"} # Out: {"streak":2} # In: {"status":"failure"} # Out: {"streak":0} # In: {"status":"success"} # Out: {"streak":1} ``` Peek at the current counter value without incrementing by using null in the set parameter: ```bloblang root.count = counter(set: if this.peek { null }) # In: {"peek":false} # Out: {"count":1} # In: {"peek":false} # Out: {"count":2} # In: {"peek":true} # Out: {"count":2} # In: {"peek":false} # Out: {"count":3} ``` ## [](#deleted)deleted Returns a deletion marker that removes the target field or message. When applied to root, the entire message is dropped while still being acknowledged as successfully processed. Use this to filter data or conditionally remove fields. ### [](#examples-7)Examples ```bloblang root = this root.bar = deleted() # In: {"bar":"bar_value","baz":"baz_value","foo":"foo value"} # Out: {"baz":"baz_value","foo":"foo value"} ``` Filter array elements by returning deleted for unwanted items: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ## [](#env)env Reads an environment variable and returns its value as a string. Returns `null` if the variable is not set. By default, values are cached for performance. ### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | The name of the environment variable to read. | | no_cache | bool | Disable caching to read the latest value on each invocation. | ### [](#examples-8)Examples ```bloblang root.api_key = env("API_KEY") ``` ```bloblang root.database_url = env("DB_URL").or("localhost:5432") ``` Use `no_cache` to read updated environment variables during runtime, useful for dynamic configuration changes: ```bloblang root.config = env(name: "DYNAMIC_CONFIG", no_cache: true) ``` ## [](#error)error Returns the error message string if the message has failed processing, otherwise `null`. Use this in error handling pipelines to log or route failed messages based on their error details. ### [](#examples-9)Examples ```bloblang root.doc.error = error() ``` Route messages to different outputs based on error presence: ```bloblang root = this root.error_msg = error() root.has_error = error() != null ``` ## [](#error_source_label)error_source_label Returns the user-defined label of the component that caused the error, empty string if no label is set, or `null` if the message has no error. Use this for more human-readable error tracking when components have custom labels. ### [](#examples-10)Examples ```bloblang root.doc.error_source_label = error_source_label() ``` Route errors based on component labels: ```bloblang root.error_category = error_source_label().or("unknown") ``` ## [](#error_source_name)error_source_name Returns the component name that caused the error, or `null` if the message has no error or the error has no associated component. Use this to identify which processor or component in your pipeline caused a failure. ### [](#examples-11)Examples ```bloblang root.doc.error_source_name = error_source_name() ``` Create detailed error logs with component information: ```bloblang root.error_details = if errored() { { "message": error(), "component": error_source_name(), "timestamp": now() } } ``` ## [](#error_source_path)error_source_path Returns the dot-separated path to the component that caused the error, or `null` if the message has no error. Use this to identify the exact location of a failed component in nested pipeline configurations. ### [](#examples-12)Examples ```bloblang root.doc.error_source_path = error_source_path() ``` Build comprehensive error context for debugging: ```bloblang root.error_info = { "path": error_source_path(), "component": error_source_name(), "message": error() } ``` ## [](#errored)errored Returns true if the message has failed processing, false otherwise. Use this for conditional logic in error handling workflows or to route failed messages to dead letter queues. ### [](#examples-13)Examples ```bloblang root.doc.status = if errored() { 400 } else { 200 } ``` Send only failed messages to a separate stream: ```bloblang root = if errored() { this } else { deleted() } ``` ## [](#fake)fake Generates realistic fake data for testing and development purposes. Supports a wide variety of data types including personal information, network addresses, dates/times, financial data, and UUIDs. Useful for creating mock data, populating test databases, or anonymizing sensitive information. Supported functions: `latitude`, `longitude`, `unix_time`, `date`, `time_string`, `month_name`, `year_string`, `day_of_week`, `day_of_month`, `timestamp`, `century`, `timezone`, `time_period`, `email`, `mac_address`, `domain_name`, `url`, `username`, `ipv4`, `ipv6`, `password`, `jwt`, `word`, `sentence`, `paragraph`, `cc_type`, `cc_number`, `currency`, `amount_with_currency`, `title_male`, `title_female`, `first_name`, `first_name_male`, `first_name_female`, `last_name`, `name`, `gender`, `chinese_first_name`, `chinese_last_name`, `chinese_name`, `phone_number`, `toll_free_phone_number`, `e164_phone_number`, `uuid_hyphenated`, `uuid_digit`. ### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | function | string | The name of the faker function to use. See description for full list of supported functions. | ### [](#examples-14)Examples Generate fake user profile data for testing: ```bloblang root.user = { "id": fake("uuid_hyphenated"), "name": fake("name"), "email": fake("email"), "created_at": fake("timestamp") } ``` Create realistic test data for network monitoring: ```bloblang root.event = { "source_ip": fake("ipv4"), "mac_address": fake("mac_address"), "url": fake("url") } ``` ## [](#file)file Reads a file and returns its contents as bytes. Paths are resolved from the process working directory. For paths relative to the mapping file, use `file_rel`. By default, files are cached after first read. ### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The absolute or relative path to the file. | | no_cache | bool | Disable caching to read the latest file contents on each invocation. | ### [](#examples-15)Examples ```bloblang root.config = file("/etc/config.json").parse_json() ``` ```bloblang root.template = file("./templates/email.html").string() ``` Use `no_cache` to read updated file contents during runtime, useful for hot-reloading configuration: ```bloblang root.rules = file(path: "/etc/rules.yaml", no_cache: true).parse_yaml() ``` ## [](#file_rel)file_rel Reads a file and returns its contents as bytes. Paths are resolved relative to the mapping file’s directory, making it portable across different environments. By default, files are cached after first read. ### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The path to the file, relative to the mapping file’s directory. | | no_cache | bool | Disable caching to read the latest file contents on each invocation. | ### [](#examples-16)Examples ```bloblang root.schema = file_rel("./schemas/user.json").parse_json() ``` ```bloblang root.lookup = file_rel("../data/lookup.csv").parse_csv() ``` Use `no_cache` to read updated file contents during runtime, useful for reloading data files without restarting: ```bloblang root.translations = file_rel(path: "./i18n/en.yaml", no_cache: true).parse_yaml() ``` ## [](#hostname)hostname Returns the hostname of the machine running Benthos. Useful for identifying which instance processed a message in distributed deployments. ### [](#examples-17)Examples ```bloblang root.processed_by = hostname() ``` ## [](#json)json Returns a field from the original JSON message by dot path, always accessing the root document regardless of mapping context. Use this to reference the source message when working in nested contexts or to extract specific fields. ### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | An optional [dot path][field_paths] identifying a field to obtain. | ### [](#examples-18)Examples ```bloblang root.mapped = json("foo.bar") # In: {"foo":{"bar":"hello world"}} # Out: {"mapped":"hello world"} ``` Access the original message from within nested mapping contexts: ```bloblang root.doc = json() # In: {"foo":{"bar":"hello world"}} # Out: {"doc":{"foo":{"bar":"hello world"}}} ``` ## [](#ksuid)ksuid Generates a K-Sortable Unique Identifier with built-in timestamp ordering. Use this for distributed unique IDs that sort chronologically and remain collision-resistant without coordination between generators. ### [](#examples-19)Examples ```bloblang root.id = ksuid() ``` Create sortable event IDs for logging: ```bloblang root.event = { "id": ksuid(), "type": this.event_type, "data": this.payload } ``` ## [](#meta)meta > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Returns the value of a metadata key from the input message as a string, or `null` if the key does not exist. Since values are extracted from the read-only input message they do NOT reflect changes made from within the map. In order to query metadata mutations made within a mapping use the [`root_meta` function](#root_meta). This function supports extracting metadata from other messages of a batch with the `from` method. ### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-20)Examples ```bloblang root.topic = meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = meta() ``` ## [](#metadata)metadata Returns metadata from the input message by key, or `null` if the key doesn’t exist. This reads the original metadata; to access modified metadata during mapping, use the `@` operator instead. Use this to extract message properties like topics, headers, or timestamps. ### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-21)Examples ```bloblang root.topic = metadata("kafka_topic") ``` Retrieve all metadata as an object by omitting the key parameter: ```bloblang root.all_metadata = metadata() ``` Copy specific metadata fields to the message body: ```bloblang root.source = { "topic": metadata("kafka_topic"), "partition": metadata("kafka_partition"), "timestamp": metadata("kafka_timestamp_unix") } ``` ## [](#nanoid)nanoid Generates a URL-safe unique identifier using Nano ID. Use this for compact, URL-friendly IDs with good collision resistance. Customize the length (default 21) or provide a custom alphabet for specific character requirements. ### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | length (optional) | integer | An optional length. | | alphabet (optional) | string | An optional custom alphabet to use for generating IDs. When specified the field length must also be present. | ### [](#examples-22)Examples ```bloblang root.id = nanoid() ``` Generate a longer ID for additional uniqueness: ```bloblang root.id = nanoid(54) ``` Use a custom alphabet for domain-specific IDs: ```bloblang root.id = nanoid(54, "abcde") ``` ## [](#nothing)nothing ## [](#now)now Returns the current timestamp as an RFC 3339 formatted string with nanosecond precision. Use this to add processing timestamps to messages or measure time between events. Chain with `ts_format` to customize the format or timezone. ### [](#examples-23)Examples ```bloblang root.received_at = now() ``` Format the timestamp in a custom format and timezone: ```bloblang root.received_at = now().ts_format("Mon Jan 2 15:04:05 -0700 MST 2006", "UTC") ``` ## [](#pi)pi Returns the value of the mathematical constant Pi. ### [](#examples-24)Examples ```bloblang root.radians = this.degrees * (pi() / 180) # In: {"degrees":45} # Out: {"radians":0.7853981633974483} ``` ```bloblang root.degrees = this.radians * (180 / pi()) # In: {"radians":0.78540} # Out: {"degrees":45.00010522957486} ``` ## [](#random_int)random_int Generates a pseudo-random non-negative 64-bit integer. Use this for creating random IDs, sampling data, or generating test values. Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance. Optional `min` and `max` parameters constrain the output range (both inclusive). For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`. ### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | seed | query expression | A seed to use, if a query is provided it will only be resolved once during the lifetime of the mapping. | | min | integer | The minimum value the random generated number will have. The default value is 0. | | max | integer | The maximum value the random generated number will have. The default value is 9223372036854775806 (math.MaxInt64 - 1). | ### [](#examples-25)Examples ```bloblang root.first = random_int() root.second = random_int(1) root.third = random_int(max:20) root.fourth = random_int(min:10, max:20) root.fifth = random_int(timestamp_unix_nano(), 5, 20) root.sixth = random_int(seed:timestamp_unix_nano(), max:20) ``` Use a dynamic seed for unique random values per mapping instance: ```bloblang root.random_id = random_int(timestamp_unix_nano()) root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100) ``` ## [](#range)range Creates an array of integers from start (inclusive) to stop (exclusive) with an optional step. Use this to generate sequences for iteration, indexing, or creating numbered lists. ### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | start | integer | The start value. | | stop | integer | The stop value. | | step | integer | The step value. | ### [](#examples-26)Examples ```bloblang root.a = range(0, 10) root.b = range(start: 0, stop: this.max, step: 2) # Using named params root.c = range(0, -this.max, -2) # In: {"max":10} # Out: {"a":[0,1,2,3,4,5,6,7,8,9],"b":[0,2,4,6,8],"c":[0,-2,-4,-6,-8]} ``` Generate a sequence for batch processing: ```bloblang root.pages = range(0, this.total_items, 100).map_each(offset -> { "offset": offset, "limit": 100 }) # In: {"total_items":250} # Out: {"pages":[{"limit":100,"offset":0},{"limit":100,"offset":100}]} ``` ## [](#root_meta)root_meta > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Returns the value of a metadata key from the new message being created as a string, or `null` if the key does not exist. Changes made to metadata during a mapping will be reflected by this function. ### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-27)Examples ```bloblang root.topic = root_meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = root_meta() ``` ## [](#snowflake_id)snowflake_id Generates a unique, time-ordered Snowflake ID. Snowflake IDs are 64-bit integers that encode timestamp, node ID, and sequence information, making them ideal for distributed systems where sortable unique identifiers are needed. Returns a string representation of the ID. ### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | node_id | integer | Optional node identifier (0-1023) to distinguish IDs generated by different machines in a distributed system. Defaults to 1. | ### [](#examples-28)Examples Generate a unique Snowflake ID for each message: ```bloblang root.id = snowflake_id() root.payload = this ``` Generate Snowflake IDs with different node IDs for multi-datacenter deployments: ```bloblang root.id = snowflake_id(42) root.data = this ``` ## [](#throw)throw Immediately fails the mapping with a custom error message. Use this to halt processing when data validation fails or required fields are missing, causing the message to be routed to error handlers. ### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | why | string | A string explanation for why an error was thrown, this will be added to the resulting error message. | ### [](#examples-29)Examples ```bloblang root.doc.type = match { this.exists("header.id") => "foo" this.exists("body.data") => "bar" _ => throw("unknown type") } root.doc.contents = (this.body.content | this.thing.body) # In: {"header":{"id":"first"},"thing":{"body":"hello world"}} # Out: {"doc":{"contents":"hello world","type":"foo"}} # In: {"nothing":"matches"} # Out: Error("failed assignment (line 1): unknown type") ``` Validate required fields before processing: ```bloblang root = if this.exists("user_id") { this } else { throw("missing required field: user_id") } # In: {"user_id":123,"name":"alice"} # Out: {"name":"alice","user_id":123} # In: {"name":"bob"} # Out: Error("failed assignment (line 1): missing required field: user_id") ``` ## [](#timestamp_unix)timestamp_unix Returns the current Unix timestamp in seconds since epoch. Use this for numeric timestamps compatible with most systems, or as a seed for random number generation. ### [](#examples-30)Examples ```bloblang root.received_at = timestamp_unix() ``` Create a sortable ID combining timestamp with a counter: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#timestamp_unix_micro)timestamp_unix_micro Returns the current Unix timestamp in microseconds since epoch. Use this for high-precision timing measurements or when microsecond resolution is required. ### [](#examples-31)Examples ```bloblang root.received_at = timestamp_unix_micro() ``` Measure elapsed time between events: ```bloblang root.processing_duration_us = timestamp_unix_micro() - this.start_time_us ``` ## [](#timestamp_unix_milli)timestamp_unix_milli Returns the current Unix timestamp in milliseconds since epoch. Use this for millisecond-precision timestamps common in web APIs and JavaScript systems. ### [](#examples-32)Examples ```bloblang root.received_at = timestamp_unix_milli() ``` Add processing time metadata: ```bloblang meta processing_time_ms = timestamp_unix_milli() ``` ## [](#timestamp_unix_nano)timestamp_unix_nano Returns the current Unix timestamp in nanoseconds since epoch. Use this for the highest precision timing or as a unique seed value that changes on every invocation. ### [](#examples-33)Examples ```bloblang root.received_at = timestamp_unix_nano() ``` Generate unique random values on each mapping: ```bloblang root.random_value = random_int(timestamp_unix_nano()) ``` ## [](#tracing_id)tracing_id Returns the OpenTelemetry trace ID for the message, or an empty string if no tracing span exists. Use this to correlate logs and events with distributed traces. ### [](#examples-34)Examples ```bloblang meta trace_id = tracing_id() ``` Add trace ID to structured logs: ```bloblang root.log_entry = this root.log_entry.trace_id = tracing_id() ``` ## [](#tracing_span)tracing_span Returns the OpenTelemetry tracing span attached to the message as a text map object, or `null` if no span exists. Use this to propagate trace context to downstream systems via headers or metadata. ### [](#examples-35)Examples ```bloblang root.headers.traceparent = tracing_span().traceparent # In: {"some_stuff":"just can't be explained by science"} # Out: {"headers":{"traceparent":"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"}} ``` Forward all tracing fields to output metadata: ```bloblang meta = tracing_span() ``` ## [](#ulid)ulid Generates a Universally Unique Lexicographically Sortable Identifier (ULID). ULIDs are 128-bit identifiers that are sortable by creation time, URL-safe, and case-insensitive. They consist of a 48-bit timestamp (millisecond precision) and 80 bits of randomness, making them ideal for distributed systems that need time-ordered unique IDs without coordination. ### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | encoding | string | Encoding format for the ULID. "crockford" produces 26-character Base32 strings (recommended). "hex" produces 32-character hexadecimal strings. | | random_source | string | Randomness source: "secure_random" uses cryptographically secure random (recommended for production), "fast_random" uses faster but non-secure random (only for non-sensitive testing). | ### [](#examples-36)Examples Generate time-sortable IDs for distributed message ordering: ```bloblang root.message_id = ulid() root.timestamp = now() root.data = this ``` Generate hex-encoded ULIDs for systems that prefer hexadecimal format: ```bloblang root.id = ulid("hex") ``` ## [](#uuid_v4)uuid_v4 Generates a random RFC-4122 version 4 UUID. Use this for creating unique identifiers that don’t reveal timing information or require ordering. Each invocation produces a new globally unique ID. ### [](#examples-37)Examples ```bloblang root.id = uuid_v4() ``` Add unique request IDs for tracing: ```bloblang root = this root.request_id = uuid_v4() ``` ## [](#uuid_v7)uuid_v7 Generates a time-ordered UUID version 7 with millisecond timestamp precision. Use this for sortable unique identifiers that maintain chronological ordering, ideal for database keys or event IDs. Optionally specify a custom timestamp. ### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | time (optional) | timestamp | An optional timestamp to use for the time ordered portion of the UUID. | ### [](#examples-38)Examples ```bloblang root.id = uuid_v7() ``` Generate a UUID with a specific timestamp for backdating events: ```bloblang root.id = uuid_v7(now().ts_sub_iso8601("PT1M")) ``` ## [](#var)var ### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | The name of the target variable. | ## [](#with_schema_registry_header)with_schema_registry_header Prepends a Confluent Schema Registry wire format header to message bytes. The header is 5 bytes: a magic byte (0x00) followed by a 4-byte big-endian schema ID. This format is required when producing messages to Kafka topics that use Confluent Schema Registry for schema validation and evolution. ### [](#parameters-20)Parameters | Name | Type | Description | | --- | --- | --- | | schema_id | unknown | The schema ID from your Schema Registry (0 to 4294967295). This ID references the schema version used to encode the message. | | message | unknown | The serialized message bytes (e.g., Avro, Protobuf, or JSON Schema encoded data) to prepend the header to. | ### [](#examples-39)Examples Add Schema Registry header to Avro-encoded message: ```bloblang root = with_schema_registry_header(123, content()) ``` Use schema ID from metadata to add header dynamically: ```bloblang root = with_schema_registry_header(meta("schema_id").number(), content()) ``` --- # Page 506: Bloblang Methods **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods.md --- # Bloblang Methods > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Bloblang Methods latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/bloblang/methods page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/bloblang/methods.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/bloblang/methods.adoc description: A list of Bloblang methods page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}} ``` Methods support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` ## [](#general)General ### [](#apply)apply Apply a declared mapping to a target value. #### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to apply. | #### [](#examples)Examples ```bloblang map thing { root.inner = this.first } root.foo = this.doc.apply("thing") # In: {"doc":{"first":"hello world"}} # Out: {"foo":{"inner":"hello world"}} ``` ```bloblang map create_foo { root.name = "a foo" root.purpose = "to be a foo" } root = this root.foo = null.apply("create_foo") # In: {"id":"1234"} # Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"} ``` ### [](#catch)catch If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead. #### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails. | #### [](#examples-2)Examples ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) ``` The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back: ```bloblang root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url}) # In: {"url":"invalid %&# url"} # Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}} ``` When the input document is not structured attempting to reference structured fields with `this` will result in an error. Therefore, a convenient way to delete non-structured data is with a catch: ```bloblang root = this.catch(deleted()) # In: {"doc":{"foo":"bar"}} # Out: {"doc":{"foo":"bar"}} # In: not structured data # Out: ``` ### [](#from)from Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are `content`, `json` and `meta`. #### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The message index to use as a perspective. | #### [](#examples-3)Examples For example, the following map extracts the contents of the JSON field `foo` specifically from message index `1` of a batch, effectively overriding the field `foo` for all messages of a batch to that of message 1: ```bloblang root = this root.foo = json("foo").from(1) ``` ### [](#from_all)from_all Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are `content`, `json` and `meta`. #### [](#examples-4)Examples ```bloblang root = this root.foo_summed = json("foo").from_all().sum() ``` ### [](#map)map Executes a query on the target value, allowing you to transform or extract data from the current context. #### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute on the target. | ### [](#not)not Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true. ### [](#or)or If the result of the target query fails or resolves to `null`, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator `|`. #### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails or resolves to null. | #### [](#examples-5)Examples ```bloblang root.doc.id = this.thing.id.or(uuid_v4()) ``` ## [](#encoding-and-encryption)Encoding and encryption ### [](#compress)compress Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage. #### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The compression algorithm: flate, gzip, pgzip (parallel gzip), lz4, snappy, zlib, or zstd. | | level | integer | Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. | #### [](#examples-6)Examples Compress and encode for safe transmission: ```bloblang root.compressed = content().bytes().compress("gzip").encode("base64") # In: {"message":"hello world I love space"} # Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"} ``` Compare compression ratios across algorithms: ```bloblang root.original_size = content().length() root.gzip_size = content().compress("gzip").length() root.lz4_size = content().compress("lz4").length() # In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. # Out: {"gzip_size":114,"lz4_size":85,"original_size":89} ``` ### [](#decode)decode Decodes an encoded string according to a chosen scheme. #### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The decoding scheme to use. | #### [](#examples-7)Examples ```bloblang root.decoded = this.value.decode("hex").string() # In: {"value":"68656c6c6f20776f726c64"} # Out: {"decoded":"hello world"} ``` ```bloblang root = this.encoded.decode("ascii85") # In: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} # Out: this is totally unstructured data ``` ### [](#decompress)decompress Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm. #### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The decompression algorithm: gzip, pgzip (parallel gzip), zlib, bzip2, flate, snappy, lz4, or zstd. | #### [](#examples-8)Examples Decompress base64-encoded compressed data: ```bloblang root = this.compressed.decode("base64").decompress("gzip") # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: hello world I love space ``` Convert decompressed bytes to string for JSON output: ```bloblang root.message = this.compressed.decode("base64").decompress("gzip").string() # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: {"message":"hello world I love space"} ``` ### [](#decrypt_aes)decrypt_aes Decrypts an AES-encrypted string or byte array. #### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for decryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to decrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-9)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.decrypted = this.value.decode("hex").decrypt_aes("ctr", $key, $vector).string() # In: {"value":"84e9b31ff7400bdf80be7254"} # Out: {"decrypted":"hello world!"} ``` ### [](#encode)encode Encodes a string or byte array according to a chosen scheme. #### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The encoding scheme to use. | #### [](#examples-10)Examples ```bloblang root.encoded = this.value.encode("hex") # In: {"value":"hello world"} # Out: {"encoded":"68656c6c6f20776f726c64"} ``` ```bloblang root.encoded = content().encode("ascii85") # In: this is totally unstructured data # Out: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} ``` ### [](#encrypt_aes)encrypt_aes Encrypts a string or byte array using AES encryption. #### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for encryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to encrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-11)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.encrypted = this.value.encrypt_aes("ctr", $key, $vector).encode("hex") # In: {"value":"hello world!"} # Out: {"encrypted":"84e9b31ff7400bdf80be7254"} ``` ### [](#hash)hash Hashes a string or byte array using a specified algorithm. #### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The hashing algorithm to use. | | key (optional) | string | An optional key to use. | | polynomial | string | An optional polynomial key to use when selecting the crc32 algorithm, otherwise ignored. Options are IEEE (default), Castagnoli and Koopman | #### [](#examples-12)Examples ```bloblang root.h1 = this.value.hash("sha1").encode("hex") root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"} ``` The `crc32` algorithm supports options for the polynomial: ```bloblang root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex") root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"c99465aa","h2":"df373d3c"} ``` ### [](#uuid_v5)uuid_v5 Generates a version 5 UUID from a namespace and name. #### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | ns (optional) | string | An optional namespace name or UUID. It supports the dns, url, oid and x500 predefined namespaces and any valid RFC-9562 UUID. If empty, the nil UUID will be used. | #### [](#examples-13)Examples ```bloblang root.id = "example".uuid_v5() ``` ```bloblang root.id = "example".uuid_v5("x500") ``` ```bloblang root.id = "example".uuid_v5("77f836b7-9f61-46c0-851e-9b6ca3535e69") ``` ## [](#geoip)GeoIP ### [](#geoip_anonymous_ip)geoip_anonymous_ip Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the anonymous IP associated with it. #### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_asn)geoip_asn Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ASN associated with it. #### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_city)geoip_city Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the city associated with it. #### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_connection_type)geoip_connection_type Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the connection type associated with it. #### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_country)geoip_country Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the country associated with it. #### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_domain)geoip_domain Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the domain associated with it. #### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_enterprise)geoip_enterprise Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the enterprise associated with it. #### [](#parameters-20)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_isp)geoip_isp Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ISP associated with it. #### [](#parameters-21)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ## [](#json-web-tokens)JSON web tokens ### [](#parse_jwt_es256)parse_jwt_es256 Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims. #### [](#parameters-22)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES256 secret that was used for signing the token. | #### [](#examples-14)Examples ```bloblang root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w== -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es384)parse_jwt_es384 Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims. #### [](#parameters-23)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES384 secret that was used for signing the token. | #### [](#examples-15)Examples ```bloblang root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY----- MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU 8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA 2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es512)parse_jwt_es512 Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims. #### [](#parameters-24)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES512 secret that was used for signing the token. | #### [](#examples-16)Examples ```bloblang root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY----- MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh 9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9 UeYyTt05zRRWuD+p5bY= -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs256)parse_jwt_hs256 Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims. #### [](#parameters-25)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS256 secret that was used for signing the token. | #### [](#examples-17)Examples ```bloblang root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs384)parse_jwt_hs384 Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims. #### [](#parameters-26)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS384 secret that was used for signing the token. | #### [](#examples-18)Examples ```bloblang root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs512)parse_jwt_hs512 Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims. #### [](#parameters-27)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS512 secret that was used for signing the token. | #### [](#examples-19)Examples ```bloblang root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs256)parse_jwt_rs256 Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims. #### [](#parameters-28)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS256 secret that was used for signing the token. | #### [](#examples-20)Examples ```bloblang root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs384)parse_jwt_rs384 Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims. #### [](#parameters-29)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS384 secret that was used for signing the token. | #### [](#examples-21)Examples ```bloblang root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs512)parse_jwt_rs512 Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims. #### [](#parameters-30)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS512 secret that was used for signing the token. | #### [](#examples-22)Examples ```bloblang root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#sign_jwt_es256)sign_jwt_es256 Hash and sign an object representing JSON Web Token (JWT) claims using ES256. #### [](#parameters-31)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-23)Examples ```bloblang root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.-8LrOdkEiv_44ADWW08lpbq41ZmHCel58NMORPq1q4Dyw0zFhqDVLrRoSvCvuyyvgXAFb9IHfR-9MlJ_2ShA9A"} ``` ```bloblang root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es384)sign_jwt_es384 Hash and sign an object representing JSON Web Token (JWT) claims using ES384. #### [](#parameters-32)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-24)Examples ```bloblang root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.8FmTKH08dl7dyxrNu0rmvhegiIBCy-O9cddGco2e9lpZtgv5mS5qHgPkgBC5eRw1d7SRJsHwHZeehzdqT5Ba7aZJIhz9ds0sn37YQ60L7jT0j2gxCzccrt4kECHnUnLw"} ``` ```bloblang root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es512)sign_jwt_es512 Hash and sign an object representing JSON Web Token (JWT) claims using ES512. #### [](#parameters-33)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-25)Examples ```bloblang root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.AQbEWymoRZxDJEJtKSFFG2k2VbDCTYSuBwAZyMqexCspr3If8aERTVGif8HXG3S7TzMBCCzxkcKr3eIU441l3DlpAMNfQbkcOlBqMvNBn-CX481WyKf3K5rFHQ-6wRonz05aIsWAxCDvAozI_9J0OWllxdQ2MBAuTPbPJ38OqXsYkCQs"} ``` ```bloblang root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs256)sign_jwt_hs256 Hash and sign an object representing JSON Web Token (JWT) claims using HS256. #### [](#parameters-34)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-26)Examples ```bloblang root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.hUl-nngPMY_3h9vveWJUPsCcO5PeL6k9hWLnMYeFbFQ"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs384)sign_jwt_hs384 Hash and sign an object representing JSON Web Token (JWT) claims using HS384. #### [](#parameters-35)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-27)Examples ```bloblang root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zGYLr83aToon1efUNq-hw7XgT20lPvZb8sYei8x6S6mpHwb433SJdXJXx0Oio8AZ"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs512)sign_jwt_hs512 Hash and sign an object representing JSON Web Token (JWT) claims using HS512. #### [](#parameters-36)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-28)Examples ```bloblang root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zBNR9o_6EDwXXKkpKLNJhG26j8Dc-mV-YahBwmEdCrmiWt5les8I9rgmNlWIowpq6Yxs4kLNAdFhqoRz3NXT3w"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs256)sign_jwt_rs256 Hash and sign an object representing JSON Web Token (JWT) claims using RS256. #### [](#parameters-37)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-29)Examples ```bloblang root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs384)sign_jwt_rs384 Hash and sign an object representing JSON Web Token (JWT) claims using RS384. #### [](#parameters-38)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-30)Examples ```bloblang root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs512)sign_jwt_rs512 Hash and sign an object representing JSON Web Token (JWT) claims using RS512. #### [](#parameters-39)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-31)Examples ```bloblang root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ## [](#number-manipulation)Number manipulation ### [](#abs)abs Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value. #### [](#examples-32)Examples ```bloblang root.outs = this.ins.map_each(ele -> ele.abs()) # In: {"ins":[9,-18,1.23,-4.56]} # Out: {"outs":[9,18,1.23,4.56]} ``` ### [](#bitwise_and)bitwise_and Performs a bitwise AND operation between the integer and the specified value. #### [](#parameters-40)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to AND with | #### [](#examples-33)Examples ```bloblang root.new_value = this.value.bitwise_and(6) # In: {"value":12} # Out: {"new_value":4} ``` ```bloblang root.masked = this.flags.bitwise_and(15) # In: {"flags":127} # Out: {"masked":15} ``` ### [](#bitwise_or)bitwise_or Performs a bitwise OR operation between the integer and the specified value. #### [](#parameters-41)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to OR with | #### [](#examples-34)Examples ```bloblang root.new_value = this.value.bitwise_or(6) # In: {"value":12} # Out: {"new_value":14} ``` ```bloblang root.combined = this.flags.bitwise_or(8) # In: {"flags":4} # Out: {"combined":12} ``` ### [](#bitwise_xor)bitwise_xor Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value. #### [](#parameters-42)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to XOR with | #### [](#examples-35)Examples ```bloblang root.new_value = this.value.bitwise_xor(6) # In: {"value":12} # Out: {"new_value":10} ``` ```bloblang root.toggled = this.flags.bitwise_xor(5) # In: {"flags":3} # Out: {"toggled":6} ``` ### [](#ceil)ceil Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-36)Examples ```bloblang root.new_value = this.value.ceil() # In: {"value":5.3} # Out: {"new_value":6} # In: {"value":-5.9} # Out: {"new_value":-5} ``` ```bloblang root.result = this.price.ceil() # In: {"price":19.99} # Out: {"result":20} ``` ### [](#cos)cos Calculates the cosine of a given angle specified in radians. #### [](#examples-37)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).cos() # In: {"value":45} # Out: {"new_value":0.7071067811865476} # In: {"value":0} # Out: {"new_value":1} # In: {"value":180} # Out: {"new_value":-1} ``` ### [](#float32)float32 Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. #### [](#examples-38)Examples ```bloblang root.out = this.in.float32() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674283e-11} ``` ### [](#float64)float64 Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. #### [](#examples-39)Examples ```bloblang root.out = this.in.float64() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674282313423544e-11} ``` ### [](#floor)floor Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-40)Examples ```bloblang root.new_value = this.value.floor() # In: {"value":5.7} # Out: {"new_value":5} # In: {"value":-3.2} # Out: {"new_value":-4} ``` ```bloblang root.whole_seconds = this.duration_seconds.floor() # In: {"duration_seconds":12.345} # Out: {"whole_seconds":12} ``` ### [](#int16)int16 Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-41)Examples ```bloblang root.a = this.a.int16() root.b = this.b.round().int16() root.c = this.c.int16() root.d = this.d.int16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int16() # In: "0xDE" # Out: 222 ``` ### [](#int32)int32 Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-42)Examples ```bloblang root.a = this.a.int32() root.b = this.b.round().int32() root.c = this.c.int32() root.d = this.d.int32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int32() # In: "0xDEAD" # Out: 57005 ``` ### [](#int64)int64 Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-43)Examples ```bloblang root.a = this.a.int64() root.b = this.b.round().int64() root.c = this.c.int64() root.d = this.d.int64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#int8)int8 Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-44)Examples ```bloblang root.a = this.a.int8() root.b = this.b.round().int8() root.c = this.c.int8() root.d = this.d.int8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int8() # In: "0xD" # Out: 13 ``` ### [](#log)log Calculates the natural logarithm (base e) of a number. #### [](#examples-45)Examples ```bloblang root.new_value = this.value.log().round() # In: {"value":1} # Out: {"new_value":0} # In: {"value":2.7183} # Out: {"new_value":1} ``` ```bloblang root.ln_result = this.number.log() # In: {"number":10} # Out: {"ln_result":2.302585092994046} ``` ### [](#log10)log10 Calculates the base-10 logarithm of a number. #### [](#examples-46)Examples ```bloblang root.new_value = this.value.log10() # In: {"value":100} # Out: {"new_value":2} # In: {"value":1000} # Out: {"new_value":3} ``` ```bloblang root.log_value = this.magnitude.log10() # In: {"magnitude":10000} # Out: {"log_value":4} ``` ### [](#max)max Returns the largest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-47)Examples ```bloblang root.biggest = this.values.max() # In: {"values":[0,3,2.5,7,5]} # Out: {"biggest":7} ``` ```bloblang root.highest_temp = this.temperatures.max() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"highest_temp":23.4} ``` ### [](#min)min Returns the smallest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-48)Examples ```bloblang root.smallest = this.values.min() # In: {"values":[0,3,-2.5,7,5]} # Out: {"smallest":-2.5} ``` ```bloblang root.lowest_temp = this.temperatures.min() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"lowest_temp":19.8} ``` ### [](#pow)pow Returns the number raised to the specified exponent. #### [](#parameters-43)Parameters | Name | Type | Description | | --- | --- | --- | | exponent | float | The exponent you want to raise to the power of. | #### [](#examples-49)Examples ```bloblang root.new_value = this.value * 10.pow(-2) # In: {"value":2} # Out: {"new_value":0.02} ``` ```bloblang root.new_value = this.value.pow(-2) # In: {"value":2} # Out: {"new_value":0.25} ``` ### [](#round)round Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-50)Examples ```bloblang root.new_value = this.value.round() # In: {"value":5.3} # Out: {"new_value":5} # In: {"value":5.9} # Out: {"new_value":6} ``` ```bloblang root.rounded = this.score.round() # In: {"score":87.5} # Out: {"rounded":88} ``` ### [](#sin)sin Calculates the sine of a given angle specified in radians. #### [](#examples-51)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).sin() # In: {"value":45} # Out: {"new_value":0.7071067811865475} # In: {"value":0} # Out: {"new_value":0} # In: {"value":90} # Out: {"new_value":1} ``` ### [](#tan)tan Calculates the tangent of a given angle specified in radians. #### [](#examples-52)Examples ```bloblang root.new_value = "%f".format((this.value * (pi() / 180)).tan()) # In: {"value":0} # Out: {"new_value":"0.000000"} # In: {"value":45} # Out: {"new_value":"1.000000"} # In: {"value":180} # Out: {"new_value":"-0.000000"} ``` ### [](#uint16)uint16 Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-53)Examples ```bloblang root.a = this.a.uint16() root.b = this.b.round().uint16() root.c = this.c.uint16() root.d = this.d.uint16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint16() # In: "0xDE" # Out: 222 ``` ### [](#uint32)uint32 Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-54)Examples ```bloblang root.a = this.a.uint32() root.b = this.b.round().uint32() root.c = this.c.uint32() root.d = this.d.uint32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint32() # In: "0xDEAD" # Out: 57005 ``` ### [](#uint64)uint64 Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-55)Examples ```bloblang root.a = this.a.uint64() root.b = this.b.round().uint64() root.c = this.c.uint64() root.d = this.d.uint64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#uint8)uint8 Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-56)Examples ```bloblang root.a = this.a.uint8() root.b = this.b.round().uint8() root.c = this.c.uint8() root.d = this.d.uint8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint8() # In: "0xD" # Out: 13 ``` ## [](#object-array-manipulation)Object & array manipulation ### [](#all)all Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays. #### [](#parameters-44)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-57)Examples ```bloblang root.all_over_21 = this.patrons.all(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"all_over_21":false} # In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]} # Out: {"all_over_21":true} ``` ```bloblang root.all_positive = this.values.all(v -> v > 0) # In: {"values":[1,2,3,4,5]} # Out: {"all_positive":true} # In: {"values":[1,-2,3,4,5]} # Out: {"all_positive":false} ``` ### [](#any)any Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays. #### [](#parameters-45)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-58)Examples ```bloblang root.any_over_21 = this.patrons.any(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"any_over_21":true} # In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]} # Out: {"any_over_21":false} ``` ```bloblang root.has_errors = this.results.any(r -> r.status == "error") # In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]} # Out: {"has_errors":true} # In: {"results":[{"status":"ok"},{"status":"ok"}]} # Out: {"has_errors":false} ``` ### [](#append)append Adds one or more elements to the end of an array and returns the new array. The original array is not modified. #### [](#examples-59)Examples ```bloblang root.foo = this.foo.append("and", "this") # In: {"foo":["bar","baz"]} # Out: {"foo":["bar","baz","and","this"]} ``` ```bloblang root.combined = this.items.append(this.new_item) # In: {"items":["apple","banana"],"new_item":"orange"} # Out: {"combined":["apple","banana","orange"]} ``` ### [](#assign)assign Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead. #### [](#parameters-46)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-60)Examples ```bloblang root = this.foo.assign(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":"foos","second_name":"barer"} ``` Override defaults with user settings: ```bloblang root.config = this.defaults.assign(this.user_settings) # In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}} # Out: {"config":{"retries":3,"timeout":60}} ``` ### [](#collapse)collapse Flattens a nested structure into a flat object with dot-notation keys. #### [](#parameters-47)Parameters | Name | Type | Description | | --- | --- | --- | | include_empty | bool | Whether to include empty objects and arrays in the resulting object. | #### [](#examples-61)Examples ```bloblang root.result = this.collapse() # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}} ``` Set include\_empty to true to preserve empty objects and arrays in the output: ```bloblang root.result = this.collapse(include_empty: true) # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}} ``` ### [](#concat)concat Concatenates an array value with one or more argument arrays. #### [](#examples-62)Examples ```bloblang root.foo = this.foo.concat(this.bar, this.baz) # In: {"foo":["a","b"],"bar":["c"],"baz":["d","e","f"]} # Out: {"foo":["a","b","c","d","e","f"]} ``` ### [](#contains)contains Tests if an array or object contains a value. #### [](#parameters-48)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-63)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} ``` ```bloblang root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} ``` ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} ``` ### [](#diff)diff Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems. #### [](#parameters-49)Parameters | Name | Type | Description | | --- | --- | --- | | other | unknown | The value to compare against the current value. Can be any structured data (object or array). | #### [](#examples-64)Examples Compare two objects to track field changes: ```bloblang root.changes = this.before.diff(this.after) # In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}} # Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]} ``` Detect deletions in configuration changes: ```bloblang root.changelog = this.old_config.diff(this.new_config) # In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}} # Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]} ``` ### [](#enumerated)enumerated Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element. #### [](#examples-65)Examples ```bloblang root.foo = this.foo.enumerated() # In: {"foo":["bar","baz"]} # Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]} ``` Useful for filtering by index position: ```bloblang root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value) # In: {"items":["a","b","c","d"]} # Out: {"first_two":["a","b"]} ``` ### [](#exists)exists Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise. #### [](#parameters-50)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field. | #### [](#examples-66)Examples ```bloblang root.result = this.foo.exists("bar.baz") # In: {"foo":{"bar":{"baz":"yep, I exist"}}} # Out: {"result":true} # In: {"foo":{"bar":{}}} # Out: {"result":false} # In: {"foo":{}} # Out: {"result":false} ``` Also returns true for null values if the field exists: ```bloblang root.has_field = this.data.exists("optional_field") # In: {"data":{"optional_field":null}} # Out: {"has_field":true} # In: {"data":{}} # Out: {"has_field":false} ``` ### [](#explode)explode Expands a nested field into multiple documents. #### [](#parameters-51)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field to explode. | #### [](#examples-67)Examples ##### [](#on-arrays)On arrays When exploding an array, each element becomes a separate document with the array element replacing the original field: ```bloblang root = this.explode("value") # In: {"id":1,"value":["foo","bar","baz"]} # Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}] ``` ##### [](#on-objects)On objects When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value: ```bloblang root = this.explode("value") # In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}} # Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}} ``` ### [](#filter)filter Filters array or object elements based on a condition. #### [](#parameters-52)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A query to apply to each element, if this query resolves to any value other than a boolean true the element will be removed from the result. | #### [](#examples-68)Examples ```bloblang root.new_nums = this.nums.filter(num -> num > 10) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[11,17]} ``` ##### [](#on-objects-2)On objects When filtering objects, the query receives a context with `key` and `value` fields for each entry: ```bloblang root.new_dict = this.dict.filter(item -> item.value.contains("foo")) # In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}} # Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}} ``` ### [](#find)find Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation. #### [](#parameters-53)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-69)Examples ```bloblang root.index = this.find("bar") # In: ["foo", "bar", "baz"] # Out: {"index":1} ``` ```bloblang root.index = this.things.find(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz"]} # Out: {"index":1} ``` ### [](#find_all)find_all Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation. #### [](#parameters-54)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-70)Examples ```bloblang root.index = this.find_all("bar") # In: ["foo", "bar", "baz", "bar"] # Out: {"index":[1,3]} ``` ```bloblang root.indexes = this.things.find_all(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz", "bar", "buz"]} # Out: {"indexes":[1,3]} ``` ### [](#find_all_by)find_all_by Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match. #### [](#parameters-55)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-71)Examples ```bloblang root.index = this.find_all_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":[0,2]} ``` Find all indexes matching criteria: ```bloblang root.error_indexes = this.logs.find_all_by(log -> log.level == "error") # In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]} # Out: {"error_indexes":[1,3]} ``` ### [](#find_by)find_by Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query. #### [](#parameters-56)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-72)Examples ```bloblang root.index = this.find_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":0} ``` Find first object matching criteria: ```bloblang root.first_adult = this.users.find_by(u -> u.age >= 18) # In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]} # Out: {"first_adult":1} ``` ### [](#flatten)flatten Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed. #### [](#examples-73)Examples ```bloblang root.result = this.flatten() # In: ["foo",["bar","baz"],"buz"] # Out: {"result":["foo","bar","baz","buz"]} ``` Deeper nesting requires multiple flatten calls: ```bloblang root.result = this.data.flatten() # In: {"data":["a",["b",["c","d"]],"e"]} # Out: {"result":["a","b",["c","d"],"e"]} ``` ### [](#fold)fold Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration. #### [](#parameters-57)Parameters | Name | Type | Description | | --- | --- | --- | | initial | unknown | The initial value to start the fold with. For example, an empty object {}, a zero count 0, or an empty string "". | | query | query expression | A query to apply for each element. The query is provided an object with two fields; tally containing the current tally, and value containing the value of the current element. The query should result in a new tally to be passed to the next element query. | #### [](#examples-74)Examples Sum numbers in an array: ```bloblang root.sum = this.foo.fold(0, item -> item.tally + item.value) # In: {"foo":[3,8,11]} # Out: {"sum":22} ``` Concatenate strings: ```bloblang root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value)) # In: {"foo":["hello ", "world"]} # Out: {"result":"hello world"} ``` Merge an array of objects into a single object: ```bloblang root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value)) # In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]} # Out: {"smoothie":{"apple":5,"banana":3,"orange":8}} ``` ### [](#get)get Extract a field value, identified via a [dot path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/), from an object. #### [](#parameters-58)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path identifying a field to obtain. | #### [](#examples-75)Examples ```bloblang root.result = this.foo.get(this.target) # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"bar"} # Out: {"result":"from bar"} # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"baz"} # Out: {"result":"from baz"} ``` ### [](#index)index Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on. #### [](#parameters-59)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The index to obtain from an array. | #### [](#examples-76)Examples ```bloblang root.last_name = this.names.index(-1) # In: {"names":["rachel","stevens"]} # Out: {"last_name":"stevens"} ``` It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer: ```bloblang root.last_byte = this.name.bytes().index(-1) # In: {"name":"foobar bazson"} # Out: {"last_byte":110} ``` ### [](#join)join Joins an array of strings with an optional delimiter. #### [](#parameters-60)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter (optional) | string | An optional delimiter to add between each string. | #### [](#examples-77)Examples ```bloblang root.joined_words = this.words.join() root.joined_numbers = this.numbers.map_each(this.string()).join(",") # In: {"words":["hello","world"],"numbers":[3,8,11]} # Out: {"joined_numbers":"3,8,11","joined_words":"helloworld"} ``` ### [](#json_path)json_path Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/). For more complex logic, you can use Gval expressions ([https://github.com/PaesslerAG/gval](https://github.com/PaesslerAG/gval)). #### [](#parameters-61)Parameters | Name | Type | Description | | --- | --- | --- | | expression | string | The JSONPath expression to execute. | #### [](#examples-78)Examples ```bloblang root.all_names = this.json_path("$..name") # In: {"name":"alice","foo":{"name":"bob"}} # Out: {"all_names":["alice","bob"]} # In: {"thing":["this","bar",{"name":"alice"}]} # Out: {"all_names":["alice"]} ``` ```bloblang root.text_objects = this.json_path("$.body[?(@.type=='text')]") # In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]} # Out: {"text_objects":[{"id":"bar","type":"text"}]} ``` ### [](#json_schema)json_schema Checks a [JSON schema](https://json-schema.org/) against a value and returns the value if it matches or throws and error if it does not. #### [](#parameters-62)Parameters | Name | Type | Description | | --- | --- | --- | | schema | string | The schema to check values against. | #### [](#examples-79)Examples ```bloblang root = this.json_schema("""{ "type":"object", "properties":{ "foo":{ "type":"string" } } }""") # In: {"foo":"bar"} # Out: {"foo":"bar"} # In: {"foo":5} # Out: Error("failed assignment (line 1): field `this`: foo invalid type. expected: string, given: integer") ``` In order to load a schema from a file use the `file` function: ```bloblang root = this.json_schema(file(env("BENTHOS_TEST_BLOBLANG_SCHEMA_FILE"))) ``` ### [](#key_values)key_values Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted. #### [](#examples-80)Examples ```bloblang root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key) # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]} ``` Filter object entries by value: ```bloblang root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key) # In: {"items":{"a":5,"b":15,"c":20,"d":3}} # Out: {"large_items":["c"]} ``` ### [](#keys)keys Extracts all keys from an object and returns them as a sorted array. #### [](#examples-81)Examples ```bloblang root.foo_keys = this.foo.keys() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_keys":["bar","baz"]} ``` Check if specific keys exist: ```bloblang root.has_id = this.data.keys().contains("id") # In: {"data":{"id":123,"name":"test"}} # Out: {"has_id":true} ``` ### [](#length)length Returns the length of an array, object, or string. #### [](#examples-82)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} ``` ```bloblang root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} ``` ### [](#map_each)map_each Applies a mapping to each element of an array or object. #### [](#parameters-63)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each element. | #### [](#examples-83)Examples ##### [](#on-arrays-2)On arrays Transforms each array element using a query. Return deleted() to remove an element, or the new value to replace it: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ##### [](#on-objects-3)On objects Transforms each object value using a query. The query receives an object with 'key' and 'value' fields for each entry: ```bloblang root.new_dict = this.dict.map_each(item -> item.value.uppercase()) # In: {"dict":{"foo":"hello","bar":"world"}} # Out: {"new_dict":{"bar":"WORLD","foo":"HELLO"}} ``` ### [](#map_each_key)map_each_key Transforms object keys using a mapping query. #### [](#parameters-64)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each key. | #### [](#examples-84)Examples ```bloblang root.new_dict = this.dict.map_each_key(key -> key.uppercase()) # In: {"dict":{"keya":"hello","keyb":"world"}} # Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}} ``` Conditionally transform keys: ```bloblang root = this.map_each_key(key -> if key.contains("kafka") { "_" + key }) # In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"} # Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"} ``` ### [](#merge)merge Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method. #### [](#parameters-65)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-85)Examples ```bloblang root = this.foo.merge(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"} ``` Merge arrays: ```bloblang root.combined = this.list1.merge(this.list2) # In: {"list1":["a","b"],"list2":["c","d"]} # Out: {"combined":["a","b","c","d"]} ``` ### [](#patch)patch Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state. #### [](#parameters-66)Parameters | Name | Type | Description | | --- | --- | --- | | changelog | unknown | The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. | #### [](#examples-86)Examples Apply recorded changes to update an object: ```bloblang root.updated = this.current.patch(this.changelog) # In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]} # Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}} ``` Restore previous state by applying inverse changes: ```bloblang root.restored = this.modified.patch(this.reverse_changelog) # In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]} # Out: {"restored":{"debug":true,"timeout":30}} ``` ### [](#slice)slice Extracts a portion of an array or string. #### [](#parameters-67)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-87)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} ``` ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} ``` ### [](#sort)sort Sorts array elements in ascending order. #### [](#parameters-68)Parameters | Name | Type | Description | | --- | --- | --- | | compare (optional) | query expression | An optional query that should explicitly compare elements left and right and provide a boolean result. | #### [](#examples-88)Examples ```bloblang root.sorted = this.foo.sort() # In: {"foo":["bbb","ccc","aaa"]} # Out: {"sorted":["aaa","bbb","ccc"]} ``` Custom comparison for complex objects - return true if left < right: ```bloblang root.sorted = this.foo.sort(item -> item.left.v < item.right.v) # In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]} # Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]} ``` ### [](#sort_by)sort_by Sorts array elements by a specified field or expression. #### [](#parameters-69)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to apply to each element that yields a value used for sorting. | #### [](#examples-89)Examples ```bloblang root.sorted = this.foo.sort_by(ele -> ele.id) # In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]} # Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]} ``` Sort by numeric field: ```bloblang root.sorted = this.items.sort_by(item -> item.priority) # In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]} # Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]} ``` ### [](#squash)squash Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the `.merge()` method). #### [](#examples-90)Examples ```bloblang root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash() # In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]} # Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}} ``` ### [](#sum)sum Returns the sum of numeric values in an array. #### [](#examples-91)Examples ```bloblang root.sum = this.foo.sum() # In: {"foo":[3,8,4]} # Out: {"sum":15} ``` Works with decimals: ```bloblang root.total = this.prices.sum() # In: {"prices":[10.5,20.25,5.00]} # Out: {"total":35.75} ``` ### [](#unique)unique Returns an array with duplicate elements removed. #### [](#parameters-70)Parameters | Name | Type | Description | | --- | --- | --- | | emit (optional) | query expression | An optional query that can be used in order to yield a value for each element to determine uniqueness. | #### [](#examples-92)Examples ```bloblang root.uniques = this.foo.unique() # In: {"foo":["a","b","a","c"]} # Out: {"uniques":["a","b","c"]} ``` Use a query to determine uniqueness by a field: ```bloblang root.unique_users = this.users.unique(u -> u.id) # In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]} # Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]} ``` ### [](#values)values Returns an array of all values from an object. #### [](#examples-93)Examples ```bloblang root.foo_vals = this.foo.values().sort() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_vals":[1,2]} ``` Find max value in object: ```bloblang root.max = this.scores.values().sort().index(-1) # In: {"scores":{"player1":85,"player2":92,"player3":78}} # Out: {"max":92} ``` ### [](#with)with Returns an object where all but one or more [field path](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/field_paths/) arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields. If a key within a nested path does not exist then it is ignored. #### [](#examples-94)Examples ```bloblang root = this.with("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"d":"fourth","inner":{"a":"first","c":"third"}} ``` ### [](#without)without Returns an object with specified keys removed. #### [](#examples-95)Examples ```bloblang root = this.without("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"e":"fifth","inner":{"b":"second"}} ``` Remove sensitive fields: ```bloblang root = this.without("password","ssn","creditCard") # In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"} # Out: {"email":"alice@example.com","username":"alice"} ``` ### [](#zip)zip Zip an array value with one or more argument arrays. Each array must match in length. #### [](#examples-96)Examples ```bloblang root.foo = this.foo.zip(this.bar, this.baz) # In: {"foo":["a","b","c"],"bar":[1,2,3],"baz":[4,5,6]} # Out: {"foo":[["a",1,4],["b",2,5],["c",3,6]]} ``` ## [](#parsing)Parsing ### [](#bloblang)bloblang Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as `file` and `env`, or that access message information directly, such as `content` or `json`, are not enabled for dynamic Bloblang mappings. #### [](#parameters-71)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to execute. | #### [](#examples-97)Examples ```bloblang root.body = this.body.bloblang(this.mapping) # In: {"body":{"foo":"hello world"},"mapping":"root.foo = this.foo.uppercase()"} # Out: {"body":{"foo":"HELLO WORLD"}} # In: {"body":{"foo":"hello world 2"},"mapping":"root.foo = this.foo.capitalize()"} # Out: {"body":{"foo":"Hello World 2"}} ``` ### [](#format_json)format_json Formats a value as a JSON string. #### [](#parameters-72)Parameters | Name | Type | Description | | --- | --- | --- | | indent | string | Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. | | no_indent | bool | Disable indentation. | | escape_html | bool | Escape problematic HTML characters. | #### [](#examples-98)Examples ```bloblang root = this.doc.format_json() # In: {"doc":{"foo":"bar"}} # Out: { "foo": "bar" } ``` Pass a string to the `indent` parameter in order to customise the indentation: ```bloblang root = this.format_json(" ") # In: {"doc":{"foo":"bar"}} # Out: { "doc": { "foo": "bar" } } ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_json().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"{\n \"foo\": \"bar\"\n}"} ``` Set the `no_indent` parameter to true to disable indentation. The result is equivalent to calling `bytes()`: ```bloblang root = this.doc.format_json(no_indent: true) # In: {"doc":{"foo":"bar"}} # Out: {"foo":"bar"} ``` Escapes problematic HTML characters: ```bloblang root = this.doc.format_json() # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo\u0026bar@benthos.dev", "name": "foo\u003ebar" } ``` Set the `escape_html` parameter to false to disable escaping of problematic HTML characters: ```bloblang root = this.doc.format_json(escape_html: false) # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo&bar@benthos.dev", "name": "foo>bar" } ``` ### [](#format_msgpack)format_msgpack Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed. #### [](#examples-99)Examples Serialize object to MessagePack and encode as hex for transmission: ```bloblang root = this.format_msgpack().encode("hex") # In: {"foo":"bar"} # Out: 81a3666f6fa3626172 ``` Serialize data to MessagePack and base64 encode for embedding in JSON: ```bloblang root.msgpack_payload = this.data.format_msgpack().encode("base64") # In: {"data":{"foo":"bar"}} # Out: {"msgpack_payload":"gaNmb2+jYmFy"} ``` ### [](#format_xml)format_xml Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array. #### [](#parameters-73)Parameters | Name | Type | Description | | --- | --- | --- | | indent | string | String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. | | no_indent | bool | Disable indentation and newlines to produce compact XML on a single line. | | root_tag (optional) | string | Custom name for the root XML element. By default, the root element name is derived from the first key in the object. | #### [](#examples-100)Examples Serialize object to pretty-printed XML with default indentation: ```bloblang root = this.format_xml() # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` Create compact XML without indentation for smaller message size: ```bloblang root = this.format_xml(no_indent: true) # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` ### [](#format_yaml)format_yaml Formats a value as a YAML string. #### [](#examples-101)Examples ```bloblang root = this.doc.format_yaml() # In: {"doc":{"foo":"bar"}} # Out: foo: bar ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_yaml().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"foo: bar\n"} ``` ### [](#infer_schema)infer_schema Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. ### [](#parse_csv)parse_csv Parses CSV data into an array. #### [](#parameters-74)Parameters | Name | Type | Description | | --- | --- | --- | | parse_header_row | bool | Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. | | delimiter | string | The delimiter to use for splitting values in each record. It must be a single character. | | lazy_quotes | bool | If set to true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. | #### [](#examples-102)Examples Parses CSV data with a header row: ```bloblang root.orders = this.orders.parse_csv() # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data without a header row: ```bloblang root.orders = this.orders.parse_csv(false) # In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]} ``` Parses CSV data delimited by dots: ```bloblang root.orders = this.orders.parse_csv(delimiter:".") # In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data containing a quote in an unquoted field: ```bloblang root.orders = this.orders.parse_csv(lazy_quotes:true) # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]} ``` ### [](#parse_form_url_encoded)parse_form_url_encoded Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result. #### [](#examples-103)Examples ```bloblang root.values = this.body.parse_form_url_encoded() # In: {"body":"noise=meow&animal=cat&fur=orange&fur=fluffy"} # Out: {"values":{"animal":"cat","fur":["orange","fluffy"],"noise":"meow"}} ``` ### [](#parse_json)parse_json Parses a JSON string into a structured value. #### [](#parameters-75)Parameters | Name | Type | Description | | --- | --- | --- | | use_number (optional) | bool | An optional flag that when set makes parsing numbers as json.Number instead of the default float64. | #### [](#examples-104)Examples ```bloblang root.doc = this.doc.parse_json() # In: {"doc":"{\"foo\":\"bar\"}"} # Out: {"doc":{"foo":"bar"}} ``` ```bloblang root.doc = this.doc.parse_json(use_number: true) # In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"} # Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}} ``` ### [](#parse_logfmt)parse_logfmt Parses logfmt formatted data into an object. #### [](#examples-105)Examples ```bloblang root = this.msg.parse_logfmt() # In: {"msg":"level=info msg=\"hello world\" dur=1.5s"} # Out: {"dur":"1.5s","level":"info","msg":"hello world"} ``` ### [](#parse_msgpack)parse_msgpack Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices. #### [](#examples-106)Examples Parse MessagePack data from hex-encoded content: ```bloblang root = content().decode("hex").parse_msgpack() # In: 81a3666f6fa3626172 # Out: {"foo":"bar"} ``` Parse MessagePack from base64-encoded field: ```bloblang root.decoded = this.msgpack_data.decode("base64").parse_msgpack() # In: {"msgpack_data":"gaNmb2+jYmFy"} # Out: {"decoded":{"foo":"bar"}} ``` ### [](#parse_parquet)parse_parquet Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array. #### [](#parameters-76)Parameters | Name | Type | Description | | --- | --- | --- | | byte_array_as_string | bool | Deprecated: This parameter is no longer used. | #### [](#examples-107)Examples Parse Parquet file data into structured objects: ```bloblang root.records = content().parse_parquet() ``` Process Parquet data from a field and extract specific columns: ```bloblang root.users = this.parquet_data.parse_parquet().map_each(row -> {"name": row.name, "email": row.email}) ``` ### [](#parse_url)parse_url Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow [https://pkg.go.dev/net/url#URL](https://pkg.go.dev/net/url#URL), and may be expanded in future in order to present more information. #### [](#examples-108)Examples ```bloblang root.foo_url = this.foo_url.parse_url() # In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"} # Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}} ``` ```bloblang root.username = this.url.parse_url().user.name | "unknown" # In: {"url":"amqp://foo:bar@127.0.0.1:5672/"} # Out: {"username":"foo"} # In: {"url":"redis://localhost:6379"} # Out: {"username":"unknown"} ``` ### [](#parse_xml)parse_xml Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules: - Element attributes are prefixed with a hyphen (e.g., `-id` for an `id` attribute) - Elements with both attributes and text content store the text in a `#text` field - Repeated elements become arrays - XML comments, directives, and processing instructions are ignored - Optionally cast numeric and boolean strings to their proper types. #### [](#parameters-77)Parameters | Name | Type | Description | | --- | --- | --- | | cast (optional) | bool | Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. | #### [](#examples-109)Examples Parse XML document into object structure: ```bloblang root.doc = this.doc.parse_xml() # In: {"doc":"This is a titleThis is some content"} # Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}} ``` Parse XML with type casting enabled to convert strings to numbers and booleans: ```bloblang root.doc = this.doc.parse_xml(cast: true) # In: {"doc":"This is a title123True"} # Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}} ``` ### [](#parse_yaml)parse_yaml Parses a YAML string into a structured value. #### [](#examples-110)Examples ```bloblang root.doc = this.doc.parse_yaml() # In: {"doc":"foo: bar"} # Out: {"doc":{"foo":"bar"}} ``` ## [](#regular-expressions)Regular expressions ### [](#re_find_all)re_find_all Finds all matches of a regular expression in a string. #### [](#parameters-78)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-111)Examples ```bloblang root.matches = this.value.re_find_all("a.") # In: {"value":"paranormal"} # Out: {"matches":["ar","an","al"]} ``` ```bloblang root.numbers = this.text.re_find_all("[0-9]+") # In: {"text":"I have 2 apples and 15 oranges"} # Out: {"numbers":["2","15"]} ``` ### [](#re_find_all_object)re_find_all_object Finds all regex matches as objects with named groups. #### [](#parameters-79)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-112)Examples ```bloblang root.matches = this.value.re_find_all_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]} ``` ```bloblang root.matches = this.value.re_find_all_object("(?m)(?P\\w+):\\s+(?P\\w+)$") # In: {"value":"option1: value1\noption2: value2\noption3: value3"} # Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]} ``` ### [](#re_find_all_submatch)re_find_all_submatch Finds all regex matches with capture groups. #### [](#parameters-80)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-113)Examples ```bloblang root.matches = this.value.re_find_all_submatch("a(x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[["axxb","xx"],["ab",""]]} ``` ```bloblang root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)") # In: {"text":"Contact: alice@example.com or bob@test.org"} # Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]} ``` ### [](#re_find_object)re_find_object Finds the first regex match as an object with named groups. #### [](#parameters-81)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-114)Examples ```bloblang root.matches = this.value.re_find_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":{"0":"axxb","foo":"xx"}} ``` ```bloblang root.matches = this.value.re_find_object("(?P\\w+):\\s+(?P\\w+)") # In: {"value":"option1: value1"} # Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}} ``` ### [](#re_match)re_match Tests if a string matches a regular expression. #### [](#parameters-82)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-115)Examples ```bloblang root.matches = this.value.re_match("[0-9]") # In: {"value":"there are 10 puppies"} # Out: {"matches":true} # In: {"value":"there are ten puppies"} # Out: {"matches":false} ``` ### [](#re_replace)re_replace Replaces all regex matches with a replacement string that can reference capture groups using `$1`, `$2`, etc. Use for pattern-based transformations or data reformatting. #### [](#parameters-83)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | ### [](#re_replace_all)re_replace_all Replaces all regex matches with a replacement string. #### [](#parameters-84)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | #### [](#examples-116)Examples ```bloblang root.new_value = this.value.re_replace_all("ADD ([0-9]+)","+($1)") # In: {"value":"foo ADD 70"} # Out: {"new_value":"foo +(70)"} ``` ```bloblang root.masked = this.email.re_replace_all("(\\w{2})\\w+@", "$1***@") # In: {"email":"alice@example.com"} # Out: {"masked":"al***@example.com"} ``` ## [](#sql)SQL ### [](#vector)vector Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database. #### [](#examples-117)Examples Convert embeddings array to vector for pgvector storage: ```bloblang root.embedding = this.embeddings.vector() root.text = this.text ``` Process ML model output into database-ready vector format: ```bloblang root.doc_id = this.id root.vector_embedding = this.model_output.map_each(num -> num.number()).vector() ``` ## [](#string-manipulation)String manipulation ### [](#capitalize)capitalize Converts a string to title case with Unicode letter mapping. #### [](#examples-118)Examples ```bloblang root.title = this.title.capitalize() # In: {"title":"the foo bar"} # Out: {"title":"The Foo Bar"} ``` ```bloblang root.name = this.name.capitalize() # In: {"name":"alice smith"} # Out: {"name":"Alice Smith"} ``` ### [](#compare_argon2)compare_argon2 Checks whether a string matches a hashed secret using Argon2. #### [](#parameters-85)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. | #### [](#examples-119)Examples ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#compare_bcrypt)compare_bcrypt Checks whether a string matches a hashed secret using bcrypt. #### [](#parameters-86)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret value to compare with the input. | #### [](#examples-120)Examples ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#contains-2)contains Tests if an array or object contains a value. #### [](#parameters-87)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-121)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} ``` ```bloblang root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} ``` ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} ``` ### [](#escape_html)escape_html Escapes HTML special characters. #### [](#examples-122)Examples ```bloblang root.escaped = this.value.escape_html() # In: {"value":"foo & bar"} # Out: {"escaped":"foo & bar"} ``` ```bloblang root.safe_html = this.user_input.escape_html() # In: {"user_input":""} # Out: {"safe_html":"<script>alert('xss')</script>"} ``` ### [](#escape_url_path)escape_url_path Escapes a string for use in URL paths. #### [](#examples-123)Examples ```bloblang root.escaped = this.value.escape_url_path() # In: {"value":"foo & bar"} # Out: {"escaped":"foo%20&%20bar"} ``` ```bloblang root.url = "https://example.com/docs/" + this.path.escape_url_path() # In: {"path":"my document.pdf"} # Out: {"url":"https://example.com/docs/my%20document.pdf"} ``` ### [](#escape_url_query)escape_url_query Escapes a string for use in URL query parameters. #### [](#examples-124)Examples ```bloblang root.escaped = this.value.escape_url_query() # In: {"value":"foo & bar"} # Out: {"escaped":"foo+%26+bar"} ``` ```bloblang root.url = "https://example.com?search=" + this.query.escape_url_query() # In: {"query":"hello world!"} # Out: {"url":"https://example.com?search=hello+world%21"} ``` ### [](#filepath_join)filepath_join Joins filepath components into a single path. #### [](#examples-125)Examples ```bloblang root.path = this.path_elements.filepath_join() # In: {"path_elements":["/foo/","bar.txt"]} # Out: {"path":"/foo/bar.txt"} ``` ### [](#filepath_split)filepath_split Splits a filepath into directory and filename components. #### [](#examples-126)Examples ```bloblang root.path_sep = this.path.filepath_split() # In: {"path":"/foo/bar.txt"} # Out: {"path_sep":["/foo/","bar.txt"]} # In: {"path":"baz.txt"} # Out: {"path_sep":["","baz.txt"]} ``` ### [](#format)format Formats a value using a specified format string. #### [](#examples-127)Examples ```bloblang root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers) # In: {"name":"lance","age":37,"fingers":13} # Out: {"foo":"lance(37): 13"} ``` ```bloblang root.message = "User %s has %v points".format(this.username, this.score) # In: {"username":"alice","score":100} # Out: {"message":"User alice has 100 points"} ``` ### [](#has_prefix)has_prefix Tests if a string starts with a specified prefix. #### [](#parameters-88)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-128)Examples ```bloblang root.t1 = this.v1.has_prefix("foo") root.t2 = this.v2.has_prefix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":true,"t2":false} ``` ### [](#has_suffix)has_suffix Tests if a string ends with a specified suffix. #### [](#parameters-89)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-129)Examples ```bloblang root.t1 = this.v1.has_suffix("foo") root.t2 = this.v2.has_suffix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":false,"t2":true} ``` ### [](#index_of)index_of Returns the index of the first occurrence of a substring. #### [](#parameters-90)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | A string to search for. | #### [](#examples-130)Examples ```bloblang root.index = this.thing.index_of("bar") # In: {"thing":"foobar"} # Out: {"index":3} ``` ```bloblang root.index = content().index_of("meow") # In: the cat meowed, the dog woofed # Out: {"index":8} ``` ### [](#length-2)length Returns the length of an array, object, or string. #### [](#examples-131)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} ``` ```bloblang root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} ``` ### [](#lowercase)lowercase Converts all letters in a string to lowercase. #### [](#examples-132)Examples ```bloblang root.foo = this.foo.lowercase() # In: {"foo":"HELLO WORLD"} # Out: {"foo":"hello world"} ``` ```bloblang root.email = this.user_email.lowercase() # In: {"user_email":"User@Example.COM"} # Out: {"email":"user@example.com"} ``` ### [](#quote)quote Wraps a string in double quotes and escapes special characters. #### [](#examples-133)Examples ```bloblang root.quoted = this.thing.quote() # In: {"thing":"foo\nbar"} # Out: {"quoted":"\"foo\\nbar\""} ``` ```bloblang root.literal = this.text.quote() # In: {"text":"hello\tworld"} # Out: {"literal":"\"hello\\tworld\""} ``` ### [](#repeat)repeat Creates a string by repeating the input a specified number of times. #### [](#parameters-91)Parameters | Name | Type | Description | | --- | --- | --- | | count | integer | The number of times to repeat the string. | #### [](#examples-134)Examples ```bloblang root.repeated = this.name.repeat(3) root.not_repeated = this.name.repeat(0) # In: {"name":"bob"} # Out: {"not_repeated":"","repeated":"bobbobbob"} ``` ```bloblang root.separator = "-".repeat(10) # In: {} # Out: {"separator":"----------"} ``` ### [](#replace)replace Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings. #### [](#parameters-92)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | ### [](#replace_all)replace_all Replaces all occurrences of a substring with another. #### [](#parameters-93)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | #### [](#examples-135)Examples ```bloblang root.new_value = this.value.replace_all("foo","dog") # In: {"value":"The foo ate my homework"} # Out: {"new_value":"The dog ate my homework"} ``` ```bloblang root.clean = this.text.replace_all(" ", " ") # In: {"text":"hello world foo"} # Out: {"clean":"hello world foo"} ``` ### [](#replace_all_many)replace_all_many Performs multiple find-and-replace operations in sequence. #### [](#parameters-94)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | #### [](#examples-136)Examples ```bloblang root.new_value = this.value.replace_all_many([ "", "<b>", "", "</b>", "", "<i>", "", "</i>", ]) # In: {"value":"Hello World"} # Out: {"new_value":"<i>Hello</i> <b>World</b>"} ``` ### [](#replace_many)replace_many Performs multiple find-and-replace operations in sequence using an array of `[old, new]` pairs. More efficient than chaining multiple `replace_all` calls. Use for bulk text transformations. #### [](#parameters-95)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | ### [](#reverse)reverse Reverses the order of characters in a string. #### [](#examples-137)Examples ```bloblang root.reversed = this.thing.reverse() # In: {"thing":"backwards"} # Out: {"reversed":"sdrawkcab"} ``` ```bloblang root = content().reverse() # In: {"thing":"backwards"} # Out: }"sdrawkcab":"gniht"{ ``` ### [](#slice-2)slice Extracts a portion of an array or string. #### [](#parameters-96)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-138)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} ``` ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} ``` ### [](#slug)slug Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters. #### [](#parameters-97)Parameters | Name | Type | Description | | --- | --- | --- | | lang (optional) | string | | #### [](#examples-139)Examples Create a URL-friendly slug from a string with special characters: ```bloblang root.slug = this.title.slug() # In: {"title":"Hello World! Welcome to Redpanda Connect"} # Out: {"slug":"hello-world-welcome-to-redpanda-connect"} ``` Create a slug preserving French language rules: ```bloblang root.slug = this.title.slug("fr") # In: {"title":"Café & Restaurant"} # Out: {"slug":"cafe-et-restaurant"} ``` ### [](#split)split Splits a string into an array of substrings. #### [](#parameters-98)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter | string | The delimiter to split with. | | empty_as_null | bool | To treat empty substrings as null values | #### [](#examples-140)Examples ```bloblang root.new_value = this.value.split(",") # In: {"value":"foo,bar,baz"} # Out: {"new_value":["foo","bar","baz"]} ``` ```bloblang root.new_value = this.value.split(",", true) # In: {"value":"foo,,qux"} # Out: {"new_value":["foo",null,"qux"]} ``` ```bloblang root.words = this.sentence.split(" ") # In: {"sentence":"hello world from bloblang"} # Out: {"words":["hello","world","from","bloblang"]} ``` ### [](#strip_html)strip_html Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others. #### [](#parameters-99)Parameters | Name | Type | Description | | --- | --- | --- | | preserve (optional) | unknown | Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. | #### [](#examples-141)Examples Extract plain text from HTML content: ```bloblang root.plain_text = this.html_content.strip_html() # In: {"html_content":"

Welcome to Redpanda Connect!

"} # Out: {"plain_text":"Welcome to Redpanda Connect!"} ``` Preserve specific HTML elements while removing others: ```bloblang root.sanitized = this.html.strip_html(["strong", "em"]) # In: {"html":"

Some bold and italic text with a

"} # Out: {"sanitized":"Some bold and italic text with a "} ``` ### [](#trim)trim Removes leading and trailing characters from a string. #### [](#parameters-100)Parameters | Name | Type | Description | | --- | --- | --- | | cutset (optional) | string | An optional string of characters to trim from the target value. | #### [](#examples-142)Examples ```bloblang root.title = this.title.trim("!?") root.description = this.description.trim() # In: {"description":" something happened and its amazing! ","title":"!!!watch out!?"} # Out: {"description":"something happened and its amazing!","title":"watch out"} ``` ### [](#trim_prefix)trim_prefix Removes a specified prefix from the beginning of a string. #### [](#parameters-101)Parameters | Name | Type | Description | | --- | --- | --- | | prefix | string | The leading prefix substring to trim from the string. | #### [](#examples-143)Examples ```bloblang root.name = this.name.trim_prefix("foobar_") root.description = this.description.trim_prefix("foobar_") # In: {"description":"unchanged","name":"foobar_blobton"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#trim_suffix)trim_suffix Removes a specified suffix from the end of a string. #### [](#parameters-102)Parameters | Name | Type | Description | | --- | --- | --- | | suffix | string | The trailing suffix substring to trim from the string. | #### [](#examples-144)Examples ```bloblang root.name = this.name.trim_suffix("_foobar") root.description = this.description.trim_suffix("_foobar") # In: {"description":"unchanged","name":"blobton_foobar"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#unescape_html)unescape_html Converts HTML entities back to their original characters. #### [](#examples-145)Examples ```bloblang root.unescaped = this.value.unescape_html() # In: {"value":"foo & bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.text = this.html.unescape_html() # In: {"html":"<p>Hello & goodbye</p>"} # Out: {"text":"

Hello & goodbye

"} ``` ### [](#unescape_url_path)unescape_url_path Unescapes URL path encoding. #### [](#examples-146)Examples ```bloblang root.unescaped = this.value.unescape_url_path() # In: {"value":"foo%20&%20bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.filename = this.path.unescape_url_path() # In: {"path":"my%20document.pdf"} # Out: {"filename":"my document.pdf"} ``` ### [](#unescape_url_query)unescape_url_query Unescapes URL query parameter encoding. #### [](#examples-147)Examples ```bloblang root.unescaped = this.value.unescape_url_query() # In: {"value":"foo+%26+bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.search = this.param.unescape_url_query() # In: {"param":"hello+world%21"} # Out: {"search":"hello world!"} ``` ### [](#unicode_segments)unicode_segments Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners. #### [](#parameters-103)Parameters | Name | Type | Description | | --- | --- | --- | | segmentation_type | string | Type of segmentation: "grapheme", "word", or "sentence" | #### [](#examples-148)Examples Split text into sentences (preserves trailing spaces): ```bloblang root.sentences = this.text.unicode_segments("sentence") # In: {"text":"Hello world. How are you?"} # Out: {"sentences":["Hello world. ","How are you?"]} ``` Split text into grapheme clusters (handles complex emoji correctly): ```bloblang root.graphemes = this.emoji.unicode_segments("grapheme") # In: {"emoji":"👨‍👩‍👧‍👦❤️"} # Out: {"graphemes":["👨‍👩‍👧‍👦","❤️"]} ``` ### [](#unquote)unquote Removes surrounding quotes and interprets escape sequences. #### [](#examples-149)Examples ```bloblang root.unquoted = this.thing.unquote() # In: {"thing":"\"foo\\nbar\""} # Out: {"unquoted":"foo\nbar"} ``` ```bloblang root.text = this.literal.unquote() # In: {"literal":"\"hello\\tworld\""} # Out: {"text":"hello\tworld"} ``` ### [](#uppercase)uppercase Converts all letters in a string to uppercase. #### [](#examples-150)Examples ```bloblang root.foo = this.foo.uppercase() # In: {"foo":"hello world"} # Out: {"foo":"HELLO WORLD"} ``` ```bloblang root.code = this.product_code.uppercase() # In: {"product_code":"abc-123"} # Out: {"code":"ABC-123"} ``` ## [](#timestamp-manipulation)Timestamp manipulation ### [](#parse_duration)parse_duration Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h". #### [](#examples-151)Examples Parse microseconds to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration() # In: {"delay_for":"50us"} # Out: {"delay_for_ns":50000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration() / 1000000000 # In: {"delay_for":"2h"} # Out: {"delay_for_s":7200} ``` ### [](#parse_duration_iso8601)parse_duration_iso8601 Parses an ISO 8601 duration string into nanoseconds. Format: "P\[n\]Y\[n\]M\[n\]DT\[n\]H\[n\]M\[n\]S" or "P\[n\]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place). #### [](#examples-152)Examples Parse complex ISO 8601 duration to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration_iso8601() # In: {"delay_for":"P3Y6M4DT12H30M5S"} # Out: {"delay_for_ns":110839937000000000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000 # In: {"delay_for":"PT2H"} # Out: {"delay_for_s":7200} ``` ### [](#ts_add_iso8601)ts_add_iso8601 Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years. #### [](#parameters-104)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-153)Examples Add one year to a timestamp: ```bloblang root.next_year = this.created_at.ts_add_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"next_year":"2021-08-14T05:54:23Z"} ``` Add a complex duration with multiple units: ```bloblang root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2020-01-01T00:00:00Z"} # Out: {"future_date":"2021-03-04T04:05:06Z"} ``` ### [](#ts_format)ts_format Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. #### [](#parameters-105)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using Go’s reference time. | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | #### [](#examples-154)Examples Format timestamp with custom format: ```bloblang root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format unix timestamp with timezone specification: ```bloblang root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC") # In: {"created_at":1597405526} # Out: {"something_at":"2020-Aug-14 11:45:26"} ``` ### [](#ts_parse)ts_parse Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. #### [](#parameters-106)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | #### [](#examples-155)Examples Parse a date with abbreviated month name: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse a custom datetime format: ```bloblang root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)") # In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"} # Out: {"parsed":"2020-08-14T05:54:00Z"} ``` ### [](#ts_round)ts_round Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. #### [](#parameters-107)Parameters | Name | Type | Description | | --- | --- | --- | | duration | integer | A duration measured in nanoseconds to round by. | #### [](#examples-156)Examples Round timestamp to the nearest hour: ```bloblang root.created_at_hour = this.created_at.ts_round("1h".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_hour":"2020-08-14T06:00:00Z"} ``` Round timestamp to the nearest minute: ```bloblang root.created_at_minute = this.created_at.ts_round("1m".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_minute":"2020-08-14T05:54:00Z"} ``` ### [](#ts_strftime)ts_strftime Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. #### [](#parameters-108)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | #### [](#examples-157)Examples Format timestamp with strftime specifiers: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format with microseconds using %f directive: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26.371000"} ``` ### [](#ts_strptime)ts_strptime Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. #### [](#parameters-109)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | #### [](#examples-158)Examples Parse date with abbreviated month using strptime format: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse datetime with microseconds using %f directive: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f") # In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}} # Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}} ``` ### [](#ts_sub)ts_sub Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration. #### [](#parameters-110)Parameters | Name | Type | Description | | --- | --- | --- | | t2 | timestamp | The timestamp to subtract from the target timestamp. | #### [](#examples-159)Examples Calculate absolute duration between two timestamps: ```bloblang root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs() # In: {"started_at":"2020-08-13T05:54:23Z"} # Out: {"between":86400000000000} ``` Calculate signed duration (can be negative): ```bloblang root.duration_ns = this.end_time.ts_sub(this.start_time) # In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"} # Out: {"duration_ns":5400000000000} ``` ### [](#ts_sub_iso8601)ts_sub_iso8601 Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years. #### [](#parameters-111)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-160)Examples Subtract one year from a timestamp: ```bloblang root.last_year = this.created_at.ts_sub_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"last_year":"2019-08-14T05:54:23Z"} ``` Subtract a complex duration with multiple units: ```bloblang root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2021-03-04T04:05:06Z"} # Out: {"past_date":"2020-01-01T00:00:00Z"} ``` ### [](#ts_tz)ts_tz Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. #### [](#parameters-112)Parameters | Name | Type | Description | | --- | --- | --- | | tz | string | The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". | #### [](#examples-161)Examples Convert timestamp to UTC timezone: ```bloblang root.created_at_utc = this.created_at.ts_tz("UTC") # In: {"created_at":"2021-02-03T17:05:06+01:00"} # Out: {"created_at_utc":"2021-02-03T16:05:06Z"} ``` Convert timestamp to a specific timezone: ```bloblang root.created_at_ny = this.created_at.ts_tz("America/New_York") # In: {"created_at":"2021-02-03T16:05:06Z"} # Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"} ``` ### [](#ts_unix)ts_unix Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. #### [](#examples-162)Examples Convert RFC 3339 timestamp to unix seconds: ```bloblang root.created_at_unix = this.created_at.ts_unix() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000} ``` Unix timestamp passthrough returns same value: ```bloblang root.timestamp = this.ts.ts_unix() # In: {"ts":1257894000} # Out: {"timestamp":1257894000} ``` ### [](#ts_unix_micro)ts_unix_micro Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. #### [](#examples-163)Examples Convert timestamp to microseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_micro() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000} ``` Preserve microsecond precision from timestamp: ```bloblang root.precise_time = this.timestamp.ts_unix_micro() # In: {"timestamp":"2020-08-14T11:45:26.123456Z"} # Out: {"precise_time":1597405526123456} ``` ### [](#ts_unix_milli)ts_unix_milli Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. #### [](#examples-164)Examples Convert timestamp to milliseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_milli() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000} ``` Useful for JavaScript timestamp compatibility: ```bloblang root.js_timestamp = this.event_time.ts_unix_milli() # In: {"event_time":"2020-08-14T11:45:26.123Z"} # Out: {"js_timestamp":1597405526123} ``` ### [](#ts_unix_nano)ts_unix_nano Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. #### [](#examples-165)Examples Convert timestamp to nanoseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_nano() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000000} ``` Preserve full nanosecond precision: ```bloblang root.precise_time = this.timestamp.ts_unix_nano() # In: {"timestamp":"2020-08-14T11:45:26.123456789Z"} # Out: {"precise_time":1597405526123456789} ``` ## [](#type-coercion)Type coercion ### [](#array)array Converts a value to an array. #### [](#examples-166)Examples ```bloblang root.my_array = this.name.array() # In: {"name":"foobar bazson"} # Out: {"my_array":["foobar bazson"]} ``` ### [](#bool)bool Converts a value to a boolean with optional fallback. #### [](#parameters-113)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | bool | An optional value to yield if the target cannot be parsed as a boolean. | #### [](#examples-167)Examples ```bloblang root.foo = this.thing.bool() root.bar = this.thing.bool(true) ``` ### [](#bytes)bytes Marshals a value into a byte array. #### [](#examples-168)Examples ```bloblang root.first_byte = this.name.bytes().index(0) # In: {"name":"foobar bazson"} # Out: {"first_byte":102} ``` ### [](#not_empty)not_empty Ensures a value is not empty. #### [](#examples-169)Examples ```bloblang root.a = this.a.not_empty() # In: {"a":"foo"} # Out: {"a":"foo"} # In: {"a":""} # Out: Error("failed assignment (line 1): field `this.a`: string value is empty") # In: {"a":["foo","bar"]} # Out: {"a":["foo","bar"]} # In: {"a":[]} # Out: Error("failed assignment (line 1): field `this.a`: array value is empty") # In: {"a":{"b":"foo","c":"bar"}} # Out: {"a":{"b":"foo","c":"bar"}} # In: {"a":{}} # Out: Error("failed assignment (line 1): field `this.a`: object value is empty") ``` ### [](#not_null)not_null Ensures a value is not null. #### [](#examples-170)Examples ```bloblang root.a = this.a.not_null() # In: {"a":"foobar","b":"barbaz"} # Out: {"a":"foobar"} # In: {"b":"barbaz"} # Out: Error("failed assignment (line 1): field `this.a`: value is null") ``` ### [](#number)number Converts a value to a number with optional fallback. #### [](#parameters-114)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | float | An optional value to yield if the target cannot be parsed as a number. | #### [](#examples-171)Examples ```bloblang root.foo = this.thing.number() + 10 root.bar = this.thing.number(5) * 10 ``` ### [](#string)string Converts a value to a string representation. #### [](#examples-172)Examples ```bloblang root.nested_json = this.string() # In: {"foo":"bar"} # Out: {"nested_json":"{\"foo\":\"bar\"}"} ``` ```bloblang root.id = this.id.string() # In: {"id":228930314431312345} # Out: {"id":"228930314431312345"} ``` ### [](#timestamp)timestamp Converts a value to a timestamp with optional fallback. #### [](#parameters-115)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | timestamp | An optional value to yield if the target cannot be parsed as a timestamp. | #### [](#examples-173)Examples ```bloblang root.foo = this.ts.timestamp() root.bar = this.none.timestamp(1234567890.timestamp()) ``` ### [](#type)type Returns the type of a value as a string. #### [](#examples-174)Examples ```bloblang root.bar_type = this.bar.type() root.foo_type = this.foo.type() # In: {"bar":10,"foo":"is a string"} # Out: {"bar_type":"number","foo_type":"string"} ``` ```bloblang root.type = this.type() # In: "foobar" # Out: {"type":"string"} # In: 666 # Out: {"type":"number"} # In: false # Out: {"type":"bool"} # In: ["foo", "bar"] # Out: {"type":"array"} # In: {"foo": "bar"} # Out: {"type":"object"} # In: null # Out: {"type":"null"} ``` ```bloblang root.type = content().type() # In: foobar # Out: {"type":"bytes"} ``` ```bloblang root.type = this.ts_parse("2006-01-02").type() # In: "2022-06-06" # Out: {"type":"timestamp"} ``` ## [](#deprecated)Deprecated ### [](#format_timestamp)format_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. #### [](#parameters-116)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using Go’s reference time. | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_strftime)format_timestamp_strftime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. #### [](#parameters-117)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_unix)format_timestamp_unix > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. ### [](#format_timestamp_unix_micro)format_timestamp_unix_micro > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. ### [](#format_timestamp_unix_milli)format_timestamp_unix_milli > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. ### [](#format_timestamp_unix_nano)format_timestamp_unix_nano > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. ### [](#parse_timestamp)parse_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. #### [](#parameters-118)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | ### [](#parse_timestamp_strptime)parse_timestamp_strptime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. #### [](#parameters-119)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | --- # Page 507: Bloblang Walkthrough **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/walkthrough.md --- # Bloblang Walkthrough > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Bloblang Walkthrough latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/bloblang/walkthrough page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/bloblang/walkthrough.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/bloblang/walkthrough.adoc description: A step by step introduction to Bloblang page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- Bloblang is the most advanced mapping language that you’ll learn from this walkthrough (probably). It is designed for readability, the power to shape even the most outrageous input documents, and to easily make erratic schemas bend to your will. Bloblang is the native mapping language of Redpanda Connect, but it has been designed as a general purpose technology ready to be adopted by other tools. In this walkthrough you’ll learn how to make new friends by mapping their documents, and lose old friends as they grow jealous and bitter of your mapping abilities. There are a few ways to execute Bloblang but the way we’ll do it in this guide is to pull a Redpanda Connect docker image and run the command `rpk connect blobl server`, which opens up an interactive Bloblang editor: ```sh docker pull docker.redpanda.com/redpandadata/connect:latest docker run -p 4195:4195 --rm docker.redpanda.com/redpandadata/connect blobl server --no-open --host 0.0.0.0 ``` Next, open your browser at `http://localhost:4195` and you should see an app with three panels, the top-left is where you paste an input document, the bottom is your Bloblang mapping and on the top-right is the output. ## [](#your-first-assignment)Your first assignment The primary goal of a Bloblang mapping is to construct a brand new document by using an input document as a reference, which we achieve through a series of assignments. Bloblang is traditionally used to map JSON documents and that’s mostly what we’ll be doing in this walkthrough. The first mapping you’ll see when you open the editor is a single assignment: ```bloblang root = this # In: {"message":"hello world"} # Out: {"message":"hello world"} ``` On the left-hand side of the assignment is our assignment target, where `root` is a keyword referring to the root of the new document being constructed. On the right-hand side is a query which determines the value to be assigned, where `this` is a keyword that refers to the context of the mapping which begins as the root of the input document. As you can see the input document in the editor begins as a JSON object `{"message":"hello world"}`, and the output panel should show the result as: ```json { "message": "hello world" } ``` This output is a (neatly formatted) replica of the input document. This is the result of our mapping because we assigned the entire input document to the root of our new thing. Let’s create a brand new document by assigning a fresh object to the root: ```bloblang root = {} root.foo = this.message # In: {"message":"hello world"} # Out: {"foo":"hello world"} ``` Bloblang supports a bunch of [literal types](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/about/#literals), and the first line of this mapping assigns an empty object literal to the root. The second line then creates a new field `foo` on that object by assigning it the value of `message` from the input document. You should see that our output has changed to: ```json { "foo": "hello world" } ``` In Bloblang, when the path that we assign to contains fields that are themselves unset then they are created as empty objects. This rule also applies to `root` itself, which means the mapping: ```bloblang root.foo.bar = this.message root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"hello world","buz me":{"baz":"I like mapping"}}} ``` Will automatically create the objects required to produce the output document: ```json { "foo": { "bar": "hello world", "buz me": { "baz": "I like mapping" } } } ``` Also note that we can use quotes in order to express path segments that contain symbols or whitespace. Great, let’s move on quick before our self-satisfaction gets in the way of progress. ## [](#basic-methods-and-functions)Basic methods and functions Nothing is ever good enough for you, why should the input document be any different? Usually in our mappings it’s necessary to mutate values whilst we map them over, this is almost always done with methods, of which [there are many](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/). To demonstrate we’re going to change our mapping to [uppercase](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#uppercase) the field `message` from our input document: ```bloblang root.foo.bar = this.message.uppercase() root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO WORLD","buz me":{"baz":"I like mapping"}}} ``` As you can see the syntax for a method is similar to many languages, simply add a dot on the target value followed by the method name and arguments within brackets. With this method added our output document should look like this: ```json { "foo": { "bar": "HELLO WORLD", "buz me": { "baz": "I like mapping" } } } ``` Since the result of any Bloblang query is a value you can use methods on anything, including other methods. For example, we could expand our mapping of `message` to also replace `WORLD` with `EARTH` using the [`replace_all` method](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#replace_all): ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO EARTH","buz me":{"baz":"I like mapping"}}} ``` As you can see this method required some arguments. Methods support both nameless (like above) and named arguments, which are often literal values but can also be queries themselves. For example try out the following mapping using both named style and a dynamic argument: ```bloblang root.foo.bar = this.message.uppercase().replace_all(old: "WORLD", new: this.message.capitalize()) root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO Hello World","buz me":{"baz":"I like mapping"}}} ``` Woah, I think that’s the plot to Inception, let’s move onto functions. Functions are just boring methods that don’t have a target, and there are [plenty of them as well](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/). Functions are often used to extract information unrelated to the input document, such as [environment variables](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#env), or to generate data such as [timestamps](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#now) or [UUIDs](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#uuid_v4). Since we’re completionists let’s add one to our mapping: ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" root.foo.id = uuid_v4() # In: {"message":"hello world"} ``` Now I can’t tell you what the output looks like since it will be different each time it’s mapped, how fun! ### [](#deletions)Deletions Everything in Bloblang is an expression to be assigned, including deletions, which is a [function `deleted()`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#deleted). To illustrate let’s create a field we want to delete by changing our input to the following: ```json { "name": "fooman barson", "age": 7, "opinions": ["trucks are cool","trains are cool","chores are bad"] } ``` If we wanted a full copy of this document without the field `name` then we can assign `deleted()` to it: ```bloblang root = this root.name = deleted() # In: {"name":"fooman barson","age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} # Out: {"age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} ``` And it won’t be included in the output: ```json { "age": 7, "opinions": [ "trucks are cool", "trains are cool", "chores are bad" ] } ``` An alternative way to delete fields is the [method `without`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#without), our above example could be rewritten as a single assignment `root = this.without("name")`. However, `deleted()` is generally more powerful and will come into play more later on. ## [](#variables)Variables Sometimes it’s necessary to capture a value for later, but we might not want it to be added to the resulting document. In Bloblang we can achieve this with variables which are created using the `let` keyword, and can be referenced within subsequent queries with a dollar sign prefix: ```bloblang let id = uuid_v4() root.id_sha1 = $id.hash("sha1").encode("hex") root.id_md5 = $id.hash("md5").encode("hex") # In: {} ``` Variables can be assigned any value type, including objects and arrays. ## [](#unstructured-and-binary-data)Unstructured and binary data So far in all of our examples both the input document and our newly mapped document are structured, but this does not need to be so. Try assigning some literal value types directly to the `root`, such as a string `root = "hello world"`, or a number `root = 5`. You should notice that when a value type is assigned to the root the output is the raw value, and therefore strings are not quoted. This is what makes it possible to output data of any format, including encrypted, encoded or otherwise binary data. Unstructured mapping is not limited to the output. Rather than referencing the input document with `this`, where it must be structured, it is possible to reference it as a binary string with the [function `content`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#content), try changing your mapping to: ```bloblang root = content().uppercase() # In: hello world # Out: HELLO WORLD ``` When you add content to the input panel, it should be the same in the output panel, but in all uppercase. ## [](#conditionals)Conditionals In order to play around with conditionals let’s set our input to something structured: ```json { "pet": { "type": "cat", "is_cute": true, "treats": 5, "toys": 3 } } ``` In Bloblang all conditionals are expressions, this is a core principal of Bloblang and will be important later on when we’re mapping deeply nested structures. ### [](#if-expression)If expression The simplest conditional is the `if` expression, where the boolean condition does not need to be in parentheses. Let’s create a map that modifies the number of treats our pet receives based on a field: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Try that mapping out and you should see the number of treats in the output increased to 15. Now try changing the input field `pet.is_cute` to `false` and the output treats count should go back to the original 5. When a conditional expression doesn’t have a branch to execute then the assignment is skipped entirely, which means when the pet is not cute the value of `pet.treats` is unchanged (and remains the value set in the `root = this` assignment). We can add an `else` block to our `if` expression to remove treats entirely when the pet is not cute: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } else { deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` This is possible because field deletions are expressed as assigned values created with the `deleted()` function. ### [](#if-statement)If statement The `if` keyword can also be used as a statement in order to conditionally apply a series of mapping assignments, the previous example can be rewritten as: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 } else { root.pet.treats = deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Converting this mapping to use a statement has resulted in a more verbose mapping as we had to specify `root.pet.treats` multiple times as an assignment target. However, using `if` as a statement can be beneficial when multiple assignments rely on the same logic: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 root.pet.toys = this.pet.toys + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":13}} ``` More treats _and_ more toys! Lucky Spot! ### [](#match-expression)Match expression Another conditional expression is `match` which allows you to list many branches consisting of a condition and a query to execute separated with `=>`, where the first condition to pass is the one that is executed: ```bloblang root = this root.pet.toys = match { this.pet.treats > 5 => this.pet.treats - 5, this.pet.type == "cat" => 3, this.pet.type == "dog" => this.pet.toys - 3, this.pet.type == "horse" => this.pet.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Try executing that mapping with different values for `pet.type` and `pet.treats`. Match expressions can also specify a new context for the keyword `this` which can help reduce some of the boilerplate in your boolean conditions. The following mapping is equivalent to the previous: ```bloblang root = this root.pet.toys = match this.pet { this.treats > 5 => this.treats - 5, this.type == "cat" => 3, this.type == "dog" => this.toys - 3, this.type == "horse" => this.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Your boolean conditions can also be expressed as value types, in which case the context being matched will be compared to the value: ```bloblang root = this root.pet.toys = match this.pet.type { "cat" => 3, "dog" => 5, "rabbit" => 8, "horse" => 20, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` ## [](#error-handling)Error handling Bloblang can simplify handling errors. First, let’s take a look at what happens when errors _aren’t_ handled, change your input to the following: ```json { "palace_guards": 10, "angry_peasants": "I couldn't be bothered to ask them" } ``` And change your mapping to something simple like a number comparison: ```bloblang root.in_trouble = this.angry_peasants > this.palace_guards # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} ``` Uh oh! It looks like our canvasser was too lazy and our `angry_peasants` count was incorrectly set for this document. You should see an error in the output window that mentions something like `cannot compare types string (from field this.angry_peasants) and number (from field this.palace_guards)`, which means the mapping was abandoned. So what if we want to try and map something, but don’t care if it fails? In this case if we are unable to compare our angry peasants with palace guards then I would still consider us in trouble just to be safe. For that we have a special [method `catch`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#catch), which if we add to any query allows us to specify an argument to be returned when an error occurs. Since methods can be added to any query we can surround our arithmetic with brackets and catch the whole thing: ```bloblang root.in_trouble = (this.angry_peasants > this.palace_guards).catch(true) # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} # Out: {"in_trouble":true} ``` Now instead of an error we should see an output with `in_trouble` set to `true`. Try changing to value of `angry_peasants` to a few different values, including some numbers. One of the powerful features of `catch` is that when it is added at the end of a series of expressions and methods it will capture errors at any part of the series, allowing you to capture errors at any granularity. For example, the mapping: ```bloblang root.abort_mission = if this.mission.type == "impossible" { !this.user.motives.contains("must clear name") } else { this.mission.difficulty > 10 }.catch(false) # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` Will catch errors caused by: - `this.mission.type` not being a string - `this.user.motives` not being an array - `this.mission.difficulty` not being a number But will always return `false` if any of those errors occur. Try it out with this input and play around by breaking some of the fields: ```json { "mission": { "type": "impossible", "difficulty": 5 }, "user": { "motives": ["must clear name"] } } ``` Now try out this mapping: ```bloblang root.abort_mission = if (this.mission.type == "impossible").catch(true) { !this.user.motives.contains("must clear name").catch(false) } else { (this.mission.difficulty > 10).catch(true) } # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` This version is more granular and will capture each of the errors individually, with each error given a unique `true` or `false` fallback. ## [](#validation)Validation Sometimes errors are what we want. Failing a mapping with an error allows us to handle the bad document in other ways, such as routing it to a dead-letter queue or filtering it entirely. You can read about common Redpanda Connect error handling patterns for bad data in the [error handling guide](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/error_handling/), but the first step is to create the error. Luckily, Bloblang has a range of ways of creating errors under certain circumstances, which can be used in order to validate the data being mapped. There are [a few helper methods](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#type-coercion) that make validating and coercing fields nice and easy, try this mapping out: ```bloblang root.foo = this.foo.number() root.bar = this.bar.not_null() root.baz = this.baz.not_empty() # In: {"foo":5,"bar":"hello world","baz":[1,2,3]} # Out: {"foo":5,"bar":"hello world","baz":[1,2,3]} ``` With some of these sample inputs: ```json {"foo":"nope","bar":"hello world","baz":[1,2,3]} {"foo":5,"baz":[1,2,3]} {"foo":10,"bar":"hello world","baz":[]} ``` However, these methods don’t cover all use cases. The general purpose error throwing technique is the [`throw` function](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/functions/#throw), which takes an argument string that describes the error. When it’s called it will throw a mapping error that abandons the mapping. For example, we can check the type of a field with the [method `type`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#type), and then throw an error if it’s not the type we expected: ```bloblang root.foos = if this.user.foos.type() == "array" { this.user.foos } else { throw("foos must be an array, but it ain't, what gives?") } # In: {"user":{"foos":[1,2,3]}} ``` Try this mapping out with a few sample inputs: ```json {"user":{"foos":[1,2,3]}} {"user":{"foos":"1,2,3"}} ``` ## [](#context)Context In Bloblang, when we refer to the context we’re talking about the value returned with the keyword `this`. At the beginning of a mapping the context starts off as a reference to the root of a structured input document, which is why the mapping `root = this` will result in the same document coming out as you put in. However, in Bloblang there are mechanisms whereby the context might change, we’ve already seen how this can happen within a `match` expression. Another useful way to change the context is by adding a bracketed query expression as a method to a query, which looks like this: ```bloblang root = this.foo.bar.(this.baz + this.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the bracketed query expression the context becomes the result of the query that it’s a method of, so within the brackets in the above mapping the value of `this` points to the result of `this.foo.bar`, and the mapping is therefore equivalent to: ```bloblang root = this.foo.bar.baz + this.foo.bar.buz # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` With this handy trick the `throw` mapping from the validation section above could be rewritten as: ```bloblang root.foos = this.user.foos.(if this.type() == "array" { this } else { throw("foos must be an array, but it ain't, what gives?") }) # In: {"user":{"foos":[1,2,3]}} # Out: {"foos":[1,2,3]} ``` ### [](#naming-the-context)Naming the context Shadowing the keyword `this` with new contexts can look confusing in your mappings, and it also limits you to only being able to reference one context at any given time. As an alternative, Bloblang supports context capture expressions that look similar to lambda functions from other languages, where you can name the new context with the syntax ` -> `, which looks like this: ```bloblang root = this.foo.bar.(thing -> thing.baz + thing.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the brackets we now have a new field `thing`, which returns the context that would have otherwise been captured as `this`. This also means the value returned from `this` hasn’t changed and will continue to return the root of the input document. ## [](#coalescing)Coalescing Being able to open up bracketed query expressions on fields leads us onto another cool trick in Bloblang referred to as coalescing. It’s very common in the world of document mapping that due to structural deviations a value that we wish to obtain could come from one of multiple possible paths. To illustrate this problem change the input document to the following: ```json { "thing": { "article": { "id": "foo", "contents": "Some people did some stuff" } } } ``` Let’s say we wish to flatten this structure with the following mapping: ```bloblang root.contents = this.thing.article.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` But articles are only one of many document types we expect to receive, where the field `contents` remains the same but the field `article` could instead be `comment` or `share`. In this case we could expand our map of `contents` to use a `match` expression where we check for the existence of `article`, `comment`, etc in the input document. However, a much cleaner way of approaching this is with the pipe operator (`|`), which in Bloblang can be used to join multiple queries, where the first to yield a non-null result is selected. Change your mapping to the following: ```bloblang root.contents = this.thing.article.contents | this.thing.comment.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And now try changing the field `article` in your input document to `comment`. You should see that the value of `contents` remains as `Some people did some stuff` in the output document. Now, rather than write out the full path prefix `this.thing` each time we can use a bracketed query expression to change the context, giving us more space for adding other fields: ```bloblang root.contents = this.thing.(this.article | this.comment | this.share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And by the way, the keyword `this` within queries can be omitted and made implicit, which allows us to reduce this even further: ```bloblang root.contents = this.thing.(article | comment | share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Finally, we can also add a pipe operator at the end to fallback to a literal value when none of our candidates exists: ```bloblang root.contents = this.thing.(article | comment | share).contents | "nothing" # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Neat. ## [](#advanced-methods)Advanced methods What happens when you need to map all of the elements of an array? Or filter the keys of an object by their values? What if the fellowship just used the eagles to fly to mount doom? Bloblang offers a bunch of advanced methods for [manipulating structured data types](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#object%E2%80%94%E2%80%8Barray-manipulation), let’s take a quick tour of some of the cooler ones. Set your input document to this list of things: ```json { "num_friends": 5, "things": [ { "name": "yo-yo", "quantity": 10, "is_cool": true }, { "name": "dish soap", "quantity": 50, "is_cool": false }, { "name": "scooter", "quantity": 1, "is_cool": true }, { "name": "pirate hat", "quantity": 7, "is_cool": true } ] } ``` Let’s say we wanted to reduce the `things` in our input document to only those that are cool and where we have enough of them to share with our friends. We can do this with a [`filter` method](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#filter): ```bloblang root = this.things.filter(thing -> thing.is_cool && thing.quantity > this.num_friends) # In: {"num_friends":5,"things":[{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"dish soap","quantity":50,"is_cool":false},{"name":"scooter","quantity":1,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}]} # Out: [{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}] ``` Try running that mapping and you’ll see that the output is reduced. What is happening here is that the `filter` method takes an argument that is a query, and that query will be mapped for each individual element of the array (where the context is changed to the element itself). We have captured the context into a field `thing` which allows us to continue referencing the root of the input with `this`. The `filter` method requires the query parameter to resolve to a boolean `true` or `false`, and if it resolves to `true` the element will be present in the resulting array, otherwise it is removed. Being able to express a query argument to be applied to a range in this way is one of the more powerful features of Bloblang, and when mapping complex structured data these advanced methods will likely be a common tool that you’ll reach for. Another such method is [`map_each`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#map_each), which allows you to mutate each element of an array, or each value of an object. Change your input document to the following: ```json { "talking_heads": [ "1:E.T. is a bad film,Pokemon corrupted an entire generation", "2:Digimon ripped off Pokemon,Cats are boring", "3:I'm important", "4:Science is just made up,The Pokemon films are good,The weather is good" ] } ``` Here we have an array of talking heads, where each element is a string containing an identifer, a colon, and a comma separated list of their opinions. We wish to map each string into a structured object, which we can do with the following mapping: ```bloblang root = this.talking_heads.map_each(raw -> { "id": raw.split(":").index(0), "opinions": raw.split(":").index(1).split(",") }) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The argument to `map_each` is a query where the context is the element, which we capture into the field `raw`. The result of the query argument will become the value of the element in the resulting array, and in this case we return an object literal. In order to separate the identifier from opinions we perform a `split` by colon on the raw string element and get the first substring with the `index` method. We then do the split again and extract the remainder, and split that by comma in order to extract all of the opinions to an array field. However, one problem with this mapping is that the split by colon is written out twice and executed twice. A more efficient way of performing the same thing is with the bracketed query expressions we’ve played with before: ```bloblang root = this.talking_heads.map_each(raw -> raw.split(":").(split_string -> { "id": split_string.index(0), "opinions": split_string.index(1).split(",") })) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` > 📝 **NOTE: Challenge!** > > Challenge! > > Try updating that map so that only opinions that mention Pokemon are kept To find more methods for manipulating structured data types check out the [methods page](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#object%E2%80%94%E2%80%8Barray-manipulation). ## [](#reusable-mappings)Reusable mappings Bloblang has cool methods, sure, but there’s nothing cooler than methods you’ve made yourself. When the going gets tough in the mapping world the best solution is often to create a named mapping, which you can do with the keyword `map`: ```bloblang map parse_talking_head { let split_string = this.split(":") root.id = $split_string.index(0) root.opinions = $split_string.index(1).split(",") } root = this.talking_heads.map_each(raw -> raw.apply("parse_talking_head")) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The body of a named map, encapsulated with squiggly brackets, is a totally isolated mapping where `root` now refers to a new value being created for each invocation of the map, and `this` refers to the root of the context provided to the map. Named maps are executed with the [method `apply`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#apply), which has a string parameter identifying the map to execute, this means it’s possible to dynamically select the target map. As you can see in the above example we were able to use a custom map in order to create our talking head objects without the object literal. Within a named map we can also create variables that exist only within the scope of the map. A nice feature of named mappings is that they can invoke themselves recursively, allowing you to define mappings that walk deeply nested structures. The following mapping will scrub all values from a document that contain the word "Voldemort" (case insensitive): ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") # In: {"summer_party":{"theme":"the woman in black","guests":["Emma Bunton","the seal I spotted in Trebarwith","Voldemort","The cast of Swiss Army Man","Richard"],"notes":{"lisa":"I don't think voldemort eats fish","monty":"Seals hate dance music"}},"crushes":["Richard is nice but he hates pokemon","Victoria Beckham but I think she's taken","Charlie but they're totally into Voldemort"]} ``` Try running that mapping with the following input document: ```json { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ], "notes": { "lisa": "I don't think voldemort eats fish", "monty": "Seals hate dance music" } }, "crushes": [ "Richard is nice but he hates pokemon", "Victoria Beckham but I think she's taken", "Charlie but they're totally into Voldemort" ] } ``` ## [](#unit-testing)Unit testing Redpanda Connect has it’s own [unit testing capabilities](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/unit_testing/) that you can also use for your mappings. To start with save a mapping into a file called something like `naughty_man.blobl`, we can use the example above from the reusable mappings section: ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") ``` Next, we can define our unit tests in an accompanying YAML file in the same directory, let’s call this `naughty_man_test.yaml`: ```yaml tests: - name: test naughty man scrubber target_mapping: './naughty_man.blobl' environment: {} input_batch: - content: | { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ] } } output_batches: - - json_equals: { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ] } } ``` As you can see we’ve defined a single test, where we point to our mapping file which will be executed in our test. We then specify an input message which is a reduced version of the document we tried out before, and finally we specify output predicates, which is a JSON comparison against the output document. We can execute these tests with `rpk connect test ./naughty_man_test.yaml`, Redpanda Connect will also automatically find our tests if you simply run `rpk connect test ./…​`. You should see an output something like: ```text Test 'naughty_man_test.yaml' failed Failures: --- naughty_man_test.yaml --- test naughty man scrubber [line 2]: batch 0 message 0: json_equals: JSON content mismatch { "summer_party": { "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith" => "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ], "theme": "the woman in black" } } ``` Because in actual fact our expected output is wrong, I’ll leave it to you to spot the error. Once the test is fixed you should see: ```text Test 'naughty_man_test.yaml' succeeded ``` And now our mapping, should we need to expand it in the future, is better protected against regressions. You can read more about the Redpanda Connect unit test specification, including alternative output predicates, in [this document](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/unit_testing/). --- # Page 508: Amazon Web Services **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/aws.md --- # Amazon Web Services > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Amazon Web Services latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/cloud/aws page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/cloud/aws.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/cloud/aws.adoc description: Find out about AWS components in Redpanda Connect. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- There are many components within Redpanda Connect which utilize AWS services. You will find that each of these components contains a configuration section under the field `credentials`, of the format: ```yml credentials: profile: "" id: "" secret: "" token: "" role: "" role_external_id: "" ``` This section contains many fields and it isn’t immediately clear which of them are compulsory and which aren’t. This document aims to make it clear what each field is responsible for and how it might be used. ## [](#credentials)Credentials By explicitly setting the credentials you are using at the component level it’s possible to connect to components using different accounts within the same Redpanda Connect process. If you are using long term credentials for your account you only need to set the fields `id` and `secret`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key ``` If you are using short term credentials then you will also need to set the field `token`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key token: baz # aws_session_token ``` ## [](#assume-a-role)Assume a role It’s also possible to configure Redpanda Connect to [assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) using your credentials by setting the field `role` to your target role ARN. ```yml credentials: role: fooarn # Role ARN ``` This does NOT require explicit credentials, but it’s possible to use both. > 📝 **NOTE** > > Redpanda Cloud pipelines cannot assume roles through the default credential chain (IAM Roles for Service Accounts), and you cannot grant Redpanda-managed IAM roles permission to assume your roles. To assume a role from a Cloud pipeline, set explicit `id` and `secret` credentials for an IAM identity that you own and that has permission to assume the target role, and make sure the target role’s trust policy trusts that identity. Store these credentials as [secrets](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/). If you need to assume a role owned by another organization they might require you to [provide an external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html), in which case place it in the field `role_external_id`: ```yml credentials: role: fooarn # Role ARN role_external_id: bar_id ``` --- # Page 509: Ingest Real-Time Sensor Telemetry with the HTTP Gateway **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/gateway.md --- # Ingest Real-Time Sensor Telemetry with the HTTP Gateway > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Ingest Real-Time Sensor Telemetry with the HTTP Gateway latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/cloud/gateway page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/cloud/gateway.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/cloud/gateway.adoc description: Learn how to stream sensor telemetry data into Redpanda Cloud using the gateway input in Redpanda Connect. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-05-26" --- In this guide, you’ll build a pipeline that uses the `gateway` input to receive real-time telemetry data from sensors over HTTP. Each incoming message is normalized, published to a Redpanda topic, and acknowledged back to the sender. This setup is ideal for IoT, mobile, and embedded systems that need to stream data to Redpanda Cloud without using a Kafka client. The `gateway` input exposes a secure HTTP endpoint, simplifying ingestion from devices. Because HTTP is universally supported, it’s easier to integrate on constrained devices, microcontrollers, or languages that don’t support Kafka natively. Additional benefits: - **Simplified security**: Devices authenticate with Redpanda Cloud API tokens (using Bearer headers). No need to embed Kafka credentials, manage TLS, or expose brokers publicly. - **Operational flexibility**: Devices are decoupled from Kafka internals like topics or schemas. You can evolve pipeline logic without touching device code. - **Automatic provisioning**: Redpanda Cloud generates a secure endpoint URL when you deploy the pipeline. ## [](#prerequisites)Prerequisites - A Redpanda Cloud cluster (Serverless, Dedicated, or BYOC) - cURL or another compatible HTTP client ## [](#create-a-sensor-user-in-redpanda-cloud)Create a sensor user in Redpanda Cloud A sensor user is required to securely authenticate and manage access to the `sensor.telemetry` topic, ensuring that only authorized devices can produce messages to the topic. 1. [Log in to Redpanda Cloud](https://cloud.redpanda.com). 2. Go to **Topics** and create a topic named `sensor.telemetry`. This topic will be used to store incoming telemetry messages. 3. Go to **Security** and create a user with the following details: - **Username**: `sensor-sasl-user` - **Password**: `` (choose a secure password) - **SASL Mechanism**: `SCRAM-SHA-256` 4. Copy the password and save it securely for the next step. 5. Go to **Secrets Store** and create a new secret named `SENSOR_SASL_PASSWORD` with the value of the password you set for the user. - Set the scope of the secret to Redpanda Cluster and Redpanda Connect. 6. Go to **Security > ACLs** and create an access policy for the `sensor-sasl-user` user. This policy should allow the user to produce messages to the `sensor.telemetry` topic. ## [](#create-a-service-account)Create a service account The service account is used to authenticate requests to the gateway endpoint. It provides a secure way to manage access to the gateway without embedding sensitive credentials in your devices. 1. [Create a new service account](https://cloud.redpanda.com/service-accounts/new) in Redpanda Cloud named `sensor-ingest` and give it a description like "Service account for sensor telemetry ingestion". 2. Copy the client ID and secret. 3. Request a new API token for the service account. This token will be used to authenticate requests to the gateway. ```bash curl --request POST \ --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \ --header 'content-type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id= \ --data client_secret= \ --data audience=cloudv2-production.redpanda.cloud ``` Replace `` and `` with the values you copied from the service account. The request response provides an access token that remains **valid for one hour**. 4. Set the access token as an environment variable: ```bash export CLOUD_API_TOKEN= ``` ## [](#create-a-redpanda-cloud-pipeline)Create a Redpanda Cloud pipeline 1. Go to **Connect** and click **Create Pipeline**. 2. Name the pipeline `sensor-telemetry-ingest` and give it a description like "Ingest real-time sensor telemetry data". 3. Paste the following pipeline configuration into the editor: ```yaml input: gateway: rate_limit: "limit" rate_limit_resources: - label: limit local: count: 100 interval: 1s pipeline: processors: - bloblang: | root.sensor_id = this.sensor_id root.type = this.type root.value = this.value root.unit = this.unit root.received_at = now() output: broker: pattern: fan_out_sequential outputs: - redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: sensor.telemetry tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: sensor-sasl-user password: ${secrets.SENSOR_SASL_PASSWORD} - sync_response: processors: - mapping: | root = { "status": "ok", "received_at": now() } ``` This pipeline listens for incoming telemetry messages over HTTP and processes each one in real time. Here’s what each section does: - `input.gateway`: Defines the input source. It exposes a secure HTTP endpoint that devices can post to. The optional `rate_limit` named `limit` is applied to protect the pipeline from overload. - `rate_limit_resources.limit`: Limits traffic to 100 requests per second. If this rate is exceeded, HTTP requests are rejected with a 429 response. - `pipeline.processors.bloblang`: Normalizes the incoming message by copying fields and adding a `received_at` timestamp (using the current time). - `output.broker`: Uses a `fan_out_sequential` pattern to send each message to two outputs: - The first output publishes the normalized message to the `sensor.telemetry` Redpanda topic. - The second output sends a synchronous JSON response back to the sender confirming receipt. 4. Click **Start**. The pipeline starts deploying. When the state changes to "Running", the pipeline is ready to accept incoming messages. 5. Click the pipeline to view its details. When the pipeline is deployed, a URL is displayed. This is the HTTP endpoint to which you’ll post sensor data. 6. Copy the URL. ## [](#send-sensor-data)Send sensor data Send test data using cURL. Replace `` with the URL provided by Redpanda Cloud when you deployed the pipeline. ```bash curl -X POST \ -H "Authorization: Bearer $CLOUD_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "sensor_id": "thermo-42", "type": "temperature", "value": 21.7, "unit": "C" }' ``` Expected response: ```json { "received_at":"2025-06-17T09:48:50.986719231Z", "sensor_id":"thermo-42", "type":"temperature", "unit":"C", "value":21.7 } ``` You can verify that the message was successfully ingested by checking the `sensor.telemetry` topic in Redpanda Cloud. To verify that the rate limit is working, try sending more than 100 requests per second. You should receive a 429 response with a `Retry-After` header indicating when to retry. ```bash seq 1 300 | xargs -n1 -P50 -I{} curl -s -o /dev/null -w "%{http_code}\n" \ -X POST \ -H "Authorization: Bearer $CLOUD_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sensor_id":"test", "value": 42}' ``` You should see a mixture of `200` and `429` responses, indicating that the rate limit is being enforced. ## [](#monitor-the-pipeline)Monitor the pipeline You can monitor the pipeline’s logs in the Redpanda Cloud UI. 1. Go to **Connect** and select the `sensor-telemetry-ingest` pipeline. 2. Click on the **Logs** tab to view real-time logs of the pipeline’s activity. You can see any errors that occur during processing. ## [](#next-steps)Next steps - Filter or enrich events with conditional Bloblang. - Route messages by `sensor.type` to different topics. ## [](#suggested-reading)Suggested reading - [`gateway` input reference](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gateway/) - [Bloblang functions](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/interpolation/) - [Redpanda Cloud API authentication](https://docs.redpanda.com/api/doc/cloud-dataplane/authentication) --- # Page 510: Google Cloud Platform **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/gcp.md --- # Google Cloud Platform > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Google Cloud Platform latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/cloud/gcp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/cloud/gcp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/cloud/gcp.adoc description: Find out about GCP components in Redpanda Connect. page-git-created-date: "2024-09-09" page-git-modified-date: "2026-08-11" --- There are many components within Redpanda Connect which utilize Google Cloud Platform (GCP) services. You will find that each of these components require valid credentials. When running Redpanda Connect inside a Google Cloud environment that has a [default service account](https://cloud.google.com/iam/docs/service-accounts#default), it can automatically retrieve the service account credentials to call Google Cloud APIs through a library called Application Default Credentials (ADC). Otherwise, if your application runs outside Google Cloud environments that provide a default service account, you need to manually create one. Once you have a service account set up which has the required permissions, you can [create](https://console.cloud.google.com/apis/credentials/serviceaccountkey) a new Service Account Key and download it as a JSON file. Then all you need to do set the path to this JSON file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. Please refer to [this document](https://cloud.google.com/docs/authentication/production) for details. --- # Page 511: Migrate to the Unified Redpanda Migrator **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/migrate-unified-redpanda-migrator.md --- # Migrate to the Unified Redpanda Migrator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Migrate to the Unified Redpanda Migrator latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/migrate-unified-redpanda-migrator page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/migrate-unified-redpanda-migrator.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/migrate-unified-redpanda-migrator.adoc description: Learn how to migrate from legacy migrator components to the unified `redpanda_migrator` input/output pair in Redpanda Connect 4.67.5+. page-git-created-date: "2025-10-24" page-git-modified-date: "2026-05-26" --- > ❗ **IMPORTANT** > > This page is about migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see [Redpanda Migrator](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/redpanda_migrator/). This guide explains how to migrate from legacy migrator components (`redpanda_migrator_bundle`, `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets`) to the unified `redpanda_migrator` input/output pair introduced in Redpanda Connect 4.67.5+. The unified migrator consolidates all migration logic into a single input/output pair, simplifying configuration and improving reliability. ## [](#overview)Overview | Available in | Redpanda Connect 4.67.5+ | | --- | --- | | Legacy status | Deprecated in 4.67.5, removed in 4.85.0 | | Compatibility | Not backward-compatible | | Configuration model | One input and one output, paired by label | | Primary control | All migration logic resides in the output component | Key concepts: - Components are paired by matching `label` values. - The input defines the source cluster and schema registry. - The output defines the destination cluster, schema registry, and migration behavior. - Topic mapping and consumer group migration are configured in the output. ## [](#architectural-changes)Architectural changes ### [](#legacy-architecture)Legacy architecture A complex bundle (`redpanda_migrator_bundle`) that managed three subcomponents: - `redpanda_migrator`: Data transfer - `schema_registry`: Schema synchronization - `redpanda_migrator_offsets`: Consumer group offsets This design required complex internal routing and sequencing. ### [](#unified-architecture)Unified architecture A single `redpanda_migrator` input/output pair replaces the bundle: - **Input**: Consumes from the source Kafka cluster. - **Output**: Handles topic creation, schema synchronization, ACLs, and consumer group offsets. Benefits: - Simplified setup: all configuration consolidated in one output component. - Improved coordination: no internal routing or wrapper logic. - Enhanced control: fine-grained schema and topic options, improved offset handling. ## [](#migration-steps)Migration steps Follow this checklist in order to ensure a safe, low-risk migration. - Back up your existing configurations. - Add new `input.redpanda_migrator` and `output.redpanda_migrator` components with matching labels. - Move source Kafka and Schema Registry settings to the input. - Move destination Kafka and Schema Registry settings to the output. - Replace `topic_prefix` with `topic` using interpolation syntax. - Move offset settings to `output.redpanda_migrator.consumer_groups`. - Remove deprecated fields. - Validate configuration with `rpk connect lint`. - Test using non-production topics first. - Monitor logs and performance during migration. - Remove legacy configuration after successful migration. ## [](#field-mapping-reference)Field mapping reference ### [](#bundle-wrapper-redpanda_migrator_bundle)Bundle wrapper (`redpanda_migrator_bundle`) #### [](#input-mapping)Input mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | input.redpanda_migrator | Moved | Source cluster connection | | schema_registry | input.redpanda_migrator.schema_registry | Moved | Source schema registry | | migrate_schemas_before_data | - | Removed | Controlled by output schema interval | | consumer_group_offsets_poll_interval | output.redpanda_migrator.consumer_groups.interval | Moved | Now controls sync frequency | #### [](#output-mapping)Output mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | output.redpanda_migrator | Moved | Destination cluster configuration | | schema_registry | output.redpanda_migrator.schema_registry | Moved | Destination schema registry | | translate_schema_ids | output.redpanda_migrator.schema_registry.translate_ids | Moved | Schema ID translation | | input_bundle_label | label | Replaced | Input and output paired by label | ### [](#data-migration-fields)Data migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | All (*) | input.redpanda_migrator.* | Moved | Direct mapping | | topics (explicit list) | input.redpanda_migrator.topics | Unchanged | Still supported for explicit lists | | regexp_topics: true | input.redpanda_migrator.regexp_topics_include, regexp_topics_exclude | Deprecated | Use include/exclude arrays for pattern-based selection | | topic_prefix | output.redpanda_migrator.topic | Replaced | Use interpolation, for example 'prefix_${! @kafka_topic }' | | replication_factor_override, replication_factor | output.redpanda_migrator.topic_replication_factor | Replaced | Unified field | | input_resource | label | Replaced | Label pairing replaces internal routing | | - | output.redpanda_migrator.provenance_header | New | Optional header for tracking message source cluster | ### [](#schema-migration-fields)Schema migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | Connection fields | input.redpanda_migrator.schema_registry.* | Moved | Source schema registry | | subject_filter | output.redpanda_migrator.schema_registry.include, exclude | Replaced | Use regex lists for filtering | | include_deleted | output.redpanda_migrator.schema_registry.include_deleted | Moved | Configured on destination | | backfill_dependencies | output.redpanda_migrator.schema_registry.versions | Replaced | Choose all or latest | ### [](#consumer-group-offset-migration)Consumer group offset migration The `redpanda_migrator_offsets` pair is replaced by the `consumer_groups` block in the output. | Legacy Component | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator_offsets (input/output) | output.redpanda_migrator.consumer_groups | Replaced | Unified control block | ## [](#migration-example)Migration example The following example demonstrates a complete migration from legacy to unified components. Legacy configuration ```yaml input: label: "source_cluster" redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "source-kafka:9092" ] topics: [ "orders", "payments" ] consumer_group: "migration_group" schema_registry: url: "http://source-registry:8081" migrate_schemas_before_data: false consumer_group_offsets_poll_interval: 30s output: redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "destination-redpanda:9092" ] topic_prefix: "migrated_" schema_registry: url: "http://destination-registry:8081" translate_schema_ids: true input_bundle_label: "source_cluster" ``` Unified configuration ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "source-kafka:9092" ] # Pattern-based topic selection (for migrating all topics except system topics) # Note: You can still use explicit lists: topics: [ "orders", "payments" ] regexp_topics_include: [ '.' ] (2) regexp_topics_exclude: [ '^_' ] (3) consumer_group: "migration_group" # Source Schema Registry settings schema_registry: url: "http://source-registry:8081" output: label: "migration_pipeline" (4) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "destination-redpanda:9092" ] # Topic mapping (replaces topic_prefix) topic: 'migrated_${! @kafka_topic }' (5) # Add source cluster tracking header provenance_header: "x-source-cluster" (6) # Destination Schema Registry and migration settings schema_registry: url: "http://destination-registry:8081" translate_ids: true # Rename subjects subject: 'migrated_${! metadata("schema_registry_subject") }' # Consumer group migration settings consumer_groups: enabled: true interval: 30s (7) ``` | 1 | Labels are now used for pairing input and output. | | --- | --- | | 2 | Match all topics using regex pattern. | | 3 | Exclude internal/system topics starting with underscore. | | 4 | Matching label pairs the input and output components. | | 5 | Use interpolation syntax to replicate topic_prefix behavior. | | 6 | Adds a header to track which cluster messages originated from, useful for debugging and auditing. | | 7 | Replaces consumer_group_offsets_poll_interval. | ## [](#validation)Validation Before running, validate your configuration: ```bash rpk connect lint config.yaml ``` Then test on a small set of topics before running full migrations. ## [](#troubleshooting)Troubleshooting | Problem | Likely Cause | Solution | | --- | --- | --- | | Labels do not match | Input and output labels differ | Use identical, case-sensitive labels. | | Topic interpolation errors | Incorrect syntax | Use topic: 'prefix_${! @kafka_topic }' with quotes and !. | | Schema registry connection fails | Incorrect registry placement | The source registry must be in the input. The destination registry must be in the output. | | Consumer group migration not working | Missing consumer_groups.enabled: true | Ensure consumer group migration is explicitly enabled. | ## [](#after-migration)After migration After verifying that the new migrator works as expected: - Remove legacy configuration files. - Update internal documentation and runbooks. - Train your team on the new configuration model. - See the [`redpanda_migrator` output](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) reference for advanced configuration options. --- # Page 512: Synchronous Responses **URL**: https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/sync_responses.md --- # Synchronous Responses > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Synchronous Responses latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: connect/guides/sync_responses page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: connect/guides/sync_responses.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/connect/guides/sync_responses.adoc description: Understand synchronous response handling in Redpanda Connect, ensuring reliable and efficient data processing. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-08-11" --- In a regular Redpanda Connect pipeline, messages flow in one direction and acknowledgements in the other: ```text ----------- Message -------------> Input (AMQP) -> Processors -> Output (AMQP) <------- Acknowledgement --------- ``` However, Redpanda Connect supports bidirectional protocols like HTTP and WebSocket, which allow responses to be returned directly from the pipeline. For example, HTTP is a request/response protocol, and inputs like `http_server` (Self-Managed) or `gateway` (Redpanda Cloud) support returning response payloads to the requester. ```text --------- Request Body --------> Input (HTTP) -> Processors -> Output (Sync Response) <--- Response Body (and ack) --- ``` ## [](#routing-processed-messages-back)Routing processed messages back To return a processed response, use the [`sync_response`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/sync_response/) output. Use the `gateway` input in Redpanda Cloud: ```yaml input: gateway: {} pipeline: processors: - mapping: | root = { city: json("location"), forecast: "Clear skies with light winds", temperature_c: 22 } output: sync_response: {} ``` Sending this request: ```json { "location": "Berlin" } ``` Returns: ```json { "city": "Berlin", "forecast": "Clear skies with light winds", "temperature_c": 22 } ``` ## [](#combine-with-other-outputs)Combine with other outputs You can route processed messages to storage and return a response using a [`broker`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/broker/) output. ```yaml input: gateway: {} output: broker: pattern: fan_out outputs: - redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: weather.requests tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: ${secrets.USERNAME} password: ${secrets.PASSWORD} - sync_response: processors: - mapping: | root = { status: "received", received_at: now() } ``` ## [](#returning-partially-processed-messages)Returning partially processed messages You can return a response before the message is fully processed by using the [`sync_response` processor](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/sync_response/). This allows continued processing after the response is set. ```yaml pipeline: processors: - mapping: root = "Received weather report for %s".format(json("location")) - sync_response: {} - mapping: root.reported_at = now() ``` This returns `"Received weather report for Berlin"` to the client, but continues modifying the message before storing or forwarding it. > 📝 **NOTE** > > Due to delivery guarantees, the response is not sent until all downstream processing and acknowledgements are complete. --- # Page 513: Consume Data **URL**: https://docs.redpanda.com/cloud-data-platform/develop/consume-data.md --- # Consume Data > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Consume Data latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: consume-data/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: consume-data/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/consume-data/index.adoc description: Learn about consumer offsets and follower fetching. page-git-created-date: "2024-07-25" page-git-modified-date: "2024-08-01" --- - [Consumer Offsets](consumer-offsets/) Redpanda uses an internal topic, `__consumer_offsets`, to store committed offsets from each Kafka consumer that is attached to Redpanda. - [Follower Fetching](follower-fetching/) Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. - [Paginate Messages in Redpanda Console](paginate-messages-events/) Retrieve more than the default batch of messages in Redpanda Console by paging through larger result sets. --- # Page 514: Consumer Offsets **URL**: https://docs.redpanda.com/cloud-data-platform/develop/consume-data/consumer-offsets.md --- # Consumer Offsets > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Consumer Offsets latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: consume-data/consumer-offsets page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: consume-data/consumer-offsets.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/consume-data/consumer-offsets.adoc description: Redpanda uses an internal topic, __consumer_offsets, to store committed offsets from each Kafka consumer that is attached to Redpanda. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- In Redpanda, all messages are organized by [topic](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#topic) and distributed across multiple partitions, based on a [partition strategy](https://www.redpanda.com/guides/kafka-tutorial-kafka-partition-strategy). For example, when using the round robin strategy, a producer writing to a topic with five partitions would distribute approximately 20% of the messages to each [partition](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#partition). Within a partition, each message (once accepted and acknowledged by the partition leader) is permanently assigned a unique sequence number called an [offset](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#offset). Offsets enable consumers to resume processing from a specific point, such as after an application outage. If an outage prevents your application from receiving events, you can use the consumer offset to retrieve only the events that occurred during the downtime. By default, the first message in a partition is assigned offset 0, the next is offset 1, and so on. You can manually specify a specific start value for offsets if needed. Once assigned, offsets are immutable, ensuring that the order of messages within a partition is preserved. ## [](#how-consumers-use-offsets)How consumers use offsets As a consumer reads messages from Redpanda, it can save its progress by “committing the offset” (known as an [offset commit](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#offset-commit)), an action initiated by the consumer, not Redpanda. Kafka client libraries provide an API for committing offsets, which communicates with Redpanda using the [consumer group](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#consumer-group) API. Each committed offset is stored as a message in the `__consumer_offsets` topic, which is a private Redpanda topic that stores committed offsets from each Kafka consumer attached to Redpanda, allowing the consumer to resume processing from the last committed point. Redpanda exposes the `__consumer_offsets` key to enable the many tools in the Kafka ecosystem that rely on this value for their operation, providing greater ecosystem interoperability with environments and applications. When a consumer group works together to consume data from topics, the partitions are divided among the consumers in the group. For example, if a topic has 12 partitions, and there are two consumers, each consumer would be assigned six partitions to consume. If a new consumer starts later and joins this consumer group, a rebalance occurs, such that each consumer ends up with four partitions to consume. You specify a consumer group by setting the `group.id` property to a unique name for the group. Kafka tracks the maximum offset it has consumed in each partition and can commit offsets to ensure it can resume processing from the same point in the event of a restart. Kafka allows offsets for a consumer group to be stored on a designated broker, known as the group coordinator. All consumers in the group send their offset commits and fetch requests to this group coordinator. > 📝 **NOTE** > > More advanced consumers can read data from Redpanda without using a consumer group by requesting to read a specific topic, partition, and offset range. This pattern is often used by stream processing systems such as Apache Spark and Apache Flink, which have their own mechanisms for assigning work to consumers. Redpanda Console derives its consumer group lists and lag information from committed consumer group offsets, so these consumers appear in Console only if they also commit offsets to a consumer group. For example, Spark Structured Streaming does not commit offsets, so its consumers never appear. Flink commits offsets only when it is configured to do so on checkpoint completion. When the group coordinator receives an OffsetCommitRequest, it appends the request to the [compacted](https://kafka.apache.org/documentation/#compaction) Kafka topic `__consumer_offsets`. The broker sends a successful offset commit response to the consumer only after all the replicas of the offsets topic receive the offsets. If the offsets fail to replicate within a configurable timeout, the offset commit fails and the consumer may retry the commit after backing off. The brokers periodically compact the `__consumer_offsets` topic, because it only needs to maintain the most recent offset commit for each partition. The coordinator also caches the offsets in an in-memory table to serve offset fetches quickly. ## [](#commit-strategies)Commit strategies There are several strategies for managing offset commits: ### [](#automatic-offset-commit)Automatic offset commit Auto commit is the default commit strategy, where the client automatically commits offsets at regular intervals. This is set with the `enable.auto.commit` property. The client then commits offsets every `auto.commit.interval.ms` milliseconds. The primary advantage of the auto commit approach is its simplicity. After it is configured, the consumer requires no additional effort. Commits are managed in the background. However, the consumer is unaware of what was committed or when. As a result, after an application restart, some messages may be reprocessed (since consumption resumes from the last committed offset, which may include already-processed messages). The strategy guarantees at-least-once delivery. > 📝 **NOTE** > > If your consume configuration is set up to consume and write to another data store, and the write to that datastore fails, the consumer might not recover when it is auto-committed. It may not only duplicate messages, but could also drop messages intended to be in another datastore. Make sure you understand the trade-off possibilities associated with this default behavior. ### [](#manual-offset-commit)Manual offset commit The manual offset commit strategy gives consumers greater control over when commits occur. This approach is typically used when a consumer needs to align commits with an external system, such as database transactions in an RDBMS. The main advantage of manual commits is that they allow you to decide exactly when a record is considered consumed. You can use two API calls for this: `commitSync` and `commitAsync`, which differ in their blocking behavior. #### [](#synchronous-commit)Synchronous commit The advantage of synchronous commits is that consumers can take appropriate action before continuing to consume messages, albeit at the expense of increased latency (while waiting for the commit to return). The commit (`commitSync`) will also retry automatically, until it either succeeds or receives an unrecoverable error. The following example shows a synchronous commit: ```java consumer.subscribe(Arrays.asList("foo", "bar")); while (true) { ConsumerRecords records = consumer.poll(100); for (ConsumerRecord record : records) { // process records here ... // ... and at the appropriate point, call commit (not after every message) consumer.commitSync(); } } ``` #### [](#asynchronous-commit)Asynchronous commit The advantage of asynchronous commits is lower latency, because the consumer does not pause to wait for the commit response. However, there is no automatic retry of the commit (`commitAsync`) if it fails. There is also increased coding complexity (due to the asynchronous callbacks). The following example shows an asynchronous commit in which the consumer will not block. Instead, the commit call registers a callback, which is executed once the commit returns: ```java void callback() { // executed when the commit returns } consumer.subscribe(Arrays.asList("foo", "bar")); while (true) { ConsumerRecords records = consumer.poll(100); for (ConsumerRecord record : records) { // process records here ... // ... and at the appropriate point, call commit consumer.commitAsync(callback); } } ``` ### [](#external-offset-management)External offset management The external offset management strategy allows consumers to manage offsets independently of Redpanda. In this approach: - Consumers bypass the consumer group API and directly assign partitions instead of subscribing to a topic. - Offsets are not committed to Redpanda, but are instead stored in an external storage system. Because consumers that use this strategy do not commit offsets to Redpanda, they do not appear in the consumer group lists in Redpanda Console. This is expected behavior for systems such as Apache Spark Structured Streaming, which manages offsets entirely in its own checkpoint storage. Systems such as Apache Flink can also commit offsets to Kafka when a checkpoint completes. In that configuration, they follow the hybrid strategy described in the next section and do appear in Console. To implement an external offset management strategy: 1. Set `enable.auto.commit` to `false`. 2. Use `assign(Collection)` to assign partitions. 3. Use the offset provided with each ConsumerRecord to save your position. 4. Upon restart, use `seek(TopicPartition, long)` to restore the position of the consumer. ### [](#hybrid-offset-management)Hybrid offset management The hybrid offset management strategy allows consumers to handle their own consumer rebalancing while still leveraging Redpanda’s offset commit functionality. In this approach: - Consumers bypass the consumer group API and directly assign partitions instead of subscribing to a topic. - Offsets are committed to Redpanda. ## [](#offset-commit-best-practices)Offset commit best practices Follow these best practices to optimize offset commits. ### [](#avoid-over-committing)Avoid over-committing The purpose of a commit is to save consumer progress. More frequent commits reduce the amount of data to re-read after an application restart, as the commit interval directly affects the recovery point objective (RPO). Because a lower RPO is desirable, application designers may believe that committing frequently is a good design choice. However, committing too frequently can result in adverse consequences. While individually small, each commit still results in a message being written to the `__consumer_offsets` topic, because the position of the consumer against every partition must be recorded. At high commit rates, this workload can become a bottleneck for both the client and the server. Additionally, many Kafka client implementations do not coalesce offset commits, meaning redundant commits in a backlog still need to be processed. In many Kafka client implementations, offset commits aren’t coalesced at the client; so if a backlog of commits forms (when using the asynchronous commit API), the earlier commits still need to be processed, even though they are effectively redundant. **Best practice**: Monitor commit latency to ensure commits are timely. If you notice performance issues, commit less frequently. ### [](#use-unique-consumer-groups)Use unique consumer groups Like many topics, the consumer group topic has multiple partitions to help with performance. When writing commit messages, Redpanda groups all of the commits for a consumer group into a specific partition to maintain ordering. Reusing a consumer group across multiple applications, even for different topics, forces all commits to use a single partition, negating the benefits of partitioning. **Best practice**: Assign a unique consumer group to each application to distribute the commit load across all partitions. ### [](#tune-the-consumer-group)Tune the consumer group In highly parallel applications, frequent consumer group heartbeats can create unnecessary overhead. For example, 3,200 consumers checking every 500 milliseconds generate 6,400 heartbeats per second. You can optimize this behavior by increasing the `heartbeat.interval.ms` (along with `session.timeout.ms`). **Best practice**: Adjust heartbeat and session timeout settings to reduce unnecessary overhead in large-scale applications. --- # Page 515: Follower Fetching **URL**: https://docs.redpanda.com/cloud-data-platform/develop/consume-data/follower-fetching.md --- # Follower Fetching > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Follower Fetching latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: consume-data/follower-fetching page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: consume-data/follower-fetching.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/consume-data/follower-fetching.adoc description: Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. ## [](#about-follower-fetching)About follower fetching **Follower fetching** enables a consumer to fetch records from the closest replica of a topic partition, regardless of whether it’s a leader or a follower. For a Redpanda cluster deployed across different data centers and availability zones (AZs), restricting a consumer to fetch only from the leader of a partition can incur greater costs and have higher latency than fetching from a follower that is geographically closer to the consumer. With follower fetching (proposed in [KIP-392](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica)), the fetch protocol is extended to support a consumer fetching from any replica. This includes [Remote Read Replicas](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/remote-read-replicas/). The first fetch from a consumer is processed by a Redpanda leader broker. The leader checks for a replica (itself or a follower) that has a rack ID that matches the consumer’s rack ID. If a replica with a matching rack ID is found, the fetch request returns records from that replica. Otherwise, the fetch is handled by the leader. ## [](#configure-follower-fetching)Configure follower fetching Redpanda decides which replica a consumer fetches from. If the consumer configures its `client.rack` property, Redpanda by default selects a replica from the same rack as the consumer, if available. For each consumer, set the `client.rack` property to a rack ID. Rack awareness is pre-enabled for cloud-based clusters in multi-AZ environments. ## [](#suggested-videos)Suggested videos - [YouTube - Redpanda Office Hour: Follower Fetching (52 mins)](https://www.youtube.com/watch?v=wV6gH5_yVaw&ab_channel=RedpandaData) --- # Page 516: Paginate Messages in Redpanda Console **URL**: https://docs.redpanda.com/cloud-data-platform/develop/consume-data/paginate-messages-events.md --- # Paginate Messages in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Paginate Messages in Redpanda Console latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: consume-data/paginate-messages-events page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: consume-data/paginate-messages-events.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/consume-data/paginate-messages-events.adoc description: Retrieve more than the default batch of messages in Redpanda Console by paging through larger result sets. page-git-created-date: "2026-04-30" page-git-modified-date: "2026-05-26" --- By default, the **Messages** tab on a topic returns the number of records set in **Max results**. Enable **Continuous Pagination** when you need to inspect a topic beyond that cap. ## [](#browse-all-messages-in-a-topic)Browse all messages in a topic 1. Go to **Topics** and select a topic. 2. Open the **Messages** tab. 3. (Optional) Set **Start offset** and **Max results**, or apply filters, to narrow the records you want to inspect. See [Programmable Push Filters](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/programmable-push-filters/) and [Deserialization](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/record-deserialization/). 4. Enable the **Continuous Pagination** toggle. 5. Scroll the message list. Redpanda Cloud keeps loading records until you reach the end of the topic. When continuous pagination is on, the max results cap no longer limits the browsing session. ## [](#performance-considerations)Performance considerations Retrieving large result sets increases load on the Redpanda Cloud backend and the cluster. To keep responses fast: - Narrow the result set with filters or a bounded offset range before enabling continuous pagination. - Use [JavaScript push filters](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/programmable-push-filters/) to match only the records you need. - Leave continuous pagination off and rely on max results when you only need a sample. --- # Page 517: Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms.md --- # Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/index.adoc description: Learn about WebAssembly data transforms within Redpanda Cloud. page-git-created-date: "2025-04-08" page-git-modified-date: "2025-04-08" --- - [How Data Transforms Work](how-transforms-work/) Learn how Redpanda data transforms work. - [Develop Data Transforms](build/) Learn how to initialize a data transforms project and write transform functions in your chosen language. - [Configure Data Transforms](configure/) Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment. - [Deploy Data Transforms](deploy/) Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. - [Write Integration Tests for Transform Functions](test/) Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. - [Monitor Data Transforms](monitor/) This topic provides guidelines on how to monitor the health of your data transforms and view logs. - [Manage Data Transforms](data-transforms/) You can monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. --- # Page 518: Develop Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build.md --- # Develop Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Develop Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/build page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/build.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/build.adoc description: Learn how to initialize a data transforms project and write transform functions in your chosen language. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Data transforms are supported on BYOC and Dedicated clusters running Redpanda version 24.3 and later. > 💡 **TIP: When to use Redpanda Connect instead** > > Data transforms do not access external networks or disks, and are best for lightweight data preparation (filtering, scrubbing, schema/format conversion). Use [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) when you need any of the following: > > - External integration (HTTP services, databases, cloud storage) for enrichment or fan-out to third-party systems > > - Batching or windowed processing for grouping/aggregation > > - Prebuilt processors and connectors to reduce custom code Learn how to initialize a data transforms project and write transform functions in your chosen language. After reading this page, you will be able to: - Initialize a data transforms project using the rpk CLI - Build transform functions that process records and write to output topics - Implement multi-topic routing patterns with Schema Registry integration ## [](#prerequisites)Prerequisites You must have the following development tools installed on your host machine: - The [`rpk` command-line client](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) installed. - For Golang projects, you must have at least version 1.20 of [Go](https://go.dev/doc/install). - For Rust projects, you must have the latest stable version of [Rust](https://rustup.rs/). - For JavaScript and TypeScript projects, you must have the [latest long-term-support release of Node.js](https://nodejs.org/en/download/package-manager). ## [](#enable-data-transforms)Enable data transforms Data transforms are disabled on all clusters by default. Before you can deploy data transforms to a cluster, you must first enable the feature with the `rpk` command-line tool. To enable data transforms, set the [`data_transforms_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#data_transforms_enabled) cluster property to `true`: ```bash rpk cluster config set data_transforms_enabled true ``` > 📝 **NOTE** > > This property requires a rolling restart, and it can take several minutes for the update to complete. ## [](#init)Initialize a data transforms project To initialize a data transforms project, use the following command to set up the project files in your current directory. This command adds the latest version of the [SDK](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/sdks/) as a project dependency: ```bash rpk transform init --language= --name= ``` If you do not include the `--language` flag, the command prompts you for the language. Supported languages include: - `tinygo-no-goroutines` (does not include [Goroutines](https://golangdocs.com/goroutines-in-golang)) - `tinygo-with-goroutines` - `rust` - `javascript` - `typescript` For example, if you choose `tinygo-no-goroutines`, `rpk` creates the following project files: . ├── go.mod ├── go.sum ├── README.md ├── transform.go └── transform.yaml The `transform.go` file contains a boilerplate transform function. The `transform.yaml` file specifies the configuration settings for the transform function. See also: [Configure Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/) ## [](#build-transform-functions)Build transform functions You can develop your transform logic with one of the available SDKs that allow your transform code to interact with a Redpanda cluster. #### Go All transform functions must register a callback with the `OnRecordWritten()` method. You should run any initialization steps in the `main()` function because it’s only run once when the transform function is first deployed. You can also use the standard predefined [`init()` function](https://go.dev/doc/effective_go#init). ```go package main import ( "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Register your transform function. // This is a good place to perform other setup too. transform.OnRecordWritten(myTransform) } // myTransform is where you read the record that was written, and then you can // output new records that will be written to the destination topic func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust All transform functions must register a callback with the `on_record_written()` method. You should run any initialization steps in the `main()` function because it’s only run once when the transform function is first deployed. ```rust use redpanda_transform_sdk::*; fn main() { // Register your transform function. // This is a good place to perform other setup too. on_record_written(my_transform); } // my_transform is where you read the record that was written, and then you can // return new records that will be written to the output topic fn my_transform(event: WriteEvent, writer: &mut RecordWriter) -> Result<(), Box> { writer.write(event.record)?; Ok(()) } ``` #### JavaScript All transform functions must register a callback with the `onRecordWritten()` method. You should run any initialization steps outside of the callback so that they are only run once when the transform function is first deployed. ```js // src/index.js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // This is a good place to perform setup steps. // Register your transform function. onRecordWritten((event, writer) => { // This is where you read the record that was written, and then you can // output new records that will be written to the destination topic writer.write(event.record); }); ``` If you need to use Node.js standard modules in your transform function, you must configure the [`polyfillNode` plugin](https://github.com/cyco130/esbuild-plugin-polyfill-node) for [esbuild](https://esbuild.github.io/). This plugin allows you to polyfill Node.js APIs that are not natively available in the Redpanda JavaScript runtime environment. `esbuild.js` ```js import * as esbuild from 'esbuild'; import { polyfillNode } from 'esbuild-plugin-polyfill-node'; await esbuild.build({ plugins: [ polyfillNode({ globals: { buffer: true, // Allow a global Buffer variable if referenced. process: false, // Don't inject the process global, the Redpanda JavaScript runtime does that. }, polyfills: { crypto: true, // Enable crypto polyfill // Add other polyfills as needed }, }), ], }); ``` ### [](#errors)Error handling By distinguishing between recoverable and critical errors, you can ensure that your transform functions are both resilient and robust. Handling recoverable errors internally helps maintain continuous operation, while allowing critical errors to escape ensures that the system can address severe issues effectively. Redpanda tracks the offsets of records that transform functions have processed. If an error escapes the Wasm virtual machine (VM), the VM will fail. When the Wasm engine detects this failure and starts a new VM, the transform function retries processing the input topics from the last processed offset, potentially leading to repeated failures if the underlying issue is not resolved. Handling errors internally by logging them and continuing to process subsequent records can help maintain continuous operation. However, this approach can result in silently discarding problematic records, which may lead to unnoticed data loss if the logs are not monitored closely. #### Go ```go package main import ( "log" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(myTransform) } func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { record := event.Record() if record.Key == nil { // Handle the error internally by logging it log.Println("Error: Record key is nil") // Skip this record and continue to process other records return nil } // Allow errors with writes to escape return writer.Write(record) } ``` #### Rust ```rust use redpanda_transform_sdk::*; use log::error; fn main() { // Set up logging env_logger::init(); on_record_written(my_transform); } fn my_transform(event: WriteEvent, writer: &mut RecordWriter) -> anyhow::Result<()> { let record = event.record; if record.key().is_none() { // Handle the error internally by logging it error!("Error: Record key is nil"); // Skip this record and continue to process other records return Ok(()); } // Allow errors with writes to escape return writer.write(record) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Register your transform function. onRecordWritten((event, writer) => { const record = event.record; if (!record.key) { // Handle the error internally by logging it console.error("Error: Record key is nil"); // Skip this record and continue to process other records return; } // Allow errors with writes to escape writer.write(record); }); ``` When you deploy this transform function, and produce a message without a key, you’ll get the following in the logs: ```js { "body": { "stringValue": "2024/06/20 08:17:33 Error: Record key is nil\n" }, "timeUnixNano": 1718871455235337000, "severityNumber": 13, "attributes": [ { "key": "transform_name", "value": { "stringValue": "test" } }, { "key": "node", "value": { "intValue": 0 } } ] } ``` You can view logs for transform functions using the `rpk transform logs ` command. To ensure that you are notified of any errors or issues in your data transforms, Redpanda provides metrics that you can use to monitor the state of your data transforms. See also: - [View logs for transform functions](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor/#logs) - [Monitor data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor/) - [Configure transform logging](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#log) - [`rpk transform logs` reference](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform-logs/) ### [](#avoid-state-management)Avoid state management Relying on in-memory state across transform invocations can lead to inconsistencies and unpredictable behavior. Data transforms operate with at-least-once semantics, meaning a transform function might be executed more than once for a given record. Redpanda may also restart a transform function at any point, which causes its state to be lost. ### [](#env-vars)Access environment variables You can access both [built-in and custom environment variables](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#environment-variables) in your transform function. In this example, environment variables are checked once during initialization: #### Go ```go package main import ( "fmt" "os" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Check environment variables before registering the transform function. outputTopic1, ok := os.LookupEnv("REDPANDA_OUTPUT_TOPIC_1") if ok { fmt.Printf("Output topic 1: %s\n", outputTopic1) } else { fmt.Println("Only one output topic is set") } // Register your transform function. transform.OnRecordWritten(myTransform) } func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust ```rust use redpanda_transform_sdk::*; use std::env; use log::error; fn main() { // Set up logging env_logger::init(); // Check environment variables before registering the transform function. match env::var("REDPANDA_OUTPUT_TOPIC_1") { Ok(output_topic_1) => println!("Output topic 1: {}", output_topic_1), Err(_) => println!("Only one output topic is set"), } // Register your transform function. on_record_written(my_transform); } fn my_transform(_event: WriteEvent, _writer: &mut RecordWriter) -> anyhow::Result<()> { Ok(()) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Check environment variables before registering the transform function. const outputTopic1 = process.env.REDPANDA_OUTPUT_TOPIC_1; if (outputTopic1) { console.log(`Output topic 1: ${outputTopic1}`); } else { console.log("Only one output topic is set"); } // Register your transform function. onRecordWritten((event, writer) => { return writer.write(event.record); }); ``` ### [](#write-to-specific-output-topics)Write to specific output topics You can configure your transform function to write records to specific output topics based on message content, enabling powerful routing and fan-out patterns. This capability is useful for: - Filtering messages by criteria and routing to different topics - Fan-out patterns that distribute data from one input topic to multiple output topics - Event routing based on message type or schema - Data distribution for downstream consumers Wasm transforms provide a simpler alternative to external connectors like Kafka Connect for in-broker data routing, with lower latency and no additional infrastructure to manage. #### [](#basic-json-validation-example)Basic JSON validation example The following example shows a filter that outputs only valid JSON from the input topic into the output topic. The transform writes invalid JSON to a different output topic. ##### Go ```go import ( "encoding/json" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(filterValidJson) } func filterValidJson(event transform.WriteEvent, writer transform.RecordWriter) error { if json.Valid(event.Record().Value) { return writer.Write(event.Record()) } // Send invalid records to separate topic return writer.Write(event.Record(), transform.ToTopic("invalid-json")) } ``` ##### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { on_record_written(filter_valid_json); } fn filter_valid_json(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { let value = event.record.value().unwrap_or_default(); if serde_json::from_slice::(value).is_ok() { writer.write(event.record)?; } else { // Send invalid records to separate topic writer.write_with_options(event.record, WriteOptions::to_topic("invalid-json"))?; } Ok(()) } ``` ##### JavaScript The JavaScript SDK does not support writing records to a specific output topic. #### [](#multi-topic-fanout)Multi-topic fan-out with Schema Registry This example shows how to route batched updates from a single input topic to multiple output topics based on a routing field in each message. Messages are encoded with the [Schema Registry wire format](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/#wire-format) for validation against the output topic schema. Consider using this pattern with Iceberg-enabled topics to fan out data directly into lakehouse tables. Input message example ```json { "updates": [ {"table": "orders", "data": {"order_id": "123", "amount": 99.99}}, {"table": "inventory", "data": {"product_id": "P456", "quantity": 50}}, {"table": "customers", "data": {"customer_id": "C789", "name": "Jane"}} ] } ``` [Configure the transform](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/) with multiple output topics: ```yaml name: event-router input_topic: events output_topics: - orders - inventory - customers ``` The transform extracts each update and routes it to the appropriate topic based on the `table` field. Schemas are registered dynamically in the `main()` function using the Schema Registry client, which returns the schema IDs needed for encoding messages in the wire format. > 📝 **NOTE** > > In this example, it is assumed that you have created the output topics and have the schema definitions ready. The transform registers the schemas dynamically on startup using the `{topic-name}-value` naming convention for schema subjects (for example, `orders-value`, `inventory-value`). ##### Go `go.mod` ```go module fanout-example go 1.20 require github.com/redpanda-data/redpanda/src/transform-sdk/go/transform v1.1.0 // v1.1.0+ required ``` `transform.go`: ```go package main import ( "encoding/binary" "encoding/json" "log" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform/sr" ) // Input message structure with array of updates type BatchMessage struct { Updates []TableUpdate `json:"updates"` } // Individual table update with routing field type TableUpdate struct { Table string `json:"table"` // Routing field - determines output topic Data json.RawMessage `json:"data"` // The actual data to write } // Schema IDs for each output topic, registered dynamically at startup var schemaIDs = make(map[string]int) func main() { // Create Schema Registry client client := sr.NewClient() // Define schemas for each output topic schemas := map[string]string{ "orders": `{"type":"record","name":"Order","fields":[{"name":"order_id","type":"string"},{"name":"amount","type":"double"}]}`, "inventory": `{"type":"record","name":"Inventory","fields":[{"name":"product_id","type":"string"},{"name":"quantity","type":"int"}]}`, "customers": `{"type":"record","name":"Customer","fields":[{"name":"customer_id","type":"string"},{"name":"name","type":"string"}]}`, } // Register schemas and store their IDs for topic, schemaStr := range schemas { subject := topic + "-value" schema := sr.Schema{ Schema: schemaStr, Type: sr.TypeAvro, } result, err := client.CreateSchema(subject, schema) if err != nil { log.Fatalf("Failed to register schema for %s: %v", topic, err) } schemaIDs[topic] = result.ID log.Printf("Registered schema for %s with ID %d", topic, result.ID) } log.Printf("Starting fanout transform with schema IDs: %v", schemaIDs) transform.OnRecordWritten(routeUpdates) } func routeUpdates(event transform.WriteEvent, writer transform.RecordWriter) error { var batch BatchMessage if err := json.Unmarshal(event.Record().Value, &batch); err != nil { log.Printf("Failed to parse batch message: %v", err) return nil // Skip invalid records } // Process each update in the batch for i, update := range batch.Updates { schemaID, exists := schemaIDs[update.Table] if !exists { log.Printf("Unknown table in update %d: %s", i, update.Table) continue } if err := writeUpdate(update, schemaID, writer, event); err != nil { log.Printf("Failed to write update %d to %s: %v", i, update.Table, err) } } return nil } func writeUpdate(update TableUpdate, schemaID int, writer transform.RecordWriter, event transform.WriteEvent) error { // Create Schema Registry wire format: [magic_byte, schema_id (4 bytes BE), data...] value := make([]byte, 5) value[0] = 0 // magic byte binary.BigEndian.PutUint32(value[1:5], uint32(schemaID)) value = append(value, update.Data...) record := transform.Record{ Key: event.Record().Key, Value: value, } return writer.Write(record, transform.ToTopic(update.Table)) } ``` ##### Rust `Cargo.toml` ```toml [package] name = "fanout-rust-example" version = "0.1.0" edition = "2021" [dependencies] redpanda-transform-sdk = "1.1.0" # v1.1.0+ required for WriteOptions API redpanda-transform-sdk-sr = "1.1.0" serde = { version = "1", features = ["derive"] } serde_json = "1" log = "0.4" env_logger = "0.11" [profile.release] opt-level = "z" lto = true strip = true ``` `src/main.rs`: ```rust use redpanda_transform_sdk::*; use redpanda_transform_sdk_sr::{SchemaRegistryClient, Schema, SchemaFormat}; use serde::Deserialize; use std::collections::HashMap; use std::error::Error; use std::sync::OnceLock; use log::{info, error}; #[derive(Deserialize)] struct BatchMessage { updates: Vec, } #[derive(Deserialize)] struct TableUpdate { table: String, data: serde_json::Value, } // Schema IDs for each output topic, registered dynamically at startup static SCHEMA_IDS: OnceLock> = OnceLock::new(); fn main() { // Initialize logging env_logger::init(); // Create Schema Registry client let mut client = SchemaRegistryClient::new(); // Define schemas for each output topic let schemas = [ ("orders", r#"{"type":"record","name":"Order","fields":[{"name":"order_id","type":"string"},{"name":"amount","type":"double"}]}"#), ("inventory", r#"{"type":"record","name":"Inventory","fields":[{"name":"product_id","type":"string"},{"name":"quantity","type":"int"}]}"#), ("customers", r#"{"type":"record","name":"Customer","fields":[{"name":"customer_id","type":"string"},{"name":"name","type":"string"}]}"#), ]; let mut schema_ids = HashMap::new(); // Register schemas and store their IDs for (topic, schema_str) in schemas { let subject = format!("{}-value", topic); let schema = Schema::new(schema_str.to_string(), SchemaFormat::Avro, vec![]); match client.create_schema(&subject, schema) { Ok(result) => { let id = result.id(); // SchemaId type schema_ids.insert(topic.to_string(), id.0); // Extract i32 from SchemaId wrapper info!("Registered schema for {} with ID {}", topic, id.0); } Err(e) => { error!("Failed to register schema for {}: {}", topic, e); panic!("Schema registration failed"); } } } let _ = SCHEMA_IDS.set(schema_ids); info!("Starting fanout transform with schema IDs"); on_record_written(route_updates); } fn write_update( update: &TableUpdate, schema_id: i32, writer: &mut RecordWriter, event: &WriteEvent, ) -> Result<(), Box> { // Create Schema Registry wire format: [magic_byte, schema_id (4 bytes BE), data...] let mut value = vec![0u8; 5]; value[0] = 0; // magic byte value[1..5].copy_from_slice(&schema_id.to_be_bytes()); let data_bytes = serde_json::to_vec(&update.data)?; value.extend_from_slice(&data_bytes); let key = event.record.key().map(|k| k.to_vec()); let record = BorrowedRecord::new(key.as_deref(), Some(&value)); writer.write_with_options(record, WriteOptions::to_topic(&update.table))?; Ok(()) } fn route_updates(event: WriteEvent, writer: &mut RecordWriter) -> Result<(), Box> { let batch: BatchMessage = serde_json::from_slice(event.record.value().unwrap_or_default())?; let schema_ids = SCHEMA_IDS.get().unwrap(); for update in batch.updates.iter() { if let Some(&schema_id) = schema_ids.get(&update.table) { write_update(update, schema_id, writer, &event)?; } } Ok(()) } ``` ##### JavaScript The JavaScript SDK does not support writing records to specific output topics. For multi-topic fan-out, use the Go or Rust SDK. ### [](#connect-to-the-schema-registry)Connect to the Schema Registry You can use the Schema Registry client library to read and write schemas as well as serialize and deserialize records. This client library is useful when working with schema-based topics in your data transforms. See also: - [Redpanda Schema Registry](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/) - [Go Schema Registry client reference](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/golang-sdk/) - [Rust Schema Registry client reference](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/rust-sdk/) - [JavaScript Schema Registry client reference](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/js/js-sdk-sr/) ## [](#next-steps)Next steps [Configure Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/) ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/) - [Data Transforms SDKs](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/sdks/) - [`rpk transform` commands](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform/) --- # Page 519: Configure Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure.md --- # Configure Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/configure page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/configure.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/configure.adoc description: Learn how to configure data transforms in Redpanda, including editing the transform.yaml file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine’s environment. ## [](#configure-transform-functions)Configure transform functions This section covers how to configure transform functions using the `transform.yaml` configuration file, command-line overrides, and environment variables. ### [](#config-file)Transform configuration file When you [initialize](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/#init) a data transforms project, a `transform.yaml` file is generated in the provided directory. You can use this configuration file to configure the transform function with settings, including input and output topics, the language used for the data transform, and any environment variables. - `name`: The name of the transform function. - `description`: A description of what the transform function does. - `input-topic`: The topic from which data is read. - `output-topics`: A list of up to eight topics to which the transformed data is written. - `language`: The language used for the transform function. The language is set to the one you defined during [initialization](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/#init). - `env`: A dictionary of custom environment variables that are passed to the transform function. Do not prefix keys with `REDPANDA_`. Check the list of all [limitations](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/#limitations). Here is an example of a transform.yaml file: ```yaml name: redpanda-example description: | This transform function is an example to demonstrate how to configure data transforms in Redpanda. input-topic: example-input-topic output-topics: - example-output-topic-1 - example-output-topic-2 language: tinygo-no-goroutines env: DATA_TRANSFORMS_ARE_AWESOME: 'true' ``` ### [](#cl)Override configurations with command-line options You can set the name of the transform function, environment variables, and input and output topics on the command-line when you deploy the transform. These command-line settings take precedence over those specified in the `transform.yaml` file. See [Deploy Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/deploy/) ### [](#built-in)Built-In environment variables As well as custom environment variables set in either the [command-line](#cl) or the [configuration file](#config-file), Redpanda makes some built-in environment variables available to your transform functions. These variables include: - `REDPANDA_INPUT_TOPIC`: The input topic specified. - `REDPANDA_OUTPUT_TOPIC_0..REDPANDA_OUTPUT_TOPIC_N`: The output topics in the order specified on the command line or in the configuration file. For example, `REDPANDA_OUTPUT_TOPIC_0` is the first variable, `REDPANDA_OUTPUT_TOPIC_1` is the second variable, and so on. Transform functions are isolated from the broker’s internal environment variables to maintain security and encapsulation. Each transform function only uses the environment variables explicitly provided to it. ## [](#configure-the-wasm-engine)Configure the Wasm engine This section covers how to configure the Wasm engine environment using Redpanda cluster configuration properties. ### [](#enable-transforms)Enable data transforms To use data transforms, you must enable it for a Redpanda cluster using the [`data_transforms_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#data_transforms_enabled) property. ### [](#log)Configure transform logging The following properties configure logging for data transforms: - [`data_transforms_logging_line_max_bytes`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#data_transforms_logging_line_max_bytes): Increase this value if your log messages are frequently truncated. Setting this value too low may truncate important log information. ## [](#next-steps)Next steps [Deploy Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/deploy/) --- # Page 520: Manage Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/data-transforms.md --- # Manage Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/data-transforms page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/data-transforms.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/data-transforms.adoc description: You can monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- You can monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. ## [](#prerequisites)Prerequisites Before you begin, ensure that you have the following: - [Data transforms enabled](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#enable-transforms) in your Redpanda cluster. - At least one transform function deployed to your Redpanda cluster. ## [](#monitor)Monitor transform functions To monitor transform functions: 1. Navigate to the **Transforms** menu. 2. Click the name of a transform function to view detailed information: - The partitions that the function is running on - The broker (node) ID - Any lag (the amount of pending records on the input topic that have yet to be processed by the transform) ## [](#logs)View logs To view logs for a transform function: 1. Navigate to the **Transforms** menu. 2. Click on the name of a transform function. 3. Click the **Logs** tab to see the logs. Redpanda Cloud displays a limited number of logs for transform functions. To view the full history of logs, use the [`rpk` command-line tool](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor/#logs). ## [](#delete)Delete transform functions To delete a transform function: 1. Navigate to the **Transforms** menu. 2. Find the transform function you want to delete from the list. 3. Click the delete icon at the end of the row. 4. Confirm the deletion when prompted. Deleting a transform function will remove it from the cluster and stop any further processing. ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/) - [Deploy Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/deploy/) - [Monitor Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor/) --- # Page 521: Deploy Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/deploy.md --- # Deploy Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Deploy Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/deploy page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/deploy.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/deploy.adoc description: Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. ## [](#prerequisites)Prerequisites Before you begin, ensure that you have the following: - [Data transforms enabled](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#enable-transforms) in your Redpanda cluster. - The [`rpk` command-line client](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - A [data transform](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/) project. ## [](#build)Build the Wasm binary To build a Wasm binary: 1. Ensure your project directory contains a `transform.yaml` file. 2. Build the Wasm binary using the [`rpk transform build`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform-build/) command. ```bash rpk transform build ``` You should now have a Wasm binary named `.wasm`, where `` is the name specified in your `transform.yaml` file. This binary is your data transform function, ready to be deployed to a Redpanda cluster or hosted on a network for others to use. ## [](#deploy)Deploy the Wasm binary You can deploy your transform function using the [`rpk transform deploy`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform-deploy/) command. 1. Validate your setup against the pre-deployment checklist: - Do you meet the [Prerequisites](#prerequisites)? - Does your transform function access any environment variables? If so, make sure to set them in the `transform.yaml` file or in the command-line when you deploy the binary. - Do your configured input and output topics already exist? Input and output topics must exist in your Redpanda cluster before you deploy the Wasm binary. 2. Deploy the Wasm binary: ```bash rpk transform deploy ``` When the transform function reaches Redpanda, it starts processing new records that are written to the input topic. ### [](#reprocess)Reprocess records In some cases, you may need to reprocess records from an input topic that already contains data. Processing existing records can be useful, for example, to process historical data into a different format for a new consumer, to re-create lost data from a deleted topic, or to resolve issues with a previous version of a transform that processed data incorrectly. To reprocess records, you can specify the starting point from which the transform function should process records in each partition of the input topic. The starting point can be either a partition offset or a timestamp. > 📝 **NOTE** > > The `--from-offset` flag is only effective the first time you deploy a transform function. On subsequent deployments of the same function, Redpanda resumes processing from the last committed offset. To reprocess existing records using an existing function, [delete the function](#delete) and redeploy it with the `--from-offset` flag. To deploy a transform function and start processing records from a specific partition offset, use the following syntax: ```bash rpk transform deploy --from-offset +/- ``` In this example, the transform function will start processing records from the beginning of each partition of the input topic: ```bash rpk transform deploy --from-offset +0 ``` To deploy a transform function and start processing records from a specific timestamp, use the following syntax: ```bash rpk transform deploy --from-timestamp @ ``` In this example, the transform function will start processing from the first record in each partition of the input topic that was committed after the given timestamp: ```bash rpk transform deploy --from-timestamp @1617181723 ``` ### [](#share-wasm-binaries)Share Wasm binaries You can also deploy data transforms on a Redpanda cluster by providing an addressable path to the Wasm binary. This is useful for sharing transform functions across multiple clusters or teams within your organization. For example, if the Wasm binary is hosted at `https://my-site/my-transform.wasm`, use the following command to deploy it: ```bash rpk transform deploy --file=https://my-site/my-transform.wasm ``` ## [](#edit-existing-transform-functions)Edit existing transform functions To make changes to an existing transform function: 1. [Make your changes to the code](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/). 2. [Rebuild](#build) the Wasm binary. 3. [Redeploy](#deploy) the Wasm binary to the same Redpanda cluster. When you redeploy a Wasm binary with the same name, it will resume processing from the last offset it had previously processed. If you need to [reprocess existing records](#reprocess), you must delete the transform function, and redeploy it with the `--from-offset` flag. Deploy-time configuration overrides must be provided each time you redeploy a Wasm binary. Otherwise, they will be overwritten by default values or the configuration file’s contents. ## [](#delete)Delete a transform function To delete a transform function, use the following command: ```bash rpk transform delete ``` For more details about this command, see [rpk transform delete](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform-delete/). > 💡 **TIP** > > You can also delete transform functions in Redpanda Cloud. ## [](#troubleshoot)Troubleshoot This section provides guidance on how to diagnose and troubleshoot issues with building or deploying data transforms. ### [](#invalid-transform-environment)Invalid transform environment This error means that one or more of your configured custom environment variables are invalid. Check your custom environment variables against the list of [limitations](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/#limitations). ### [](#invalid-webassembly)Invalid WebAssembly This error indicates that the binary is missing a required callback function: Invalid WebAssembly - the binary is missing required transform functions. Check the broker support for the version of the data transforms SDK being used. All transform functions must register a callback with the `OnRecordWritten()` method. For more details, see [Develop Data Transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/). ## [](#next-steps)Next steps [Set up monitoring](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor/) for data transforms. --- # Page 522: How Data Transforms Work **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work.md --- # How Data Transforms Work > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: How Data Transforms Work latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/how-transforms-work page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/how-transforms-work.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/how-transforms-work.adoc description: Learn how Redpanda data transforms work. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Data transforms are supported on BYOC and Dedicated clusters running Redpanda version 24.3 and later. Redpanda provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. Redpanda does this directly inside the broker, eliminating the need to manage a separate stream processing environment or use third-party tools. ![Data transforms in a broker](https://docs.redpanda.com/cloud-data-platform/shared/_images/wasm1.png) Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types. To learn how to build and deploy data transforms, see [How Data Transforms Work](./). ## [](#data-transforms-with-webassembly)Data transforms with WebAssembly Data transforms use [WebAssembly](https://webassembly.org/) (Wasm) engines inside a Redpanda broker, allowing Redpanda to control the entire transform lifecycle. For example, Redpanda can stop and start transforms when partitions are moved or to free up system resources for other tasks. Data transforms take data from an input topic and map it to one or more output topics. For each topic partition, a leader is responsible for handling the data. Redpanda runs a Wasm virtual machine (VM) on the same CPU core (shard) as these partition leaders to execute the transform function. Transform functions are the specific implementations of code that carry out the transformations. They read data from input topics, apply the necessary processing logic, and write the transformed data to output topics. To execute a transform function, Redpanda uses just-in-time (JIT) compilation to compile the bytecode in memory, write it to an executable space, then run the directly translated machine code. This JIT compilation ensures efficient execution of the machine code, as it is tailored to the specific hardware it runs on. When you deploy a data transform to a Redpanda broker, it stores the Wasm bytecode and associated metadata, such as input and output topics and environment variables. The broker then replicates this data across the cluster using internal Kafka topics. When the data is distributed, each shard runs its own instance of the transform function. This process includes several resource management features: - Each shard can run only one instance of the transform function at a time to ensure efficient resource utilization and prevent overload. - CPU time is dynamically allocated to the Wasm runtime to ensure that the code does not run forever and cannot block the broker from handling traffic or doing other work, such as Tiered Storage uploads. ## [](#flow-of-data-transforms)Flow of data transforms When a shard becomes the leader of a given partition on the input topic of one or more active transforms, Redpanda does the following: 1. Spins up a Wasm VM using the JIT-compiled Wasm module. 2. Pushes records from the input partition into the Wasm VM. 3. Writes the output. The output partition may exist on the same broker or on another broker in the cluster. Within Redpanda, a single Raft controller manages cluster information, including data transforms. On every shard, Redpanda knows what data transforms exist in the cluster, as well as metadata about the transform function, such as input and output topics and environment variables. ![Wasm architecture in Redpanda](https://docs.redpanda.com/cloud-data-platform/shared/_images/wasm_architecture.png) Each transform function reads from a specified input topic and writes to a specified output topic. The transform function processes every record produced to an input topic and returns zero or more records that are then produced to the specified output topic. Data transforms are applied to all partitions on an input topic. A record is processed after it has been successfully written to disk on the input topic. Because the transform happens in the background after the write finishes, the transform doesn’t affect the original produced record, doesn’t block writes to the input topic, and doesn’t block produce and consume requests. A new transform function reads the input topic from the latest offset. That is, it only reads new data produced to the input topic: it does not read records produced to the input topic before the transform was deployed. If a partition leader moves from one broker to another, then the instance of the transform function assigned to that partition moves with it. When a partition replica [loses leadership](https://docs.redpanda.com/cloud-data-platform/get-started/architecture/#partition-leadership-elections), the broker hosting that partition replica stops the instance of the transform function running on the same shard. The broker that is now hosting the partition’s new leader starts the transform function on the same shard as that leader, and the transform function resumes from the last committed offset. If the previous instance of the transform function failed to commit its latest offsets before moving with the partition leader (for example, if the broker crashed), then it’s likely that the new instance will reprocess some events. For broker failures, transform functions have at-least-once semantics, because records are retried from the committed last offset, and offsets are committed periodically. For more information, see [How Data Transforms Work](./). ## [](#limitations)Limitations This section outlines the limitations of data transforms. These constraints are categorized into general limitations affecting the overall functionality and specific limitations related to giving data transforms access to custom environment variables. ### [](#general)General - **No external access**: Transform functions have no external access to disk or network resources. - **Single message transforms**: Only single record transforms are supported, but multiple output records from a single input record are supported. For aggregations, joins, or complex transformations, consider using [Redpanda Connect](https://docs.redpanda.com/connect/get-started/about/) or [Apache Flink](https://flink.apache.org/). - **Output topic limit**: Up to eight output topics are supported. - **Delivery semantics**: Transform functions have at-least-once delivery. - **Transactions API**: When clients use the Kafka Transactions API on partitions of an input topic, transform functions process only committed records. ### [](#javascript)JavaScript - **No native extensions**: Native Node.js extensions are not supported. Packages that require compiling native code or interacting with low-level system features cannot be used. - **Limited Node.js standard modules**: Only modules that can be polyfilled by the [esbuild plugin](https://www.npmjs.com/package/esbuild-plugin-polyfill-node#implemented-polyfills) can be used. Even if a module can be polyfilled, certain functionalities, such as network connections, will not work because the necessary browser APIs are not exposed in the Redpanda JavaScript runtime environment. For example, while the plugin can provide stubs for some Node.js modules such as `http` and `process`, these stubs will not work in the Redpanda JavaScript runtime environment. - **No write options**: The JavaScript SDK does not support write options, such as specifying which output topic to write to. ### [](#environment-variables)Environment variables - **Maximum number of variables**: You can set up to 128 custom environment variables. - **Reserved prefix**: Variable keys must not start with `REDPANDA_`. This prefix is reserved for [built-in environment variables](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#built-in). - **Key length**: Each key must be less than 128 bytes in length. - **Total value length**: The combined length of all values for the environment variables must be less than 2000 bytes. - **Encoding**: All keys and values must be encoded in UTF-8. - **Control characters**: Keys and values must not contain any control characters, such as null bytes. ## [](#suggested-reading)Suggested reading - [Golang SDK for Data Transforms](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/golang-sdk/) - [Rust SDK for Data Transforms](https://docs.redpanda.com/cloud-data-platform/reference/data-transforms/rust-sdk/) - [`rpk transform` commands](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform/) --- # Page 523: Monitor Data Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/monitor.md --- # Monitor Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Monitor Data Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/monitor page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/monitor.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/monitor.adoc description: This topic provides guidelines on how to monitor the health of your data transforms and view logs. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- This topic provides guidelines on how to monitor the health of your data transforms and view logs. ## [](#prerequisites)Prerequisites [Set up monitoring](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/) for your cluster. ## [](#performance)Performance You can identify performance bottlenecks by monitoring latency and CPU usage: - [`redpanda_transform_execution_latency_sec`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_execution_latency_sec) - [`redpanda_wasm_engine_cpu_seconds_total`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_wasm_engine_cpu_seconds_total) If latency is high, investigate the transform logic for inefficiencies or consider scaling the resources. High CPU usage might indicate the need for optimization in the code or an increase in [allocated CPU resources](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/). ## [](#reliability)Reliability Tracking execution errors and error states helps in maintaining the reliability of your data transforms: - [`redpanda_transform_execution_errors`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_execution_errors) - [`redpanda_transform_failures`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_failures) - [`redpanda_transform_state`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_state) Make sure to [implement robust error handling and logging](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/#errors) within your transform functions to help with troubleshooting. ## [](#resource-usage)Resource usage Monitoring memory usage metrics and total execution time ensures that the Wasm engine does not exceed allocated resources, helping in efficient resource management: - [`redpanda_wasm_engine_memory_usage`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_wasm_engine_memory_usage) - [`redpanda_wasm_engine_max_memory`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_wasm_engine_max_memory) - [`redpanda_wasm_binary_executable_memory_usage`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_wasm_binary_executable_memory_usage) If memory usage is consistently high or exceeds the maximum allocated memory: - Review and optimize your transform functions to reduce memory consumption. This step can involve optimizing data structures, reducing memory allocations, and ensuring efficient handling of records. ## [](#throughput)Throughput Keeping track of read and write bytes and processor lag helps in understanding the data flow through your transforms, enabling better capacity planning and scaling: - [`redpanda_transform_read_bytes`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_read_bytes) - [`redpanda_transform_write_bytes`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_write_bytes) - [`redpanda_transform_processor_lag`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_transform_processor_lag) If there is a significant lag or low throughput, investigate potential bottlenecks in the data flow or consider scaling your infrastructure to handle higher throughput. ## [](#logs)View logs for data transforms Runtime logs for transform functions are written to an internal topic called `_redpanda.transform_logs`. You can read these logs by using the [`rpk transform logs`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-transform/rpk-transform-logs/) command. ```bash rpk transform logs ``` Replace `` with the [configured name](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/) of the transform function. > 💡 **TIP** > > You can also view logs in the UI. By default, Redpanda provides several settings to manage logging for data transforms, such as buffer capacity, flush interval, and maximum log line length. These settings ensure that logging operates efficiently without overwhelming the system. However, you may need to adjust these settings based on your specific requirements and workloads. For information on how to configure logging, see the [Configure transform logging](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/configure/#log) section of the configuration guide. ## [](#suggested-reading)Suggested reading - [Data transforms metrics](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#data_transform_metrics) --- # Page 524: Write Integration Tests for Transform Functions **URL**: https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/test.md --- # Write Integration Tests for Transform Functions > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Write Integration Tests for Transform Functions latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: data-transforms/test page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: data-transforms/test.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/data-transforms/test.adoc description: Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. This guide covers how to write both unit tests and integration tests for your transform functions. While unit tests focus on testing individual components in isolation, integration tests verify that the components work together as expected in a real environment. ## [](#unit-tests)Unit tests You can create unit tests for transform functions by mocking the interfaces injected into the transform function and asserting that the input and output work correctly. This typically includes mocking the `WriteEvent` and `RecordWriter` interfaces. ```go package main import ( "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) // MockWriteEvent is a mock implementation of the WriteEvent interface. type MockWriteEvent struct { mock.Mock } func (m *MockWriteEvent) Record() transform.Record { args := m.Called() return args.Get(0).(transform.Record) } // MockRecordWriter is a mock implementation of the RecordWriter interface. type MockRecordWriter struct { mock.Mock } func (m *MockRecordWriter) Write(record transform.Record) error { args := m.Called(record) return args.Error(0) } // copyRecord copies the record to the output topic. func copyRecord(event transform.WriteEvent, writer transform.RecordWriter) error { record := event.Record() return writer.Write(record) } // TestCopyRecord tests the copyRecord function. func TestCopyRecord(t *testing.T) { // Create mocks for the WriteEvent and RecordWriter event := new(MockWriteEvent) writer := new(MockRecordWriter) // Set up the expected behavior record := transform.Record{Value: []byte("test")} event.On("Record").Return(record) writer.On("Write", record).Return(nil) // Call the function under test err := copyRecord(event, writer) // Assert that no error occurred and that the expectations were met assert.NoError(t, err) event.AssertExpectations(t) writer.AssertExpectations(t) } ``` To run your unit tests, use the following command: ```bash go test ``` This will execute all tests in the current directory. ## [](#integration-tests)Integration tests Integration tests verify that your transform functions work correctly in a real Redpanda environment. You can use [testcontainers](https://github.com/testcontainers/testcontainers-go/tree/main) to set up and manage a Redpanda instance for testing. For more detailed examples and helper code for setting up integration tests, refer to the SDK integration tests on [GitHub](https://github.com/redpanda-data/redpanda/tree/dev/src/transform-sdk/tests). --- # Page 525: Use Redpanda with the HTTP Proxy API **URL**: https://docs.redpanda.com/cloud-data-platform/develop/http-proxy.md --- # Use Redpanda with the HTTP Proxy API > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use Redpanda with the HTTP Proxy API latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: http-proxy page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: http-proxy.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/http-proxy.adoc description: HTTP Proxy exposes a REST API to list topics, produce events, and subscribe to events from topics using consumer groups. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Redpanda HTTP Proxy (`pandaproxy`) allows access to your data through a REST API. For example, you can list topics or brokers, get events, produce events, subscribe to events from topics using consumer groups, and commit offsets for a consumer. See the [HTTP Proxy API reference](https://docs.redpanda.com/api/doc/http-proxy/) for a full list of available endpoints. > 📝 **NOTE** > > The HTTP Proxy API is supported for BYOC and Dedicated clusters only. ## [](#prerequisites)Prerequisites ### [](#start-redpanda)Start Redpanda To log in to your Redpanda Cloud account, run `rpk cloud login`. HTTP Proxy is enabled by default on port 30082. For clusters with private connectivity (AWS PrivateLink, GCP Private Service Connect, and Azure Private Link) enabled, the default seed port for HTTP Proxy is 30282. You can find the HTTP Proxy endpoint on the **How to connect** section of the cluster overview in the Cloud UI. > 📝 **NOTE** > > The rest of this guide assumes that the HTTP Proxy port is `30082`. ## [](#authenticate-with-http-proxy)Authenticate with HTTP Proxy HTTP Proxy supports authentication using SCRAM credentials or OIDC tokens. The authentication method depends on the cluster’s [`http_authentication`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#http_authentication) settings. ### [](#scram-authentication)SCRAM Authentication If HTTP Proxy is configured to support SASL, you can provide the SCRAM username and password as part of the Basic Authentication header in your request. For example, to list topics as an authenticated user: #### curl ```bash curl -s -u ":" "http://:30082/topics" ``` #### NodeJS ```javascript let options = { auth: { username: "", password: "" }, }; axios .get("http://:30082/topics", options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python auth = ("", "") res = requests.get("http://:30082/topics", auth=auth).json() pretty(res) ``` ### [](#oidc-authentication)OIDC Authentication If HTTP Proxy is configured to support OIDC, you can provide an OIDC token in the Authorization header. For example: #### curl ```bash curl -s -H "Authorization: Bearer " "http://:30082/topics" ``` #### NodeJS ```javascript let options = { headers: { Authorization: `Bearer ` }, }; axios .get("http://:30082/topics", options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python headers = {"Authorization": "Bearer "} res = requests.get("http://:30082/topics", headers=headers).json() pretty(res) ``` ## [](#set-up-libraries)Set up libraries You need an app that calls the HTTP Proxy endpoint. This app can be curl (or a similar CLI), or it could be your own custom app written in any language. Below are curl, JavaScript and Python examples. > 📝 **NOTE** > > In the examples, `` refers to your Redpanda cluster’s hostname or IP address. All following examples use a `base_uri` variable that combines the protocol, host, and port for consistency across curl, JavaScript, and Python examples. ### curl Curl is likely already installed on your system. If not, see [curl download instructions](https://curl.se/download.html). Set the base URI for your HTTP Proxy: ```bash base_uri="http://:30082" ``` ### NodeJS > 📝 **NOTE** > > This is based on the assumption that you’re in the root directory of an existing NodeJS project. See [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) for an example of a NodeJS project. In a terminal window, run: ```bash npm install axios ``` Import the library into your code: ```javascript const axios = require('axios'); const base_uri = 'http://:30082'; ``` ### Python In a terminal window, run: ```bash pip install requests ``` Import the library into your code: ```python import requests import json def pretty(text): print(json.dumps(text, indent=2)) base_uri = "http://:30082" ``` ## [](#create-a-topic)Create a topic To create a test topic for this guide, use [`rpk`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). You can configure `rpk` for your Redpanda deployment, using [profiles](https://docs.redpanda.com/cloud-data-platform/manage/rpk/config-rpk-profile/), flags, or [environment variables](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-x-options/#environment-variables). To create a topic named `test_topic` with three partitions, run: ```bash rpk topic create test_topic -p 3 ``` For more information, see the [rpk topic create](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-topic/rpk-topic-create/) reference. ## [](#access-your-data)Access your data Here are some sample commands to produce and consume streams: ### [](#get-list-of-topics)Get list of topics #### curl ```bash curl -s "$base_uri/topics" ``` #### NodeJS ```javascript axios .get(`${base_uri}/topics`) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application. If your file name is `index.js` for example, you would run the following command: ```bash node index.js ``` #### Python ```python res = requests.get(f"{base_uri}/topics").json() pretty(res) ``` Expected output: ```bash ["test_topic"] ``` ### [](#send-events-to-a-topic)Send events to a topic Use POST to send events in the REST endpoint query. The header must include the following line: Content-Type:application/vnd.kafka.json.v2+json The following commands show how to send events to `test_topic`: #### curl ```bash curl -s \ -X POST \ "$base_uri/topics/test_topic" \ -H "Content-Type: application/vnd.kafka.json.v2+json" \ -d '{ "records":[ { "value":"Redpanda", "partition":0 }, { "value":"HTTP proxy", "partition":1 }, { "value":"Test event", "partition":2 } ] }' ``` #### NodeJS ```javascript let payload = { records: [ { "value":"Redpanda", "partition": 0 }, { "value":"HTTP proxy", "partition": 1 }, { "value":"Test event", "partition": 2 } ]}; let options = { headers: { "Content-Type" : "application/vnd.kafka.json.v2+json" }}; axios .post(`${base_uri}/topics/test_topic`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/topics/test_topic", data=json.dumps( dict(records=[ dict(value="Redpanda", partition=0), dict(value="HTTP Proxy", partition=1), dict(value="Test Event", partition=2) ])), headers={"Content-Type": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output (may be formatted differently depending on the chosen application): ```bash {"offsets":[{"partition":0,"offset":0},{"partition":2,"offset":0},{"partition":1,"offset":0}]} ``` ### [](#get-events-from-a-topic)Get events from a topic After events have been sent to the topic, you can retrieve these same events. #### curl ```bash curl -s \ "$base_uri/topics/test_topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\ -H "Accept: application/vnd.kafka.json.v2+json" ``` #### NodeJS ```javascript let options = { headers: { accept: "application/vnd.kafka.json.v2+json" }, params: { offset: 0, timeout: "1000", max_bytes: "100000", }, }; axios .get(`${base_uri}/topics/test_topic/partitions/0/records`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.get( url=f"{base_uri}/topics/test_topic/partitions/0/records", params={"offset": 0, "timeout":1000,"max_bytes":100000}, headers={"Accept": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output: ```bash [{"topic":"test_topic","key":null,"value":"Redpanda","partition":0,"offset":0}] ``` ### [](#get-list-of-brokers)Get list of brokers #### curl ```bash curl "$base_uri/brokers" ``` #### NodeJS ```javascript axios .get(`${base_uri}/brokers`) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python res = requests.get(f"{base_uri}/brokers").json() pretty(res) ``` Expected output: ```bash {brokers: [0]} ``` ### [](#create-a-consumer)Create a consumer To retrieve events from a topic using consumers, you must create a consumer and a consumer group, and then subscribe the consumer instance to a topic. Each action involves a different endpoint and method. The first endpoint is: `/consumers/`. For this REST call, the payload is the group information. #### curl ```bash curl -s \ -X POST \ "$base_uri/consumers/test_group" \ -H "Content-Type: application/vnd.kafka.v2+json" \ -d '{ "format":"json", "name":"test_consumer", "auto.offset.reset":"earliest", "auto.commit.enable":"false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }' ``` #### NodeJS ```javascript let payload = { "name": "test_consumer", "format": "json", "auto.offset.reset": "earliest", "auto.commit.enable": "false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }; let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .post(`${base_uri}/consumers/test_group`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group", data=json.dumps({ "name": "test_consumer", "format": "json", "auto.offset.reset": "earliest", "auto.commit.enable": "false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }), headers={"Content-Type": "application/vnd.kafka.v2+json"}).json() pretty(res) ``` Expected output: ```bash {"instance_id":"test_consumer","base_uri":"http://:30082/consumers/test_group/instances/test_consumer"} ``` > 📝 **NOTE** > > - Consumers expire after five minutes of inactivity. To prevent this from happening, try consuming events within a loop. If the consumer has expired, you can create a new one with the same name. > > - The output `base_uri` is the full URL path for this specific consumer instance and differs from the `base_uri` variable used in the code examples. ### [](#subscribe-to-the-topic)Subscribe to the topic After creating the consumer, subscribe to the topic that you created. #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X POST \ "$base_uri/consumers/test_group/instances/test_consumer/subscription"\ -H "Content-Type: application/vnd.kafka.v2+json" \ -d '{ "topics": [ "test_topic" ] }' ``` #### NodeJS ```javascript let payload = { topics: ["test_topic"]}; let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .post(`${base_uri}/consumers/test_group/instances/test_consumer/subscription`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group/instances/test_consumer/subscription", data=json.dumps({"topics": ["test_topic"]}), headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` Expected response is an HTTP 204, without a body. Now you can get the events from `test_topic`. ### [](#retrieve-events)Retrieve events Retrieve the events from the topic: #### curl ```bash curl -s \ "$base_uri/consumers/test_group/instances/test_consumer/records?timeout=1000&max_bytes=100000"\ -H "Accept: application/vnd.kafka.json.v2+json" ``` #### NodeJS ```javascript let options = { headers: { Accept: "application/vnd.kafka.json.v2+json" }, params: { timeout: "1000", max_bytes: "100000", }, }; axios .get(`${base_uri}/consumers/test_group/instances/test_consumer/records`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.get( url=f"{base_uri}/consumers/test_group/instances/test_consumer/records", params={"timeout":1000,"max_bytes":100000}, headers={"Accept": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output: ```bash [{"topic":"test_topic","key":null,"value":"Redpanda","partition":0,"offset":0},{"topic":"test_topic","key":null,"value":"HTTP proxy","partition":1,"offset":0},{"topic":"test_topic","key":null,"value":"Test event","partition":2,"offset":0}] ``` ### [](#get-offsets-from-consumer)Get offsets from consumer #### curl ```bash curl -s \ -X 'GET' \ curl -s -o /dev/null -w "%{http_code}" \ -X 'POST' \ "$base_uri/consumers/test_group/instances/test_consumer/offsets" \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'Content-Type: application/vnd.kafka.v2+json' \ -d '{ "partitions": [ { "topic": "test_topic", "partition": 0 }, { "topic": "test_topic", "partition": 1 }, { "topic": "test_topic", "partition": 2 } ] }' ``` #### Python ```python res = requests.get( url=f"{base_uri}/consumers/test_group/instances/test_consumer/offsets", data=json.dumps( dict(partitions=[ dict(topic="test_topic", partition=p) for p in [0, 1, 2] ])), headers={"Content-Type": "application/vnd.kafka.v2+json"}).json() pretty(res) ``` Expected output: ```bash { "offsets": [{ "topic": "test_topic", "partition": 0, "offset": 0, "metadata": "" },{ "topic": "test_topic", "partition": 1, "offset": 0, "metadata": "" }, { "topic": "test_topic", "partition": 2, "offset": 0, "metadata": "" }] } ``` ### [](#commit-offsets-for-consumer)Commit offsets for consumer After events have been handled by a consumer, the offsets can be committed, so that the consumer group won’t retrieve them again. #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X 'POST' \ "$base_uri/consumers/test_group/instances/test_consumer/offsets" \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'Content-Type: application/vnd.kafka.v2+json' \ -d '{ "partitions": [ { "topic": "test_topic", "partition": 0, "offset": 0 }, { "topic": "test_topic", "partition": 1, "offset": 0 }, { "topic": "test_topic", "partition": 2, "offset": 0 } ] }' ``` #### NodeJS ```javascript let options = { headers: { accept: "application/vnd.kafka.v2+json", "Content-Type": "application/vnd.kafka.v2+json", } }; let payload = { partitions: [ { topic: "test_topic", partition: 0, offset: 0 }, { topic: "test_topic", partition: 1, offset: 0 }, { topic: "test_topic", partition: 2, offset: 0 }, ]}; axios .post(`${base_uri}/consumers/test_group/instances/test_consumer/offsets`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group/instances/test_consumer/offsets", data=json.dumps( dict(partitions=[ dict(topic="test_topic", partition=p, offset=0) for p in [0, 1, 2] ])), headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` Expected output: none. ### [](#delete-a-consumer)Delete a consumer To remove a consumer from a group, send a DELETE request as shown below: #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X 'DELETE' \ "$base_uri/consumers/test_group/instances/test_consumer" \ -H 'Content-Type: application/vnd.kafka.v2+json' ``` #### NodeJS ```javascript let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .delete(`${base_uri}/consumers/test_group/instances/test_consumer`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python res = requests.delete( url=f"{base_uri}/consumers/test_group/instances/test_consumer", headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` ## [](#authenticate-with-http-proxy-2)Authenticate with HTTP Proxy HTTP Proxy supports authentication using SCRAM credentials or OIDC tokens. The authentication method depends on the cluster’s [`http_authentication`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#http_authentication) settings. ### [](#scram-authentication-2)SCRAM Authentication If HTTP Proxy is configured to support SASL, you can provide the SCRAM username and password as part of the Basic Authentication header in your request. For example, to list topics as an authenticated user: #### curl ```bash curl -s -u ":" ":8082/topics" ``` #### NodeJS ```javascript let options = { auth: { username: "", password: "" }, }; axios .get(`${base_uri}/topics`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python auth = ("", "") res = requests.get(f"{base_uri}/topics", auth=auth).json() pretty(res) ``` ### [](#oidc-authentication-2)OIDC Authentication If HTTP Proxy is configured to support OIDC, you can provide an OIDC token in the Authorization header. For example: #### curl ```bash curl -s -H "Authorization: Bearer " ":8082/topics" ``` #### NodeJS ```javascript let options = { headers: { Authorization: `Bearer ` }, }; axios .get(`${base_uri}/topics`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python headers = {"Authorization": "Bearer "} res = requests.get(f"{base_uri}/topics", headers=headers).json() pretty(res) ``` ## [](#use-swagger-with-http-proxy)Use Swagger with HTTP Proxy You can use Swagger UI to test and interact with Redpanda HTTP Proxy endpoints. Use Docker to start Swagger UI: ```bash docker run -p 80:8080 -d swaggerapi/swagger-ui ``` Verify that the Swagger container is available: ```bash docker ps ``` Verify that the Docker container has been added and is running: `swaggerapi/swagger-ui` with `Up…` status In a browser, enter `` in the address bar to open the Swagger console. Change the URL to `[http://:30082/v1](http://\:30082/v1)`, and click `Explore` to update the page with Redpanda HTTP Proxy endpoints. You can call the endpoints in any application and language that supports web interactions. --- # Page 526: Kafka Compatibility **URL**: https://docs.redpanda.com/cloud-data-platform/develop/kafka-clients.md --- # Kafka Compatibility > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Kafka Compatibility latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: kafka-clients page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: kafka-clients.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/kafka-clients.adoc description: Kafka clients, version 0.11 or later, are compatible with Redpanda. Validations and exceptions are listed. page-topic-type: reference personas: developer learning-objective-1: Identify which Kafka clients are validated with Redpanda learning-objective-2: Identify unsupported Kafka features when integrating with Redpanda page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-29" --- Apache Kafka® clients developed for Kafka protocol version 0.11 or later work with Redpanda with minimal or no changes to your application. This page identifies which clients are validated and calls out any exceptions. Use this reference to: - Identify which Kafka clients are validated with Redpanda - Identify unsupported Kafka features when integrating with Redpanda ## [](#kafka-client-compatibility)Kafka client compatibility Redpanda validates the Apache Kafka Java client and a set of widely used non-Java clients, at their current versions that support Kafka 4.x, using the ducktape and chaos test suites. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations, at current client versions. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers. > 💡 **TIP** > > Always use the latest supported version of a Kafka client. The following clients have been validated with Redpanda. | Language | Client | | --- | --- | | Java | Apache Kafka Java Client | | C/C++ | librdkafka | | Go | franz-goconfluent-kafka-goSarama | | Python | kafka-pythonconfluent-kafka-python | | Rust | kafka-rust | | Node.js | KafkaJSconfluent-kafka-javascript | Clients that have not been validated by Redpanda Data, but use the Kafka protocol, remain compatible with Redpanda subject to the limitations in the next section (particularly those based on librdkafka, such as confluent-kafka-dotnet). If you find a client that does not work with Redpanda, reach out in the [Redpanda community Slack](https://redpanda.com/slack). ## [](#compatibility-exceptions)Compatibility exceptions Redpanda is compatible with the Kafka protocol, with the following exceptions: - Multiple SCRAM mechanisms simultaneously for SASL users are not supported. For example, a user cannot have both a `SCRAM-SHA-256` and a `SCRAM-SHA-512` credential. Redpanda supports only one SASL/SCRAM mechanism per user: either `SCRAM-SHA-256` or `SCRAM-SHA-512`. For details, see [Authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/). - HTTP Proxy (`pandaproxy`): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. - The `delete.retention.ms` topic configuration in Kafka is not supported for Tiered Storage topics. Cloud Topics and local storage topics support Tombstone marker deletion using `delete.retention.ms`, but in Tiered Storage topics, Tombstone markers are only removed in accordance with normal topic retention, and only if the cleanup policy is `delete` or `compact, delete`. - The Kafka request rate quota (`request_percentage`), which limits the share of broker request-handling capacity a client can consume, is not supported. Redpanda supports byte-rate (`producer_byte_rate`, `consumer_byte_rate`) and topic-mutation (`controller_mutation_rate`) quotas, which you can apply [per user, per client, or per client group](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/#client-throughput-limits). - [KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890) (Transactions Server-Side Defense): Redpanda does not implement the server-side portion of KIP-890, which addresses transaction errors specific to Kafka’s replication model. Redpanda’s implementation of transactions is not susceptible to this class of errors. When connecting to Redpanda, Kafka 4.x clients detect that Transactions V2 is unsupported and fall back to the original transaction protocol (per-transaction epoch bumping is part of V2 and does not apply). If you find an unsupported feature or incompatibility, [file an issue](https://github.com/redpanda-data/redpanda/issues/new) with the Redpanda team. --- # Page 527: Kafka Connect **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors.md --- # Kafka Connect > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Kafka Connect latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/index.adoc description: Use Kafka Connect to stream data into and out of Redpanda. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-07" --- Use Kafka Connect to integrate your Redpanda data with different data systems. As managed solutions, connectors offer a simpler way to integrate your data than manually creating a solution with the Kafka API. You can set up and manage these connectors for BYOC and Dedicated clusters in the Redpanda Cloud UI or Cloud API. > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. Each connector is either a source or a sink: - A source connector imports data from a source system into a Redpanda cluster. The source connector’s main task is to fetch data from these sources and convert them into a format suitable for Redpanda. - A sink connector exports data from a Redpanda cluster and pushes it into a target system. Sink connectors read the data from Redpanda and transform it into a format that the target system can use. These sources and sinks work together to create a data pipeline that can move and transform data from one system to another. > ⚠️ **WARNING** > > Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors. This may lead to connector and cluster failures. - [Converters and Serialization](converters-and-serialization/) Use converters to handle the serialization and deserialization of data between a Redpanda topic and an external system with Kafka Connect. - [Monitor Kafka Connect](monitor-connectors/) Use metrics to monitor the health of Kafka Connect. - [Disable Kafka Connect](disable-kc/) Learn how to disable Kafka Connect using the Cloud API. - [Single Message Transforms](transforms/) Single Message Transforms (SMTs) let you modify the data and its characteristics as it passes through a connector. - [Sizing Connectors](sizing-connectors/) How to choose number of tasks to set for a connector. - [Create an S3 Sink Connector](create-s3-sink-connector/) Use the Redpanda Cloud UI to create an AWS S3 Sink Connector. - [Create a Google BigQuery Sink Connector](create-gcp-bigquery-connector/) Use the Redpanda Cloud UI to create a Google BigQuery Sink Connector. - [Create a GCS Sink Connector](create-gcs-connector/) Use the Redpanda Cloud UI to create a GCS Sink Connector. - [Create an Iceberg Sink Connector](create-iceberg-sink-connector/) Use the Redpanda Cloud UI to create an Iceberg Sink Connector. - [Create a JDBC Sink Connector](create-jdbc-sink-connector/) Use the Redpanda Cloud UI to create a JDBC Sink Connector. - [Create a JDBC Source Connector](create-jdbc-source-connector/) Use the Redpanda Cloud UI to create a JDBC Source Connector. - [Create a MirrorMaker2 Source Connector](create-mmaker-source-connector/) Use the Redpanda Cloud UI to create a MirrorMaker2 Source Connector. - [Create a MirrorMaker2 Checkpoint Connector](create-mmaker-checkpoint-connector/) Use the Redpanda Cloud UI to create a MirrorMaker2 Checkpoint Connector. - [Create a MirrorMaker2 Heartbeat Connector](create-mmaker-heartbeat-connector/) Use the Redpanda Cloud UI to create a MirrorMaker2 Heartbeat Connector. - [Create a MongoDB Sink Connector](create-mongodb-sink-connector/) Use the Redpanda Cloud UI to create a MongoDB Sink Connector. - [Create a MongoDB Source Connector](create-mongodb-source-connector/) Use the Redpanda Cloud UI to create a MongoDB Source Connector. - [Create a MySQL (Debezium) Source Connector](create-mysql-source-connector/) Use the Redpanda Cloud UI to create a MySQL (Debezium) Source Connector. - [Create a PostgreSQL (Debezium) Source Connector](create-postgresql-connector/) Use the Redpanda Cloud UI to create a PostgreSQL (Debezium) Source Connector. - [Create a SQL Server (Debezium) Source Connector](create-sqlserver-connector/) Use the Redpanda Cloud UI to create a SQL Server (Debezium) Source Connector. - [Create a Snowflake Sink Connector](create-snowflake-connector/) Use the Redpanda Cloud UI to create a Snowflake Sink Connector. --- # Page 528: Converters and Serialization **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/converters-and-serialization.md --- # Converters and Serialization > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Converters and Serialization latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/converters-and-serialization page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/converters-and-serialization.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/converters-and-serialization.adoc description: Use converters to handle the serialization and deserialization of data between a Redpanda topic and an external system with Kafka Connect. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-09-26" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. Connectors are a translation layer working between Redpanda and the remote system. For **sink** connectors the translation happens in the following phases: 1. Converter deserializes data from Redpanda message format (for example JSON or Avro) to a universal in-memory connect data format. 2. The in-memory connect data structure is translated by the connector to the data model of the remote system. For **source** connectors it is vice versa, the phases are: 1. Connector translates the data model from remote system format to the in-memory connect data structure. 2. Converter serializes the data from a universal in-memory connect format to a Redpanda message. Each Redpanda message is a key and value record. Record key and value converters are configured separately with the `Redpanda message key format` and `Redpanda message value format` properties. Key and value converters can be different. > 📝 **NOTE** > > If an external system requires structured data (like BigQuery or a SQL database), then you must provide data with a schema. Use the Avro, Protobuf, or JSON converter with a schema. ## [](#bytearray-converter)ByteArray converter The ByteArray converter is the most primitive and high-throughput converter. Schema is ignored. This is the default converter type for managed connectors. To use the converter, select the `ByteArray` option as a key or value message format. ## [](#string-converter)String converter The String converter is a high-throughput converter. Schema is ignored. All data is converted to a string. To use the converter, select the `String` option as a key or value message format. ## [](#json-converter)JSON converter The JSON converter supports a JSON schema embedded in the message, where each message contains a schema. It results in a bigger message size. The connector needs a message schema to check message format. To use the converter, select the `JSON` option as a key or value message format. Example JSON message with embedded schema: ```json { "schema": { "type": "struct", "fields": [ { "type": "int64", "optional": false, "field": "person_id" }, { "type": "string", "optional": false, "field": "name" } ] }, "payload": { "person_id": 1, "name": "Redpanda" } } ``` If you consume JSON data with no message schema, the schema check for the connector must be disabled with the `Message key JSON contains schema` or `Message value JSON contains schema` option. ## [](#avro-converter)Avro converter The Avro converter requires a schema in Schema Registry. Avro supports primitive types and complex types, like records, enums, arrays, maps, and unions. To specify a timestamp in an Avro schema for use with Kafka Connect, use: ```json { "name": "time1", "type": [ "null", { "type": "long", "connect.version": 1, "connect.name": "org.apache.kafka.connect.data.Timestamp", "logicalType": "timestamp-millis" } ], "default": null } ``` See also: - [Redpanda Schema Registry](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/) - [Avro specification](https://avro.apache.org/docs/1.11.1/specification) ## [](#cloudevents-converter)CloudEvents converter The CloudEvents converter is specific to Debezium PostgreSQL and MySQL source connectors. See also: [CloudEvents Converter documentation](https://debezium.io/documentation/reference/2.2/integrations/cloudevents.html) ## [](#protobuf-converter)Protobuf converter ![Beta](https://img.shields.io/badge/Beta-red.svg) The Protobuf converter requires a schema in Schema Registry. The converter only supports sink connectors. Source connectors are not supported. To use the converter, select the `Protobuf` option as a key or value message format. See also: [Redpanda Schema Registry](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/) ## [](#set-property-keys)Set property keys Kafka Connect connectors use a set of `=` to set up properties. For example if you want to set the property `topic.creation.enable` to `true`, use `topic.creation.enable=true` in the property settings page. --- # Page 529: Create a Google BigQuery Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-gcp-bigquery-connector.md --- # Create a Google BigQuery Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a Google BigQuery Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-gcp-bigquery-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-gcp-bigquery-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-gcp-bigquery-connector.adoc description: Use the Redpanda Cloud UI to create a Google BigQuery Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. The Google BigQuery Sink connector enables you to stream any structured data from Redpanda to BigQuery for advanced analytics. ## [](#prerequisites)Prerequisites Before you can create a Google BigQuery Sink connector in the Redpanda Cloud, you must: 1. Create a [Google Cloud](https://cloud.google.com/) account. 2. In the **Google home** page: 1. [Select an existing project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#get_an_existing_project) or [create a new one](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project). 2. [Create a new dataset](https://cloud.google.com/bigquery/docs/datasets) for the project. 3. (_Optional if your data has a schema_) After creating the dataset, [create a new table](https://cloud.google.com/bigquery/docs/tables) to hold the data you intend to stream from Redpanda Cloud topics. Specify a structure for the table using schema values that align with your Redpanda topic data. > 📝 **NOTE** > > This step is mandatory only if the data in Redpanda does not have a schema. If the data in Redpanda includes a schema, then the connector automatically creates the tables in BigQuery. 3. Create a [custom role](https://cloud.google.com/iam/docs/creating-custom-roles). The role must have the following permissions: bigquery.datasets.get bigquery.tables.create bigquery.tables.get bigquery.tables.getData bigquery.tables.list bigquery.tables.update bigquery.tables.updateData 4. Create a [service account](https://cloud.google.com/iam/docs/service-accounts-create). 5. [Add the custom role to your service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access). 6. [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete), and then download it. ## [](#limitations)Limitations The Google BigQuery Sink connector doesn’t support schemas with recursion. ## [](#create-a-google-bigquery-sink-connector)Create a Google BigQuery Sink connector To create the Google BigQuery Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to Google BigQuery**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | A comma-separated list of the cluster topics you want to replicate to Google BigQuery. | | Topics regex | topics.regex | A Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | Credentials JSON | keyfile | A JSON key with BigQuery service account credentials. | | Project | project | The BigQuery project to which topic data will be written. | | Default dataset | defaultDataset | The default Google BigQuery dataset to be used. | | Kafka message value format | value.converter | The format of the value in the Redpanda topic. The default is JSON. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-google-bigquery-sink-connector-configuration)Advanced Google BigQuery Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require any additional property settings (for example, automatically create BigQuery tables or map topics to tables), then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Auto create tables | autoCreateTables | Automatically create BigQuery tables if they don’t already exist. If the table does not exist, then it is created based on the record schema. | | Topic to table map | topic2TableMap | Map of topics to tables. Format: comma-separated tuples, for example topic1:table1,topic2:table2. | | Allow new BigQuery fields | allowNewBigQueryFields | If true, new fields can be added to BigQuery tables during subsequent schema updates. | | Allow BigQuery required field relaxation | allowBigQueryRequiredFieldRelaxation | If true, fields in the BigQuery schema can be changed from REQUIRED to NULLABLE. | | Upsert enabled | upsertEnabled | Enables upsert functionality on the connector. | | Delete enabled | deleteEnabled | Enable delete functionality on the connector. | | Kafka key field name | kafkaKeyFieldName | The name of the BigQuery table field for the Kafka key. Must be set when upsert or delete is enabled. | | Time partitioning type | timePartitioningType | The time partitioning type to use when creating tables. | | BigQuery retry attempts | bigQueryRetry | The number of retry attempts made for each BigQuery request that fails with a backend or quota exceeded error. | | BigQuery retry attempts interval | bigQueryRetryWait | The minimum amount of time, in milliseconds, to wait between BigQuery backend or quota exceeded error retry attempts. | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` (`org.apache.kafka.connect.json.JsonConverter`) when your messages are JSON-encoded. Select `Message JSON contains schema`, with the `schema` and `payload` fields. If your messages do not contain schema, manually create tables in BigQuery. - `AVRO` (`io.confluent.connect.avro.AvroConverter`) when your messages contain AVRO-encoded messages, with schema stored in the Schema Registry. ## [](#topic-name-to-table-name-mapping)Topic name to table name mapping By default, the table name is the name of the topic. Use the `Topic to table map` (`topic2TableMap`) configuration property to remap topic names. For example, `topic1:table1,topic2:table2`. ## [](#test-the-connection)Test the connection After the connector is created, go to your BigQuery worksheets and query your table: ```sql SELECT * FROM `project.dataset.table` ``` It may take a couple of minutes for the records to be visible in BigQuery. ## [](#troubleshoot)Troubleshoot Google credentials are checked for validity during connector creation, upon clicking **Finish**. In cases where there are invalid credentials, the connector is not created. Other issues are reported using a failed task error message. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | Not found: Project invalid-project-name | Check to make sure Project contains a valid BigQuery project. | | Not found: Dataset project:invalid-dataset | Check to make sure Default dataset contains a valid BigQuery dataset. | | An unexpected error occurred while validating credentials for BigQuery: Failed to create credentials from input stream | The credentials given as a JSON file in the Credentials JSON property are incorrect. Copy a valid key from the Google Cloud service account. | | JsonConverter with schemas.enable requires "schema" and "payload" fields | The connector encountered an incorrect message format when reading from a topic. | | JsonParseException: Unrecognized token 'test': was expecting JSON | During reading from a topic the connector encountered a message that is invalid JSON. | | Streaming to metadata partition of column-based partitioning table {table_name} is disallowed. | Check to confirm that the bigQueryPartitionDecorator property is set to false. You can check the property in the connector configuration JSON view. | | Caused by: table: GenericData{classInfo=…​ insertion failed for the following rows:…​ no such field: | The Redpanda message contains a property that does not exist in a BigQuery table schema. | | BigQueryConnectException …​ insertion failed for the following rows: …​ [row index 0] (location fieldname[0], reason: invalid): This field: fieldname is not a record. | The Redpanda message contains an array of records, but the BigQuery table expects an array of strings. | | BigQueryConnectException: Failed to unionize schemas of records for the table…​ Could not convert to BigQuery schema with a batch of tombstone records. | The Redpanda message does not contain a schema, so the connector cannot create a BigQuery table. Create the BigQuery table manually. | --- # Page 530: Create a GCS Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-gcs-connector.md --- # Create a GCS Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a GCS Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-gcs-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-gcs-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-gcs-connector.adoc description: Use the Redpanda Cloud UI to create a GCS Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. The Google Cloud Storage (GCS) Sink connector stores Redpanda messages in a Google Cloud Storage bucket. ## [](#prerequisites)Prerequisites Before you can create a GCS Sink connector in the Redpanda Cloud, you must: 1. Create a [Google Cloud](https://cloud.google.com/) account. 2. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create) that will be used to connect to the GCS service. 3. [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete) and download it. 4. Create a [custom role](https://cloud.google.com/iam/docs/creating-custom-roles), which must have the following permissions: - `storage.objects.create` to create items in the GCS bucket - `storage.objects.delete` to overwrite items in the GCS bucket 5. [Create a GCS bucket](https://cloud.google.com/storage/docs/creating-buckets) to which to send data. 6. [Grant permissions](https://cloud.google.com/storage/docs/access-control/using-iam-permissions) to the bucket your created for your service account. Use the role created in step 4. ## [](#limitations)Limitations The GCS Sink connector has the following limitations: - You can use only the `STRING` and `BYTES` input formats for `CSV` output format. - You can use only the `PARQUET` format when your messages contain schema. ## [](#create-a-gcs-sink-connector)Create a GCS Sink connector To create the GCS Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to Google Cloud Storage**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | Comma-separated list of the cluster topics you want to replicate to GCS. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | GCS Credentials JSON | gcs.credentials.json | JSON object with GCS credentials. | | GCS bucket name | gcs.bucket.name | Name of an existing GCS bucket to store output files in. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. Use BYTES for no conversion. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. Use BYTES for no conversion. | | GCS file format | format.output.type | Format of the files created in GCS: CSV (the default), JSON, JSONL AVRO, or PARQUET. You can use the CSV format output only with BYTES and STRING. | | Avro codec | avro.codec | The Avro compression codec to be used for Avro output files. Available values: null (the default), deflate, snappy, and bzip2. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-gcs-sink-connector-configuration)Advanced GCS Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require any additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | File name template | file.name.template | The template for file names on GCS. Supports {{ variable }} placeholders for substituting variables. Supported placeholders are:topicpartitionstart_offset (the offset of the first record in the file)timestamp:unit=yyyy|MM|dd|HH (the timestamp of the record)key (when used, other placeholders are not substituted) | | File name prefix | file.name.prefix | The prefix to be added to the name of each file put in GCS. | | Output fields | format.output.fields | Fields to place into output files. Supported values are: 'key', 'value', 'offset', 'timestamp', and 'headers'. | | Value field encoding | format.output.fields.value.encoding | The type of encoding to be used for the value field. Supported values are: 'none' and 'base64'. | | Envelope for primitives | format.output.envelope | Specifies whether or not to enable additional JSON object wrapping of the actual value. | | Output file compression | file.compression.type | The compression type to be used for files put into GCS. Supported values are: 'none', 'gzip', 'snappy', and 'zstd'. | | Max records per file | file.max.records | The maximum number of records to put in a single file. Must be a non-negative number. 0 is interpreted as "unlimited", which is the default. In this case files are only flushed after file.flush.interval.ms. | | File flush interval milliseconds | file.flush.interval.ms | The time interval to periodically flush files and commit offsets. Value specified must be a non-negative number. Default is 60 seconds. 0 indicates that it is disabled. In this case, files are only flushed after reaching file.max.records record size. | | GCS bucket check | gcs.bucket.check | If set to true, the connector will attempt to put a test file to the GCS bucket to validate access. Default is true. | | GCS retry backoff initial delay milliseconds | gcs.retry.backoff.initial.delay.ms | Initial retry delay in milliseconds. The default value is 1000. | | GCS retry backoff max delay milliseconds | gcs.retry.backoff.max.delay.ms | Maximum retry delay in milliseconds. The default value is 32000. | | GCS retry backoff delay multiplier | gcs.retry.backoff.delay.multiplier | Retry delay multiplier. The default value is 2.0. | | GCS retry backoff max attempts | gcs.retry.backoff.max.attempts | Retry max attempts. The default value is 6. | | GCS retry backoff total timeout milliseconds | gcs.retry.backoff.total.timeout.ms | Retry total timeout in milliseconds. The default value is 50000. | | Retry back-off | kafka.retry.backoff.ms | Retry backoff in milliseconds. In case of transient exceptions, useful for performing recovery. Maximum value is 86400000 (24 hours). | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` (`org.apache.kafka.connect.json.JsonConverter`) when your messages are JSON-encoded. Select `Message JSON contains schema`, with the `schema` and `payload` fields. - `AVRO` (`io.confluent.connect.avro.AvroConverter`) when your messages contain AVRO-encoded messages, with schema stored in the Schema Registry. - `STRING` (`org.apache.kafka.connect.storage.StringConverter`) when your messages contain textual data. - `BYTES` (`org.apache.kafka.connect.converters.ByteArrayConverter`) when your messages contain arbitrary data. You can also select the output data format for your GCS files as follows: - `CSV` to produce data in the `CSV` format. For `CSV` only, you can set `STRING` and `BYTES` input formats. - `JSON` to produce data in the `JSON` format as an array of record objects. - `JSONL` to produce data in the `JSON` format, each message as a separate JSON, one per line. - `PARQUET` to produce data in the `PARQUET` format when your messages contain schema. - `AVRO` to produce data in the `AVRO` format when your messages contain schema. ## [](#test-the-connection)Test the connection After the connector is created, check the GCS bucket for a new file. Files should appear after the file flush interval (default is 60 seconds). ## [](#troubleshoot)Troubleshoot If there are any connection issues, an error message is returned. Depending on the `GCS bucket check` property value, the error results in a failed connector (`GCS bucket check = true`) or a failed task (`GCS bucket check = false`). Select **Show Logs** to view error details. Additional errors and corrective actions follow. | Message | Action | | --- | --- | | Failed to read credentials from JSON string | The credentials given as JSON file in the GCS credentials JSON property are incorrect. Copy a valid key from the Google Cloud service account. | | The specified bucket does not exist | Create the bucket if the bucket does not exist, or correct the bucket name if the bucket exists, but the specified GCS bucket name value is incorrect. | | No files in the GCS bucket | Be sure to wait until the connector performs the first file flush (default is 60 seconds). | --- # Page 531: Create an Iceberg Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-iceberg-sink-connector.md --- # Create an Iceberg Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create an Iceberg Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-iceberg-sink-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-iceberg-sink-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-iceberg-sink-connector.adoc description: Use the Redpanda Cloud UI to create an Iceberg Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2026-03-31" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use the Iceberg Sink connector to accomplish the following: - Write data into Iceberg tables - Commit coordination for centralized Iceberg commits - Exactly-once delivery semantics - Multi-table fan-out - Row mutations (update/delete rows), upsert mode - Automatic table creation and schema evolution - Field name mapping via Iceberg’s column mapping functionality ## [](#prerequisites)Prerequisites Before you can create an Iceberg Sink connector in Redpanda Cloud, you must: 1. [Set up an Iceberg catalog](https://iceberg.apache.org/concepts/catalog/). 2. Create the Iceberg connector control topic, which cannot be used by other connectors. For details, see [Create a Topic](https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic/). ## [](#limitations)Limitations - Each Iceberg sink connector must have its own control topic, which you should create before creating the connector. ## [](#create-an-iceberg-sink-connector)Create an Iceberg Sink connector To create the Iceberg Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu and then click **Create Connector**. 2. Select **Export to Iceberg**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | Comma-separated list of the cluster topics you want to replicate. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | Iceberg control topic | iceberg.control.topic | The name of the control topic. You must create this topic before creating the Iceberg connector. It cannot be used by other Iceberg connectors. | | Iceberg catalog type | iceberg.catalog.type | The type of Iceberg catalog. Allowed options are: REST, HIVE, HADOOP. | | Iceberg tables | iceberg.tables | Comma-separated list of Iceberg table names, which are specified using the format {namespace}.{table}. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-iceberg-sink-connector-configuration)Advanced Iceberg Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Iceberg commit timeout | iceberg.control.commit.timeout-ms | Commit timeout interval in ms. The default is 30000 (30 sec). | | Iceberg tables route field | iceberg.tables.route-field | For multi-table fan-out, the name of the field used to route records to tables. | | Iceberg tables CDC field | iceberg.tables.cdc-field | Name of the field containing the CDC operation, I, U, or D. Default is none. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` when your messages are JSON-encoded. Select `Message JSON contains schema` with the `schema` and `payload` fields. If your messages do not contain schema, create Iceberg tables manually. - `AVRO` when your messages contain AVRO-encoded messages, with schema stored in the Schema Registry. An Iceberg table’s schema is a list of named columns. All data types are either primitives or nested types, which are maps, lists, or structs. A table schema is also a struct type. See also: [Schemas and Data Types](https://iceberg.apache.org/spec/#schemas-and-data-types) ## [](#sinking-data-produced-by-debezium-source-connector)Sinking data produced by Debezium source connector Debezium connectors produce data in CDC format. The message structure can be flattened by using Debezium built-in New Record State Extraction Single Message Transformation (SMT). Add the following properties to the Debezium connector configuration to make it produce flat messages: ```json { ... "transforms", "unwrap", "transforms.unwrap.type", "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones", "false", ... } ``` Depending on your particular use case, you can apply the SMT to a Debezium connector, or to a sink connector that consumes messages that the Debezium connector produces. To enable Apache Kafka to retain the Debezium change event messages in their original format, configure the SMT for a sink connector. See also: [Debezium New Record State Extraction SMT](https://debezium.io/documentation/reference/stable/transformations/event-flattening.html) ## [](#use-analytical-tools-with-iceberg)Use analytical tools with Iceberg Iceberg serves as a single storage solution for analytical data. It is inexpensive to read from various tools such as AWS Athena, Snowflake, or Apache Spark. Traditionally, data import involved pushing data to every tool, incurring high costs for data transfer and storage. Alternatively, you could use plain S3 buckets with Avro or CSV files, but this struggles with schema evolution. [Apache Iceberg](https://iceberg.apache.org) addresses all of these challenges: cost of data transfer, multiple data copies in storage, and support for schema evolution. ![Iceberg sink connector diagram](https://docs.redpanda.com/cloud-data-platform/shared/_images/iceberg_sink_connector_diagram.png) The following example uses: - Iceberg REST catalog - AWS S3 bucket as the storage for Iceberg files - Apache Spark, which reads the Iceberg data from an S3 bucket ```yaml version: '3' services: redpanda: image: docker.redpanda.com/redpandadata/redpanda:latest command: - redpanda start - --smp 1 - --overprovisioned - --node-id 0 - --reserve-memory 0M - --check=false - --set redpanda.auto_create_topics_enabled=false - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092 - --pandaproxy-addr 0.0.0.0:8082 - --advertise-pandaproxy-addr localhost:8082 ports: - 8081:8081 - 8082:8082 - 9092:9092 - 9644:9644 - 29092:29092 console: image: docker.redpanda.com/redpandadata/console:latest restart: on-failure entrypoint: /bin/sh command: -c "echo \"$$CONSOLE_CONFIG_FILE\" > /tmp/config.yml; /app/console" environment: CONFIG_FILEPATH: /tmp/config.yml CONSOLE_CONFIG_FILE: | kafka: brokers: ["redpanda:29092"] schemaRegistry: enabled: true urls: ["http://redpanda:8081"] connect: enabled: true clusters: - name: connectors url: http://connect:8083 ports: - "8090:8080" depends_on: - redpanda connect: image: docker.redpanda.com/redpandadata/connectors:latest hostname: connect depends_on: - redpanda - spark-iceberg ports: - "8083:8083" - "9404:9404" environment: CONNECT_CONFIGURATION: | key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter group.id=connectors-cluster offset.storage.topic=_internal_connectors_offsets config.storage.topic=_internal_connectors_configs status.storage.topic=_internal_connectors_status config.storage.replication.factor=-1 offset.storage.replication.factor=-1 status.storage.replication.factor=-1 producer.linger.ms=1 producer.batch.size=131072 config.providers=file config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider CONNECT_BOOTSTRAP_SERVERS: redpanda:29092 SCHEMA_REGISTRY_URL: http://redpanda:8081 CONNECT_GC_LOG_ENABLED: "false" CONNECT_HEAP_OPTS: -Xms512M -Xmx512M CONNECT_LOG_LEVEL: info CONNECT_TOPIC_LOG_ENABLED: "true" CONNECT_PLUGIN_PATH: "/opt/kafka/connect-plugins" spark-iceberg: image: tabulario/spark-iceberg:3.4.1_1.3.1 build: spark/ depends_on: - rest volumes: - ./warehouse:/home/iceberg/warehouse environment: - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} - AWS_REGION=${AWS_REGION} ports: - 8888:8888 - 8080:8080 - 10000:10000 - 10001:10001 rest: image: tabulario/iceberg-rest:0.6.0 ports: - 8181:8181 environment: - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} - AWS_REGION=${AWS_REGION} - CATALOG_WAREHOUSE=s3://bucket-name/ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO ``` Use Spark-SQL to: - List databases: ```none spark-sql ()> show databases; testdb ``` - Show tables in database: ```none spark-sql ()> show tables in testdb; testtable ``` - Select data from table: ```none spark-sql ()> select * from testdb.testtable; ``` ## [](#use-with-aws-glue-data-catalog-and-aws-lake-formation)Use with AWS Glue Data Catalog and AWS Lake Formation The connector can be used with the AWS Glue Data Catalog and the AWS Lake Formation service. AWS Lake Formation only lets you use the role form of authentication. The connectors UI does not support Lake Formation-specific properties. Use the JSON editor instead. Sample configuration: ```json { ... "iceberg.catalog.client.assume-role.region": "the-region", "iceberg.catalog.client.assume-role.arn": "arn:aws:iam::account-number:role/role-name", "iceberg.catalog.glue.account-id": "NNN", "iceberg.catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", "iceberg.catalog.client.assume-role.tags.LakeFormationAuthorizedCaller": "iceberg-connect", "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "iceberg.catalog": "catalog_name", "iceberg.catalog.warehouse": "s3://bucket-name/my/data", "iceberg.catalog.s3.path-style-access": "true" } ``` ## [](#test-the-connection)Test the connection After the connector is created, execute SELECT query on the Iceberg table to verify data. It may take a couple of minutes for the records to be visible in Iceberg. Check connector state and logs for errors. ## [](#troubleshoot)Troubleshoot Iceberg connection settings are checked for validity during first data processing. The connector can be successfully created with incorrect configuration and fail only when there are messages in source topic to process. | Message | Action | | --- | --- | | NoSuchTableException: Table does not exist | Make sure Iceberg table exists and the connector iceberg.tables configuration contains correct table name in {namespace}.{table} format. | | UnknownHostException: incorrectcatalog: Name or service not known | Cannot connect to Iceberg catalog. Check if Iceberg catalog URI is correct and accessible. | | DataException: An error occurred converting record, topic: topicName, partition, 0, offset: 0 | The connector cannot read the message format. Ensure the connector mapping configuration and data format are correct. | | NullPointerException: Cannot invoke "java.lang.Long.longValue()" because "value" is null | The connector cannot read the message format. Ensure the connector mapping configuration and data format are correct. | ## [](#suggested-reading)Suggested reading - For details about the Iceberg Sink connector configuration properties, see [Iceberg-Kafka-Connect](https://github.com/tabular-io/iceberg-kafka-connect) - For details about the Iceberg Sink connector internals, see [Iceberg-Kafka-Connect documentation](https://github.com/tabular-io/iceberg-kafka-connect/tree/main/docs) --- # Page 532: Create a JDBC Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-jdbc-sink-connector.md --- # Create a JDBC Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a JDBC Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-jdbc-sink-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-jdbc-sink-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-jdbc-sink-connector.adoc description: Use the Redpanda Cloud UI to create a JDBC Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a JDBC Sink connector to export structured data from Redpanda to a relational database. ## [](#prerequisites)Prerequisites Before you can create a JDBC Sink connector in the Redpanda Cloud, you must have a: - Relational database instance that is accessible from the JDBC Sink connector instance - Database user ## [](#limitations)Limitations The JDBC Sink connector has the following limitations: - Only `JSON` or `AVRO` formats can be used as a value converter. - Only the following databases are supported: - MySQL 5.7 and 8.0 - PostgreSQL 8.2 and higher using the version 3.0 of the PostgreSQL® protocol - SQLite - SQL Server - Microsoft SQL versions: Azure SQL Database, Azure Synapse Analytics, Azure SQL Managed Instance, SQL Server 2014, SQL Server 2016, SQL Server 2017, SQL Server 2019 ## [](#create-a-jdbc-sink-connector)Create a JDBC Sink connector To create the JDBC Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to JDBC**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | Comma-separated list of the cluster topics you want to replicate. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | JDBC URL | connection.url | The database connection JDBC URL. | | User | connection.user | Name of the database user to be used when connecting to the database. | | Password | connection.password | Password of the database user to be used when connecting to the database. | | Redpanda message key format | key.converter | Format of the key in the Redpanda topic. BYTES is the default. | | Redpanda message value format | value.converter | Format of the value in the Redpanda topic. JSON is the default. | | Auto-create | auto.create | When enabled, automatically creates the destination table (if it is missing) based on the record schema (issues a CREATE). The default is disabled. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-jdbc-sink-connector-configuration)Advanced JDBC Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Include fields | fields.whitelist | List of comma-separated record value field names. If the value of this property is empty, the connector uses all fields from the record to migrate to a database. Otherwise, the connector uses only the record fields that are specified (in a comma-separated format). Note that Primary Key Fields is applied independently in the context of which fields form the primary key columns in the destination database, while this configuration is applicable for the other columns. | | Topics to tables mapping | topics.to.tables.mapping | Kafka topics to database tables mapping. Comma-separated list of topic to table mapping in the format: topic_name:table_name. If the destination table is found in the mapping, then it overrides the generated one defined in table.name.format. | | Table name format | table.name.format | A format string for the destination table name, which may contain ${topic} as a placeholder for the original topic name. For example, kafka_${topic} for the topic orders maps to the table name kafka_orders. The default is ${topic}. | | Table name normalize | table.name.normalize | Specifies whether or not to normalize destination table names for topics. When enabled, the alphanumeric characters (a-z, A-Z, 0-9) and remain as is, others (such as .) are replaced with . By default, is disabled. | | Quote SQL identifiers | sql.quote.identifiers | Specifies whether or not to delimit (in most databases, a quote with double quotation marks) identifiers (for example, table names and column names) in SQL statements. By default, enabled. | | Auto-evolve | auto.evolve | Whether to automatically add columns in the table schema when found to be missing relative to the record schema by issuing ALTER. | | Batch size | batch.size | Specifies how many records to attempt to batch together for insertion into the destination table, when possible. The default is 3000. | | DB time zone | db.timezone | Name of the JDBC timezone that should be used in the connector when querying with time-based criteria. Default is UTC. | | Insert mode | insert.mode | The insertion mode to use. The supported modes are:INSERT: standard SQL INSERT statementsMULTI: multi-row INSERT statementsUPSERT: use the appropriate upsert semantics for the target database if it is supported by the connector; for example, INSERT .. ON CONFLICT .. DO UPDATE SET ..UPDATE: use the appropriate update semantics for the target database if it is supported by the connector; for example, UPDATE. | | Primary key mode | pk.mode | The primary key mode to use. Supported modes are:NONE: no keys utilizedkafka: Kafka coordinates (the topic, partition, and offset) are used as the primary keyRECORD_KEY: fields from the record key are used, which may be a primitive or a structRECORD_VALUE: fields from the record value are used, which must be a struct. | | Primary key fields | pk.fields | Comma-separated list of primary key field names. The runtime interpretation of this configuration depends on the pk.mode. Supported modes are:none: ignored because no fields are used as primary key in this mode.kafka: must be a trio representing the Kafka coordinates (the topic, partition, and offset). Defaults to connect_topic,connect_partition,__connect_offset if empty.record_key: if empty, all fields from the key struct will be used, otherwise used to extract the desired fields. For primitive key, only a single field name must be configured.record_value: if empty, all fields from the value struct will be used, otherwise used to extract the desired fields. | | Maximum retries | max.retries | The maximum number of times to retry on errors before failing the task. The default is 10. | | Retry backoff (ms) | retry.backoff.ms | The time in milliseconds to wait before a retry attempt is made following an error. The default is 3000. | | Database dialect | dialect.name | The name of the database dialect that should be used for this connector. By default. the connector automatically determines the dialect based upon the JDBC connection URL. Use if you want to override that behavior and specify a specific dialect. | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - Use the default `Redpanda message value format` = `JSON` (`org.apache.kafka.connect.json.JsonConverter`) property in your configuration. - Topics should contain data in JSON format with a defined JSON schema. For example: ```json { "schema": { "type": "struct", "fields": [ ] }, "payload": { } } ``` ## [](#test-the-connection)Test the connection After the connector is created, ensure that: - There are no errors in logs and in Redpanda Console. - Database tables contain data from Redpanda topics. ## [](#troubleshoot)Troubleshoot JDBC Sink connector issues are reported as failed tasks. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | PSQLException: FATAL: database "invalid-database" does not exist | Make sure the JDBC URL specifies an existing database name. | | UnknownHostException: invalid-host | Make sure the JDBC URL specifies a valid database host name. | | PSQLException: Connection to postgres:1234 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections | Make sure the JDBC URL specifies a valid database host name and port, and that the port is accessible. | | PSQLException: FATAL: password authentication failed for user "postgres" | Verify that the User and Password are correct. | | ConnectException: topic_name.Value (STRUCT) type doesn’t have a mapping to the SQL database column type | The JDBC Sink connector is not compatible with the Debezium PostgreSQL Source connector. Kafka Connect JSON produced by the Debezium Connector is not compatible with what the JDBC Sink Connector is expecting. Try changing a topic name. The JDBC Source connector is compatible with the JDBC Sink connector, and can be used as an alternative for a Debezium PostgreSQL source connector. | --- # Page 533: Create a JDBC Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-jdbc-source-connector.md --- # Create a JDBC Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a JDBC Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-jdbc-source-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-jdbc-source-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-jdbc-source-connector.adoc description: Use the Redpanda Cloud UI to create a JDBC Source Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a JDBC Source connector to import batches of rows from MySQL, PostgreSQL, SQLite, and SQL Server relational databases into Redpanda topics. ## [](#prerequisites)Prerequisites - Relational database instance that is accessible from the JDBC Source connector instance. - Database user has been created. ## [](#limitations)Limitations The JDBC Source connector has the following limitations: - Only `JSON` or `AVRO` formats can be used as a value converter. - Only the following databases are supported: - MySQL 5.7 and 8.0 - PostgreSQL 8.2 and higher using the version 3.0 of the PostgreSQL® protocol - SQLite - SQL Server - Microsoft SQL versions: Azure SQL Database, Azure Synapse Analytics, Azure SQL Managed Instance, SQL Server 2014, SQL Server 2016, SQL Server 2017, SQL Server 2019 ## [](#create-a-jdbc-source-connector)Create a JDBC Source connector To create the JDBC Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from JDBC**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topic prefix | topic.prefix | Prefix to prepend to table names to generate the name of the Kafka topic to which to publish data, or in the case of a custom query, the full name of the topic to publish to. | | JDBC URL | connection.url | The database connection JDBC URL. | | User | connection.user | Name of the database user to be used when connecting to the database. | | Password | connection.password | Password of the database user to be used when connecting to the database. | | Redpanda message value format | value.converter | Format of the value in the Redpanda topic. JSON is the default. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates an exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-jdbc-source-connector-configuration)Advanced JDBC Source connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | JDBC connection attempts | connection.attempts | Maximum number of attempts to retrieve a valid JDBC connection. The default is 3. | | JDBC connection backoff (ms)) | connection.backoff.ms | Backoff time between connection attempts. The default is 10000. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. BYTES is the default. | | Kafka message headers format | header.converter | Format of the headers in the Kafka topic. The default is SIMPLE. | | Include tables | table.whitelist | List of tables to include when copying. If specified, you cannot specify the Exclude Tables property. | | Exclude tables | table.blacklist | List of tables to exclude when copying. If specified, you cannot specify the Include Tables property. | | Qualify table names | table.names.qualify | Specifies whether or not to use fully-qualified table names when querying the database. If disabled, queries are performed with unqualified table names. This property may be useful if the database has been configured with a search path that automatically directs unqualified queries to the correct table when there are multiple tables available with the same unqualified name. | | Catalog pattern | catalog.pattern | Catalog pattern used to fetch table metadata from the database. null (default) means that the catalog name is not to be used to narrow the search to fetch all table metadata, regardless of the catalog. `""`retrieves those without a catalog. | | Schema pattern | schema.pattern | Schema pattern used to fetch table metadata from the database: * "" retrieves those without a schema. * null (default) specifies that the schema name is not to be used to narrow the search, so that all table metadata is fetched, regardless of the schema. | | DB time zone | db.timezone | Name of the JDBC timezone that should be used in the connector when querying with time-based criteria. Default is UTC. | | Max rows per batch | batch.max.rows | Maximum number of rows to include in a single batch when polling for new data. You can use this property to limit the amount of data buffered internally in the connector. The default is 100. | | Incrementing column name | incrementing.column.name | The name of the strictly incrementing column to use to detect new rows. An empty value indicates the column should be autodetected by looking for an auto-incrementing column. This column cannot not be nullable. | | Incrementing column initial value | incrementing.initial | For the incrementing column, consider only the rows that have a value greater than this. Specify if you need to pick up rows with negative or zero value, or if you want to skip rows. The default is -1. To avoid excessive memory usage leading to a large data set, carefully select the initial value. | | Table loading mode | mode | The mode for updating a table each time it is polled. Options include:bulk: perform a bulk load of the entire table each time it is polled.incrementing: use a strictly incrementing column on each table to detect only new rows. Note that this does not detect modifications or deletions of existing rows.timestamp: use a timestamp (or timestamp-like) column to detect new and modified rows. Based on the assumption that the column is updated with each write, and that values are monotonically incrementing, but not necessarily unique.timestamp+incrementing: use two columns, a timestamp column that detects new and modified rows, and a strictly incrementing column, which provides a globally unique ID for updates so that each row can be assigned a unique stream offset. | | Map Numeric Values, Integral or Decimal, By Precision and Scale | numeric.mapping | Map NUMERIC values by precision and optionally scale to integral or decimal types:none (default): use if all NUMERIC columns are to be represented by Connect’s DECIMAL logical type. This may lead to serialization issues with Avro because Connect’s DECIMAL type is mapped to its binary representationbest_fit: use if NUMERIC columns should be cast to Connect’s INT8, INT16, INT32, INT64, or FLOAT64 based upon the column’s precision and scale. Is often preferred because it maps to the most appropriate primitive type.precision_only: use to map NUMERIC columns based only on the column’s precision (assuming that column’s scale is 0). | | Poll interval (ms) | poll.interval.ms | Frequency used to poll for new data in each table. The default is 5000. | | Query | query | Specifies the query to use to select new or updated rows. Use to join tables, select subsets of columns in a table, or to filter data. When specified, this connector will only copy data using this query, and whole-table copying will be disabled. Different query modes may still be used for incremental updates, but to properly construct the incremental query, it must be possible to append a WHERE clause to this query (that is, no WHERE clauses can be used). If you use a WHERE clause, it must handle incremental queries itself. | | Quote SQL identifiers | sql.quote.identifiers | Specifies whether or not to delimit (in most databases, a quote with double quotation marks) identifiers (for example, table names and column names) in SQL statements. | | Metadata change monitoring interval (ms) | table.poll.interval.ms | Frequency to poll for new or removed tables, which may result in updated task configurations to start polling for data in added tables, or stop polling for data in removed tables. The default is 60000. | | Table types | table.types | By default, the JDBC connector only detects tables with type TABLE from the source Database. This property allows a command separated list of table types to extract. Options include: TABLE (default) VIEW SYSTEM TABLE GLOBAL TEMPORARY LOCAL TEMPORARY ALIAS SYNONYM. In most cases, it is best to specify TABLE or VIEW. | | Timestamp column name | timestamp.column.name | Comma separated list of one or more timestamp columns to detect new or modified rows using the COALESCE SQL function. Rows whose first non-null timestamp value is greater than the largest previous timestamp value seen aare discovered with each poll. At least one column should not be nullable. | | Delay interval (ms) | timestamp.delay.interval.ms | The amount of time to wait after a row with a certain timestamp appears before including it in the result. You can add a delay to allow transactions with earlier timestamp to complete. The first execution fetches all available records (that is, starting at a timestamp greater than 0) until current time minus the delay. Every following execution will get data from the last time fetched until the current time, minus the delay. | | Initial timestamp (ms) since epoch | timestamp.initial.ms | The initial value of the timestamp when selecting records. Value can be negative. The records having a timestamp greater than the value are included in the result. To avoid excessive memory usage leading to a large data set, carefully select the initial timestamp. | | Validate non null | validate.non.null | By default, the JDBC connector validates that all incrementing and timestamp tables have NOT NULL set for the columns being used as their ID/timestamp. If the tables don’t, then the JDBC connector will fail to start. Setting to false disables these checks. | | Database dialect | dialect.name | The name of the database dialect that should be used for this connector. By default. the connector automatically determines the dialect based upon the JDBC connection URL. Use if you want to override that behavior and specify a specific dialect. | | Topic creation enabled | topic.creation.enable | Specifies whether or not to allow automatic creation of topics. Default is enabled. | | Topic creation partitions | topic.creation.default. partitions | Specifies the number of partitions for the created topics. The default is 1. | | Topic creation replication factor | topic.creation.default. replication.factor | Specifies the replication factor for the created topics. The default is -1. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - You can use Schema Registry as an alternative to the JSON schema. - Use `Kafka message value format` = `AVRO` (`io.confluent.connect.avro.AvroConverter`) to use Schema Registry with `AvroConverter`. Use the following properties to select the database data set to read from: - `Include tables` - `Exclude tables` - `Catalog pattern` - `Schema pattern` ## [](#test-the-connection)Test the connection After the connector is created, check to ensure that: - There are no errors in logs and in Redpanda Console. - Redpanda topics contain data from relational database tables. ## [](#troubleshoot)Troubleshoot Most JDBC Source connector issues are identified in the connector creation phase. Invalid `Include tables` are reported in logs. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | PSQLException: FATAL: database "invalid-database" does not exist | Make sure the JDBC URL specifies an existing database name. | | PSQLException: The connection attempt failed. for configuration Couldn’t open connection / PSQLException: Connection to postgres:1234 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections | Make sure the JDBC URL specifies a valid database host name and port, and that the port is accessible. | | PSQLException: FATAL: password authentication failed for user "postgres" | Verify that the User and Password are correct. | | IllegalArgumentException: Number of groups must be positive. | Make sure Include tables contains a valid tables list.Include tables setting is case-sensitive, even though the underlying database isn’t. Revise Include tables = tablename to Include Tables: tableName.Postgres occasionally refuses a connection for the first time. Retry creating the connector. | --- # Page 534: Create a MirrorMaker2 Checkpoint Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mmaker-checkpoint-connector.md --- # Create a MirrorMaker2 Checkpoint Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MirrorMaker2 Checkpoint Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mmaker-checkpoint-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mmaker-checkpoint-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mmaker-checkpoint-connector.adoc description: Use the Redpanda Cloud UI to create a MirrorMaker2 Checkpoint Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use the MirrorMaker2 Checkpoint connector to import consumer group offsets from other Kafka clusters. ## [](#prerequisites)Prerequisites - The external Kafka cluster is accessible. - A service account with read-only access to the external cluster is available. - The Kafka cluster topics connector is running for the same source cluster, with a matching configuration. ## [](#limitations)Limitations The MirrorMaker2 Checkpoint connector does not migrate consumer group offsets that are lower than the highest offsets synced by the MirrorMaker2 Source connector by the time the MirrorMaker2 Checkpoint connector is started. ## [](#create-a-mirrormaker2-checkpoint-connector)Create a MirrorMaker2 Checkpoint connector To create the MirrorMaker2 Checkpoint connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from Kafka cluster offsets**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to replicate | topics | Comma-separated topic names and regexes you want to replicate. | | Source cluster broker list | source.cluster.bootstrap.servers | A comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers regardless of which servers are specified here for bootstrapping. | | Source cluster security protocol | source.cluster.security.protocol | The protocol used to communicate with source brokers. The default is PLAINTEXT. | | Source cluster SASL mechanism | source.cluster.sasl.mechanism | SASL mechanism used for connections to source cluster. Default is PLAIN. | | Source cluster SASL username | source.cluster.sasl.username | SASL username used for connections to source cluster. | | Source cluster SASL password | source.cluster.sasl.password | SASL password used for connections to source cluster. | | Groups | groups | Consumer groups to replicate. Supports comma-separated group IDs and regexes. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-mirrormaker2-checkpoint-connector-configuration)Advanced MirrorMaker2 Checkpoint connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Source cluster SSL custom certificate | source.cluster.ssl.truststore.certificates | Trusted certificates in the PEM format. | | Source cluster SSL keystore key | source.cluster.ssl.keystore.key | Private key in the PEM format. | | Source cluster SSL keystore certificate chain | source.cluster.ssl.keystore.certificate.chain | Certificate chain in the PEM format. | | Topics exclude | topics.exclude | Excluded topics. Supports comma-separated topic names and regexes. | | Source cluster alias | source.cluster.alias | When using DefaultReplicationPolicy, topic names will be prefixed with it. | | Replication policy class | replication.policy.class | Class that defines the remote topic naming convention. Use IdentityReplicationPolicy to preserve topic names. DefaultReplicationPolicy prefixes the topic with the source cluster alias. | | Emit checkpoints interval seconds | emit.checkpoints.interval.seconds | Frequency of checkpoints. The default is 60. | | Sync group offsets enabled | sync.group.offsets.enabled | Specifies whether or not to periodically write the translated offsets to the __consumer_offsets topic in the target cluster, as long as no active consumers in that group are connected to the target cluster. | | Sync group offsets interval seconds | sync.group.offsets.interval.seconds | Frequency of consumer group offset sync. The default is 60. | | Refresh groups interval seconds | refresh.groups.interval.seconds | Frequency of group refreshes. The default is 600. | | Offset-Syncs topic location | offset-syncs.topic.location | The location (source or target) of the offset-syncs topic. The default is source. | | Checkpoints topic replication factor | checkpoints.topic.replication.factor | Replication factor for checkpoints topic. The default is -1. | ## [](#test-the-connection)Test the connection After the connector is created: - Ensure that there are no errors in logs and in Redpanda Console. - Wait for the Kafka cluster topics connector to catch up. Then check to confirm that the consumer groups are replicated. ## [](#use-the-connectors-api)Use the Connectors API When using the Connectors API, instead of specifying a value for `source.cluster.sasl.username` and `source.cluster.sasl.password`, you can specify a value for `source.cluster.sasl.jaas.config`. ## [](#troubleshoot)Troubleshoot Most MirrorMaker2 Checkpoint connector issues are reported as a failed task at the time of creation. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. / LOGS: Timed out while checking for or creating topic 'mm2-offset-syncs.target.internal'. This could indicate a connectivity issue / TimeoutException: Timed out waiting for a node assignment | Make sure broker URLs are correct and that the source cluster security protocol is correct. | | SaslAuthenticationException: SASL authentication failed: security: Invalid credentials | Check to confirm that the username and password specified are correct. | | java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config | Check to confirm that the username and password specified are correct. | | Client SASL mechanism 'PLAIN' not enabled in the server, enabled mechanisms are [SCRAM-SHA-256, SCRAM-SHA-512] | Check to confirm that the respective Source cluster SASL mechanism is correct. | | SaslAuthenticationException: SASL authentication failed: security: Invalid credentials | Make sure the respective Source cluster SASL mechanism is correct (for example, SCRAM-SHA-256 instead of SCRAM-SHA-512). | | terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue | Enable the SSL using Source cluster security protocol (specify SSL or SASL_SSL). | --- # Page 535: Create a MirrorMaker2 Heartbeat Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mmaker-heartbeat-connector.md --- # Create a MirrorMaker2 Heartbeat Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MirrorMaker2 Heartbeat Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mmaker-heartbeat-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mmaker-heartbeat-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mmaker-heartbeat-connector.adoc description: Use the Redpanda Cloud UI to create a MirrorMaker2 Heartbeat Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a MirrorMaker2 Heartbeat connector to generate heartbeat messages to a local cluster’s `heartbeat` topic. There are no prerequisites or limitations associated with this connector. ## [](#create-a-mirrormaker2-heartbeat-connector)Create a MirrorMaker2 Heartbeat connector To create the MirrorMaker2 Heartbeat connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from Heartbeat**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Emit heartbeats interval seconds | emit.heartbeats.interval.seconds | Frequency of heartbeats. The default is 1. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-mirrormaker2-heartbeat-connector-configuration)Advanced MirrorMaker2 Heartbeat connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Source cluster alias | source.cluster.alias | Used to generate the heartbeat topic key. The default is source. | | Target cluster alias | target.cluster.alias | Used to generate the heartbeat topic key. The default is target. | | Heartbeats topic replication factor | heartbeats.topic.replication.factor | Replication factor for heartbeats topic. The default is -1. | ## [](#test-the-connection)Test the connection After the connector is created, check to ensure that: - There are no errors in logs and in Redpanda Console. - Check to confirm the `heartbeat` topic has heartbeat messages. --- # Page 536: Create a MirrorMaker2 Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mmaker-source-connector.md --- # Create a MirrorMaker2 Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MirrorMaker2 Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mmaker-source-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mmaker-source-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mmaker-source-connector.adoc description: Use the Redpanda Cloud UI to create a MirrorMaker2 Source Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a MirrorMaker2 Source connector to import messages from another Kafka cluster. You can also use it to: - Replicate messages from an external Kafka or Redpanda cluster. - Create topics on the local cluster, with a configuration matching external topics. - Replicate topic access-control lists (ACLs). ## [](#prerequisites)Prerequisites - The external Kafka cluster must be accessible. - A service account with full access to the external cluster must be available. You can also use a service account with read-only ACLs when the `offset-syncs` topic location is set to `target`. You must have describe and/or describe-configs ACLs for the connector to read topic configurations on the source cluster and create the topics on the target cluster, unless you create the topics yourself. ## [](#limitations)Limitations - ACLs are copied, but service accounts are not created. - Only topic ACLs are copied (group ACLs are not). - Only ACLs for topics matching the connector configuration are copied (write ACLs are not copied). - All permissions ACLs are downgraded to read-only. ## [](#create-a-mirrormaker2-source-connector)Create a MirrorMaker2 Source connector To create the MirrorMaker2 Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from Kafka cluster topics**. 3. On the **Create Connector** form page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Regexes of topics to import | topics | Comma-separated topic names and regexes you want to replicate. | | Source cluster broker list | source.cluster.bootstrap.servers | A comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers regardless of which servers are specified here for bootstrapping. This list only impacts the initial hosts used to discover the full set of servers, and should be in the form host1:port1,host2:port2,.... Because these servers are only used for the initial connection to discover the full cluster membership (which may change dynamically), it need not contain the full set of servers (you may want more than one, though, in case a server is down). | | Source cluster security protocol | source.cluster.security.protocol | The protocol to use to communicate with source brokers. Default is PLAINTEXT. | | Source cluster SASL mechanism | source.cluster.sasl.mechanism | SASL mechanism used for connections to source cluster. Default is PLAIN. | | Source cluster SASL username | source.cluster.sasl.username | SASL username used for connections to source cluster. | | Source cluster SASL password | source.cluster.sasl.password | SASL password used for connections to source cluster. | | Sync topic configs enabled | sync.topic.configs.enabled | Specifies whether to periodically configure remote topics to match their corresponding upstream topics. | | Sync topic ACLs enabled | sync.topic.acls.enabled | Specifies whether or not to periodically configure remote topic ACLs to match their corresponding upstream topics. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. > 📝 **NOTE** > > Offsets are not guaranteed to match between the source and target. For example, if data-retention deletes occur on the source topic and the earliest offset is `#5000`, then when that event is created on the target topic the offset for that event will be `#0`. > > Events written on the target topic use the timestamp that was set on the source event. For example, if the source event has a timestamp `2023-05-22 17:00`, then this would also be the timestamp on the target event. ### [](#advanced-mirrormaker2-source-connector-configuration)Advanced MirrorMaker2 Source connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Source cluster SSL custom certificate | source.cluster.ssl.truststore.certificates | Trusted certificates in the PEM format. | | Source cluster SSL keystore key | source.cluster.ssl.keystore.key | Private key in the PEM format. | | Source cluster SSL keystore certificate chain | source.cluster.ssl.keystore.certificate.chain | Certificate chain in the PEM format. | | Sync topic configs interval seconds | sync.topic.configs.interval.seconds | Frequency of topic config sync. | | Sync topic ACLs interval seconds | sync.topic.acls.interval.seconds | Frequency of topic ACL sync. | | Topics exclude | topics.exclude | Excluded topics. Supports comma-separated topic names and regexes. | | Source cluster alias | source.cluster.alias | When using DefaultReplicationPolicy, topic names will be prefixed with it. | | Replication policy class | replication.policy.class | Class that defines the remote topic naming convention. Use IdentityReplicationPolicy to preserve topic names. DefaultReplicationPolicy prefixes the topic with the source cluster alias. | | Replication factor | replication.factor | Replication factor for newly created remote topics. Set -1 for cluster default. | | Refresh topics interval seconds | refresh.topics.interval.seconds | Frequency of topic refresh. | | Offset-Syncs topic location | offset-syncs.topic.location | The location (source or target) of the offset-syncs topic. The default is source. | | Offset-Syncs topic replication factor | offset-syncs.topic.replication.factor | Replication factor for offset-syncs topic. The default is -1. | | Config properties exclude | config.properties.exclude | Topic config properties that should not be replicated. Supports comma-separated property names and regexes. | | Compression type | producer.override.compression.type | The compression type for all data generated by the producer. The default is none (no compression). | | Max size of a request | producer.override.max.request.size | The maximum size of a request in bytes. The default is 1048576. | | Auto offset reset | consumer.auto.offset.reset | What to do when there is no initial offset in Kafka, or if the current offset does not exist any more on the server (for example, because that data has been deleted). 'earliest' - automatically reset the offset to the earliest offset. 'latest' - automatically reset the offset to the latest offset. 'none' - throw exception to the consumer if no previous offset is found for the consumer’s group. | | Offset lag max | offset.lag.max | How out-of-sync a remote partition can be before it is resynced. This setting impacts the MirrorMaker2 Checkpoint connector as it is the maximum lag for syncing consumer groups. The default is 100 records. | ## [](#map-data)Map data The value converter does not require any schema; it copies data as bytes. ## [](#test-the-connection)Test the connection After the connector is created: - Ensure that there are no errors in logs and in Redpanda Console. - Confirm that Redpanda topics are being replicated. You should see messages coming into the topics. ## [](#use-the-connectors-api)Use the Connectors API When using the Connectors API, instead of specifying a value for `source.cluster.sasl.username` and `source.cluster.sasl.password`, you can specify a value for `source.cluster.sasl.jaas.config`. ## [](#troubleshoot)Troubleshoot Most MirrorMaker2 Source connector issues are reported as a failed task at the time of creation. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available. / LOGS: Timed out while checking for or creating topic 'mm2-offset-syncs.target.internal'. This could indicate a connectivity issue / TimeoutException: Timed out waiting for a node assignment | Make sure broker URLs are correct and that the security.protocol is correct. | | SaslAuthenticationException: SASL authentication failed: security: Invalid credentials | Confirm that the username and password specified are correct. | | Terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue | Error indicates that the SSL should be enabled using Source cluster security protocol (use SSL or SASL_SSL). | | RecordTooLargeException: The message is N bytes (…​) | Use producer.override.max.request.size property to change max request size. | | RecordTooLargeException: The request included (…​) | The target server is not able to receive messages because it is too large in size. Disabled compression can be a root cause. Consider enabling compression: "Compression type": "snappy", | | Scheduler for MirrorSourceConnector caught exception in scheduled task: syncing topic ACLs | MirrorMaker2 requires an authorizer to be configured by the broker side, but it is not. Change the Sync topic ACLs enabled MirrorMaker2 property to false (default is true) to disable ACL syncing. | | TopicAuthorizationException: Topic authorization failed | Confirm the service account for the source cluster contains describe and/or describe-configs ACLs. | | OffsetOutOfRangeException Fetch position FetchPosition{offset=0, …​ ] | If the 0 offset for your topic does not exist in the source cluster, set Auto offset reset to either earliest or latest. | --- # Page 537: Create a MongoDB Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mongodb-sink-connector.md --- # Create a MongoDB Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MongoDB Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mongodb-sink-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mongodb-sink-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mongodb-sink-connector.adoc description: Use the Redpanda Cloud UI to create a MongoDB Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. The MongoDB Sink managed connector exports Redpanda structured data to a MongoDB database. ## [](#prerequisites)Prerequisites - Valid credentials with the `readWrite` role to access the MongoDB database. For more granular access, you need to allow `insert`, `remove` and `update` actions for specific databases or collections. ## [](#limitations)Limitations If you want to use the MongoDB sink connector with the `MongoDB` CDC handler for data sourced from MongoDB (using the MongoDB source connector), you must select `STRING` or `BYTES` as the value converter for both the source and sink connectors. ## [](#create-a-mongodb-sink-connector)Create a MongoDB Sink connector To create a MongoDB Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to MongoDB Sink**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | A comma-separated list of the cluster topics you want to export to MongoDB. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | MongoDB Connection URL | connection.url | The MongoDB connection URI string to connect to your MongoDB instance or cluster. For example, mongodb://locahost/. | | MongoDB username | connection.username | A valid MongoDB user. | | MongoDB password | connection.password | The password for the account associated with the MongoDB user. | | MongoDB database name | database | The name of an existing MongoDB database to store output files in. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. Default is STRING. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. Default is STRING. | | Default MongoDB collection name | collection | (Optional). Single sink collection name to write to. If following multiple topics, then this will be the default collection to which they are mapped. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-mongodb-sink-connector-configuration)Advanced MongoDB Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | CDC handler | change.data.capture.handler | The CDC (change data capture) handler to use for processing. The MongoDB handler requires plain JSON or BSON format. The default is NONE. | | Key projection type | key.projection.type | The type of key projection to use: either AllowList or BlockList. | | Key projection list | key.projection.list | A comma-separated list of field names for key projection. | | Value projection type | value.projection.type | Only use with Value projection list. The type of value projection to use: AllowList or BlockList. The default is NONE. | | Value projection list | value.projection.list | A comma-separated list of field names for value projection. | | Field renamer mapping | field.renamer.mapping | An inline JSON array with objects describing field name mappings. For example: [{"oldName":"key.fieldA","newName":"field1"},{"oldName":"value.xyz","newName":"abc"}]. | | Field used for time | timeseries.timefield | Name of the top level field used for time. Inserted documents must specify this field, and it must be of the BSON datetime type. | | Field describing the series | timeseries.metafield | The name of the top-level field that contains metadata in each time series document. The metadata in the specified field should be data that is used to label a unique series of documents. The metadata should rarely, if ever, change. This field is used to group related data and may be of any BSON type, except for array. The metadata field may not be the same as the timeField or _id. | | Convert the field to a BSON datetime type | timeseries.timefield.auto.convert | Converts the timeseries field to a BSON datetime type. If the value is a numeric value it will use the milliseconds from epoch. Any fractional parts are discarded. If the value is a STRING it will use the timeseries.timefield.auto.convert.date.format property to parse the date. | | DateTimeFormatter pattern for the date | timeseries.timefield.auto.convert .date.format | The DateTimeFormatter pattern to use when converting string dates. Defaults to support ISO style date times. A string is expected to contain both the date and time. If the string only contains date information, then the time since epoch is taken from the start of that day. If a string representation does not contain a timezone offset, then the extracted date and time is interpreted as UTC. | | Data expiry time in seconds | timeseries.expire.after.seconds | The amount of time in seconds that the data will be kept in MongoDB before being automatically deleted. | | Data expiry time | timeseries.granularity | The expected interval between subsequent measurements for a time series. Possible values are "seconds", "minutes" or "hours". | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` (`org.apache.kafka.connect.json.JsonConverter`) when your messages are structured JSON. Select `Message JSON contains schema`, with the `schema` and `payload` fields. - `AVRO` (`io.confluent.connect.avro.AvroConverter`) when your messages contain AVRO-encoded messages, with schema stored in the Schema Registry. - `STRING` (`org.apache.kafka.connect.storage.StringConverter`) when your messages contain plaintext JSON. - `BYTES` (`org.apache.kafka.connect.converters.ByteArrayConverter`) when your messages contain BSON. ## [](#test-the-connection)Test the connection After the connector is created, verify that your new collections apper in your MongoDB database: show collections ## [](#use-the-connectors-api)Use the Connectors API When using the Connectors API, instead of specifying a value for `connection.url`, `connection.username`, and `connection.password`, you can specify a value for `connection.uri` in the form `mongodb+srv://username:password@cluster0.xxx.mongodb.net`. ## [](#troubleshoot)Troubleshoot Issues are reported using a failed task error message. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | Invalid value wrong_uri for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv:// | Check to make sure the Connection URI is a valid MongoDB URL. | | Unable to connect to the server. | Check to ensure that the Connection URI is valid and that the MongoDB server accepts connections. | | Invalid user permissions authentication failed. Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='user', source='admin', password=, mechanismProperties=}. | Check to ensure that you specified valid username and password credentials. | | DataException: Could not convert key into a BsonDocument. | Make sure your message keys are valid JSONs or skip configuration for fields that require valid JSON keys. | | DataException: Error: operationType field doc is missing. | Make sure the input record format is correct (produced by a MongoDB source connector if you use MongoDB CDC handler). | | DataException: Value document is missing or CDC operation is not a string | Make sure the input record format is correct (produced by a Debezium source connector if you use Debezium CDC handler). | | JsonParseException: Unrecognized token 'text': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') | Make sure the input record format is JSON. | | Unexpected documentKey field type, expecting a document but found BsonString…​: {…​} | Make sure the source data is in the plain JSON or BSON format (value converter STRING or BYTES). | ## [](#suggested-reading)Suggested reading - [MongoDB Kafka Sink Connector](https://www.mongodb.com/docs/kafka-connector/current/sink-connector/) --- # Page 538: Create a MongoDB Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mongodb-source-connector.md --- # Create a MongoDB Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MongoDB Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mongodb-source-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mongodb-source-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mongodb-source-connector.adoc description: Use the Redpanda Cloud UI to create a MongoDB Source Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. The MongoDB Source managed connector imports collections from a MongoDB database into Redpanda topics. ## [](#prerequisites)Prerequisites - Valid credentials with the `read` role to access the MongoDB database. For more granular access, you need to allow `find` and `changeStream` actions for specific databases or collections. ## [](#create-a-mongodb-source-connector)Create a MongoDB Source connector To create a MongoDB Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from MongoDB**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topic prefix | topic.prefix | Prefix to prepend to database and collection names to generate the name of the Kafka topic to which to publish data. Used by the DefaultTopicMapper. | | MongoDB Connection URL | connection.url | The MongoDB connection URL string as supported by the official drivers. For example, mongodb://locahost/. | | MongoDB username | connection.username | A valid MongoDB user. | | MongoDB password | connection.password | The password for the account associated with the MongoDB user. | | Database to watch | database | The MongoDb database from which the connector imports data into Redpanda topics. The connector monitors changes in this database. Leave the field empty to watch all databases. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. Default is STRING. Use AVRO or JSON for schematic output, STRING for plain JSON, or BYTES for BSON. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. Default is STRING. Use AVRO or JSON for schematic output, STRING for plain JSON, or BYTES for BSON. | | Collection to watch | collection | The collection in the MongoDB database to watch. If not set, then all collections are watched. | | Start up behavior when there is no source offset available | startup.mode | Specifies how the connector should start up when there is no source offset available. Resuming a change stream requires a resume token, which the connector stores as reads from the source offset. If no source offset is available, the connector may either ignore all or some existing source data, or may at first copy all existing source data and then continue with processing new data. Possible values are:latest (default): The connector creates a new change stream, processes change events from it and stores resume tokens from them, thus ignoring all existing source data.timestamp: actuates startup.mode.timestamp.* properties. If no such properties are configured, then timestamp is equivalent to latest.copy_existing: actuates startup.mode.copy.existing.* properties. The connector creates a new change stream and stores its resume token, copies all existing data from all the collections being used as the source, then processes new data starting from the stored resume token. Note that reads of all the data during the copy and subsequent change stream events may produce duplicated events. During the copy, clients can make changes to the source data, which may be represented both by the copying process and the change stream. However, as the change stream events are idempotent, it’s possible to apply them multiple times with the same effect as if they were applied once. Renaming a collection during the copying process is not supported. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-mongodb-source-connector-configuration)Advanced MongoDB Source connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Enable Infer Schemas for the value | output.schema.infer.value | Specifies whether or not to infer the schema for the value. Each Document is processed in isolation, which may lead to multiple schema definitions for the data. Only enable when Kafka message value format is set to AVRO or JSON. | | startAtOperationTime | startup.mode.timestamp .start.at.operation.time | Actuated only if startup.mode = timestamp specifies the starting point for the change stream. Must be either an integer number of seconds because the Epoch is in the decimal format (for example: 30), or an instant in the ISO-8601 format with one second precision (for example: 1970-01-01T00:00:30Z), or a BSON timestamp in the canonical extended JSON (v2) format (for example: {"$timestamp": {"t": 30, "i": 0}}). You can specify 0 to start at the beginning of the oplog. Requires MongoDB 4.0 or above. For more detail, see the $changeStream definition. | | Copy existing namespace regex | startup.mode.copy.existing .namespace.regex | Use a regular expression to define which existing namespaces data should be copied from. A namespace is the database name and collection, separated by a period (for example, database.collection). Example: The following regular expression only includes collections starting with a in the demo database: demo\.a.*. | | Copy existing initial pipeline | startup.mode.copy.existing .pipeline | An inline JSON array with objects describing the pipeline operations to run when copying existing data. Specifying this property can improve the use of indexes by the copying manager and make copying more efficient. Use this property if there is any filtering of collection data in the pipeline configuration to speed up the copying process. For example: [{"$match": {"closed": "false"}}]. | | Pipeline to apply to the change stream | pipeline | An inline JSON array with objects describing the pipeline operations to run. For example: [{"$match": {"operationType": "insert"}}, {"$addFields": {"Kafka": "Rules!"}}]. | | fullDocument | change.stream.full.document | Specifies what to return for update operations when using a change stream. When set to updateLookup, the change stream for partial updates will include both a delta describing the changes to the document, and a copy of the entire document that was changed _ at some point_ after the change occurred. See db.collection.watch for more detail. | | fullDocumentBeforeChange | change.stream.full.document .before.change | Specifies the pre-image configuration when creating a change stream. The pre-image is not available in source records published while copying existing data as a result of enabling copy.existing. The pre-image configuration has no effect on copying. Requires MongoDB 6.0 or above. For details, see possible values. | | Publish only the fullDocument | publish.full.document.only | When enabled, only publishes the actual changed document (rather than the full change stream document). Automatically sets change.stream.full.document=updateLookup so updated documents will be included. | | Send a null value on a delete event | publish.full.document.only .tombstone.on.delete | When enabled, requires publish.full.document.only=true. Default is false (disabled). | | Error tolerance | mongo.errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Heartbeat interval milliseconds | heartbeat.interval.ms | The length of time it takes when sending heartbeat messages to record the post-batch resume token when no source records have been published. Improves the resumability of the connector for low volume namespaces. Specify 0 to disable. | | heartbeat topic name | heartbeat.topic.name | The name of the topic to publish heartbeats to. Defaults to __mongodb_heartbeats. | | Offset partition name | offset.partition.name | Use to specify a custom offset partition name. If blank, the default partition name based on the connection details is used. | | Topic creation enabled | topic.creation.enable | Specifies whether or not to allow automatic creation of topics. Default is true. | | Topic creation partitions | topic.creation.default. partitions | Specifies the number of partitions for the created topics. The default is 1. | | Topic creation replication factor | topic.creation.default. replication.factor | Specifies the replication factor for the created topics. The default is -1. | ## [](#map-data)Map data - `AVRO` (`io.confluent.connect.avro.AvroConverter`) or `JSON` (`org.apache.kafka.connect.json.JsonConverter`) for output with a preset schema. Additionally, you can set `Enable Infer Schemas` for the value. Each document will be processed in isolation, which may lead to multiple schema definitions for the data. - `STRING` (`org.apache.kafka.connect.storage.StringConverter`) when your messages contain plaintext JSON. - `BYTES` (`org.apache.kafka.connect.converters.ByteArrayConverter`) when your messages contain BSON. After the connector is created, check to ensure that: - There are no errors in logs and in Redpanda Console. - Redpanda topics contain data from relational database tables. ## [](#use-the-connectors-api)Use the Connectors API When using the Connectors API, instead of specifying a value for `connection.url`, `connection.username`, and `connection.password`, you can specify a value for `connection.uri` in the form `mongodb+srv://username:password@cluster0.xxx.mongodb.net`. ## [](#troubleshoot)Troubleshoot Most MongoDB Source connector issues are identified in the connector creation phase. Invalid Include Tables are reported in logs. Select **Show Logs** to view error details. | Message | Action | | --- | --- | | Invalid value wrong_uri for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv:// | Check to make sure the MongoDB Connection URL is a valid MongoDB URL. | | Unable to connect to the server. | Check to ensure that the MongoDB Connection URL is valid and that the MongoDB server accepts connections. | | Invalid user permissions authentication failed. Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='user', source='admin', password=, mechanismProperties=}. | Check to ensure that you specified valid username and password credentials. | | MongoCommandException: Command failed with error 8000 (AtlasError): 'user is not allowed to do action [find] on [db1.characters]' on server ac-nboibsg-shard-00-01.4hagsz0.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "user is not allowed to do action [find] on [db1.characters]", "code": 8000, "codeName": "AtlasError"} | Check the permissions of the MongoDB user. Also confirm that the MongoDB server accepts connections. | | Command failed with error 286 (ChangeStreamHistoryLost): 'PlanExecutor error during aggregation :: caused by :: Resume of change stream was not possible, as the resume point may no longer be in the oplog | See Troubleshoot invalid resume token | ## [](#suggested-reading)Suggested reading - [MongoDB Kafka Source Connector](https://www.mongodb.com/docs/kafka-connector/current/source-connector/) --- # Page 539: Create a MySQL (Debezium) Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-mysql-source-connector.md --- # Create a MySQL (Debezium) Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a MySQL (Debezium) Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-mysql-source-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-mysql-source-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-mysql-source-connector.adoc description: Use the Redpanda Cloud UI to create a MySQL (Debezium) Source Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a MySQL (Debezium) Source connector to import a stream of changes from MySQL, AmazonRDS, and Amazon Aurora. ## [](#prerequisites)Prerequisites - A MySQL database that is accessible from the connector instance. - A MySQL user exists. This database user for the Debezium connector must have LOCK TABLES privileges. For details, see [MySQL Creating a user](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-creating-user). - A [binlog must be enabled](https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-mysql-binlog) for the source MySQL cluster. ## [](#limitations)Limitations - Only `JSON`, `CloudEvents` or `AVRO` formats can be used as a a Kafka message key and value format. - The MySQL (Debezium) Source connector can work with only a single task at a time. ## [](#create-a-mysql-debezium-source-connector)Create a MySQL (Debezium) Source connector To create the MySQL (Debezium) Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from MySQL (Debezium)**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topic prefix | topic.prefix | A topic prefix that identifies and provides a namespace for the particular database server/cluster that is capturing changes. The topic prefix should be unique across all other connectors because it is used as a prefix for all Kafka topic names that receive events emitted by this connector. Only alphanumeric characters, hyphens, dots, and underscores are accepted. | | Hostname | database.hostname | A resolvable hostname or IP address of the MySQL database server. | | Port | database.port | Integer port number of the MySQL database server. | | User | database.user | Name of the MySQL user to be used when connecting to the MySQL database. | | Password | database.password | The password of the MySQL database user who will be connecting to the MySQL database. | | SSL mode | database.ssl.mode | Specifies whether to use an encrypted connection to the MySQL server. Select disable to use an unencrypted connection. Select 'preferred' to use an encrypted connection if the server supports secure connections. If the server does not support secure connections, falls back to an unencrypted connection. Select require to use a secure, or encrypted connection. If a secure connection cannot be established when required is selected, then the connector fails. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. | | Message key JSON contains schema | key.converter.schemas.enable | Enable to specify that the message key contains schema in the schema field. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. | | Message value JSON contains schema | value.converter.schemas.enable | Enable to specify that the message value contains schema in the schema field. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ## [](#map-data)Map data Use `Include databases`, `Include tables`, and `Include columns` to define data mapping. Alternatively, use `Exclude databases`, `Exclude tables`, and `Exclude columns`. Following is an example table in `db` database: ```sql CREATE TABLE IF NOT EXISTS Persons ( Id int PRIMARY KEY, FirstName varchar(255), LastName varchar(255) ); ``` The table has one record: ```sql INSERT INTO Persons (FirstName, LastName) VALUES (1, 'Winnie', 'the Pooh'); ``` The connector configuration for the table: ```bash column.include.list = db\\.Persons\\.(Id|FirstName|LastName) table.include.list = db\\.Persons database.include.list = db topic.prefix = frommysql ``` The connector configuration will create the Redpanda topic `frommysql.db.Persons`. For `Kafka message value format` = `JSON` (`org.apache.kafka.connect.json.JsonConverter`), the connector produces JSON messages with a schema like the following: ```json { "payload": { "schema": { // schema definition }, "payload": { "before": null, "after": { "Id": 1, "FirstName": "Winnie", "LastName": "the Pooh" }, ... } }, "encoding": "json", "schemaId": 0 } ``` For `Kafka message value format` = `AVRO` (`io.confluent.connect.avro.AvroConverter`), the connector creates a Schema Registry `frommysql.db.Persons-value` record and produces messages like the following: ```js { "payload": { "before": null, "after": { "mysql.db.Persons.Value": { "Id": 1, "FirstName": { "string": "Winnie" }, "LastName": { "string": "the Pooh" } } }, ... }, "encoding": "avro", "schemaId": 2 } ``` For `Kafka message value format` = `CloudEvents` (`io.debezium.converters.CloudEventsConverter`), the connector uses `JSON` or `AVRO` data serializer. - For `JSON` data serializer, enable `Message value CloudEvents JSON contains schema` to include JSON schema in message - For `AVRO` data serializer, connector creates schema in Schema Registry and produces messages in CloudEvents data format. ## [](#test-the-connection)Test the connection After the connector is created: - Check the connector status and confirm that there are no errors in logs and in Redpanda Console. - Review the Redpanda topic to confirm that it contains the expected data. ## [](#troubleshoot)Troubleshoot If the connector configuration is invalid, an error appears upon clicking **Finish**. If the connector fails, check the error message or select **Show Logs** to view error details. - **Topics not created by the connector** Create the topic manually or let the connector create it by setting (use desired number of partitions and replication factor): Topic creation enabled: true Topic creation partitions: 1 Topic creation replication factor: -1 Or in JSON: ```json "topic.creation.enable": true, "topic.creation.default.partitions": "1", "topic.creation.default.replication.factor": "-1" ``` - **Connector requires binlog file 'mysql-bin-changelog.257116', but MySQL only has mysql-bin-changelog.257123** Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted" Connector requires binlog file 'mysql-bin-changelog.257116', but MySQL only has mysql-bin-changelog.257123, mysql-bin-changelog.257124, mysql-bin-changelog.257125 The connector needs a binlog file that was already purged. Change the `Snapshot mode` property from the default to `when_needed`. Additional errors and corrective actions follow. | Message | Action | | --- | --- | | Unable to connect: Public Key Retrieval is not allowed | Set Allow public key retrieval property to true. | | Unable to connect: Communications link failure | Confirm that Hostname and Port are correct. | | Access denied for user | Confirm that User and Password credentials are valid. | | Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Invalid schema Invalid namespace: from-mysql.db.Persons; error code: 422 | The Schema Registry namespace is incorrect. Consider changing the Topic prefix value, remove unallowed characters. | ## [](#suggested-reading)Suggested reading - [Debezium connector for MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html) --- # Page 540: Create a PostgreSQL (Debezium) Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-postgresql-connector.md --- # Create a PostgreSQL (Debezium) Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a PostgreSQL (Debezium) Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-postgresql-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-postgresql-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-postgresql-connector.adoc description: Use the Redpanda Cloud UI to create a PostgreSQL (Debezium) Source Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use a PostgreSQL (Debezium) Source connector to import updates to Redpanda from PostgreSQL. ## [](#prerequisites)Prerequisites Before you can create a PostgreSQL (Debezium) Source connector in the Redpanda Cloud, you must: - [Make the PostgreSQL (Debezium) database accessible](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-security) from connectors instance. - [Create a PostgreSQL (Debezium) user](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-permissions) with the necessary permissions. ## [](#limitations)Limitations The PostgreSQL (Debezium) Source connector has the following limitations: - Only `JSON`, `CloudEvents` or `AVRO` formats can be used for a Kafka message key and value format. - PostgreSQL (Debezium) connector can work with only a single task at a time. ## [](#create-a-postgresql-debezium-source-connector)Create a PostgreSQL (Debezium) Source connector To create the PostgreSQL (Debezium) Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from PostgreSQL (Debezium)**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topic prefix | topic.prefix | A topic prefix that identifies and provides a namespace for the particular database server/cluster that is capturing changes. The topic prefix should be unique across all other connectors because it is used as a prefix for all Kafka topic names that receive events emitted by this connector. Only alphanumeric characters, hyphens, dots, and underscores are accepted. | | Hostname | database.hostname | A resolvable hostname or IP address of the PostgreSQL database server. | | Port | database.port | Integer port number of the PostgreSQL database server. | | User | database.user | Name of the PostgreSQL user to be used when connecting to the PostgreSQL database. | | Password | database.password | The password of the PostgreSQL database user who will be connecting to the PostgreSQL database. | | Database | database.dbname | The name of the database from which the connector will import changes. | | SSL mode | database.sslmode | Specifies whether to use an encrypted connection to the PostgreSQL server. Select disable to use an unencrypted connection. Select require to use a secure, or encrypted connection. If a secure connection cannot be established when required is selected, then the connector fails. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. | | Message key JSON contains schema | key.converter.schemas.enable | Enable to specify that the message key contains schema in the schema field. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. | | Message value JSON contains schema | value.converter.schemas.enable | Enable to specify that the message value contains schema in the schema field. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - Use `Include Schemas`, `Include Tables` and `Include Columns` properties to define lists of columns, tables, and schemas to read from. Alternatively, use `Exclude Schemas`, `Exclude Tables`, and `Exclude Columns` to define lists of columns, tables, and schemas to exclude from sources list. - Use only `JSON` (`org.apache.kafka.connect.json.JsonConverter`), `AVRO` (`io.confluent.connect.avro.AvroConverter`) and `CloudEvents` (`io.debezium.converters.CloudEventsConverter`) formats for the Kafka message key and value format. ## [](#test-the-connection)Test the connection After the connector is created: 1. Open Redpanda Console, click the **Topics** tab and select a topic. Check to check to confirm that it contains data migrated from PostgreSQL. Alternatively, use the `rpk consume` to check the topic. 2. Click the **Connectors** tab to confirm no issues have been reported for the connector. ## [](#troubleshoot)Troubleshoot If the connector configuration is invalid, an error appears upon clicking **Finish**. Select **Show Logs** to view error details. Additional errors and corrective actions follow. | Message | Action | | --- | --- | | Missing tables or topics | The Debezium connector replicates tables one by one. Wait for other tables to be replicated. If the database is quite large, then replication takes longer to complete. | | non-existing-db | Make sure the provided database name in Database is correct, and that the database exists. | | The connection attempt failed / Connection to postgres:9999 refused | Check to make sure that hostname and port are correct. | | Password authentication failed for user | Make sure that the User and Password credentials are valid. | | The Plugin name value is invalid | Make sure that Plugin contains a valid value, either decoderbufs or pgoutput. | | Postgres server wal_level property is replica | Specify wal_level as logical for your database. | | RecordTooLargeException: The message is 1050766 bytes when serialized, which is larger than 1048576, the value of the max.request.size configuration. | Increase the max request size to unblock the connector and allow large messages to pass: "producer.override.max.request.size": "209715200". The connector may be reaching memory limits and failing if the amount of data to pass or your messages are too large. | ## [](#suggested-reading)Suggested reading - [Debezium connector for PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) --- # Page 541: Create an S3 Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-s3-sink-connector.md --- # Create an S3 Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create an S3 Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-s3-sink-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-s3-sink-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-s3-sink-connector.adoc description: Use the Redpanda Cloud UI to create an AWS S3 Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. The Amazon S3 Sink connector exports Apache Kafka messages to files in AWS S3 buckets. ## [](#prerequisites)Prerequisites Before you can create an AWS S3 sink connector in the Redpanda Cloud, you must complete these tasks: 1. [Create an AWS account](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html). 2. [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) that you will send data to. 3. [Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) that will be used to connect to the S3 service. 4. [Attach the following policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html) to the user, replacing `bucket-name` with the name you specified in step 2. ```js { "Version": "2012-10-17", "Statement": [ { "Principal": "*", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::bucket-name/*", "arn:aws:s3:::bucket-name" ] } ] } ``` 5. [Create access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for the user created in step 3. 6. Copy the access key ID and the secret access key. You will need them to configure the connector. ## [](#limitations)Limitations - You can use only the `STRING` and `BYTES` input formats for `CSV` output format. - You can use only the `PARQUET` format when your messages contain schema. ## [](#create-an-aws-s3-sink-connector)Create an AWS S3 Sink connector To create the AWS S3 Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to S3**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | Comma-separated list of the cluster topics whose records will be exported to the S3 bucket. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | AWS access key ID | aws.access.key.id | Enter the AWS access key ID. | | AWS secret access key | aws.secret.access.key | Enter the AWS secret access key. | | AWS S3 bucket name | aws.s3.bucket.name | Specify the name of the AWS S3 bucket to which the connector is to send data. | | AWS S3 region | aws.s3.region | Select the region for the S3 bucket used for storing the records. The default us-east-1. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. The default is BYTES. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. The default is BYTES. | | S3 file format | format.output.type | Format of the files created in S3: CSV (the default), AVRO, JSON, JSONL, or PARQUET. You can use the CSV format output only with BYTES and STRING. | | Avro codec | avro.codec | The Avro compression codec to be used for Avro output files. Available values: null (the default), deflate, snappy, and bzip2. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-aws-s3-sink-connector-configuration)Advanced AWS S3 Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | File name template | file.name.template | The template for file names on S3. Supports {{ variable }} placeholders for substituting variables. Supported placeholders are:topicpartitionstart_offset (the offset of the first record in the file)timestamp:unit=yyyy|MM|dd|HH (the timestamp of the record)key (when used, other placeholders are not substituted) | | File name prefix | file.name.prefix | The prefix to be added to the name of each file put in S3. | | Output fields | format.output.fields | Fields to place into output files. Supported values are: 'key', 'value', 'offset', 'timestamp', and 'headers'. | | Value field encoding | format.output.fields.value.encoding | The type of encoding to be used for the value field. Supported values are: 'none' and 'base64'. | | Envelope for primitives | format.output.envelope | Specifies whether or not to enable additional JSON object wrapping of the actual value. | | Output file compression | file.compression.type | The compression type to be used for files put into S3. Supported values are: 'none' (default), 'gzip', 'snappy', and 'zstd'. | | Max records per file | file.max.records | The maximum number of records to put in a single file. Must be a non-negative number. 0 is interpreted as "unlimited", which is the default. In this case files are only flushed after file.flush.interval.ms. | | File flush interval milliseconds | file.flush.interval.ms | The time interval to periodically flush files and commit offsets. Value specified must be a non-negative number. Default is 60 seconds. 0 indicates that it is disabled. In this case, files are only flushed after reaching file.max.records record size. | | AWS S3 bucket check | aws.s3.bucket.check | If set to true (default), the connector will attempt to put a test file to the S3 bucket to validate access. | | AWS S3 part size bytes | s3.part.size | The part size in S3 multi-part uploads in bytes. Maximum is 2147483647 (2GB) and default is 5242880 (5MB). | | S3 retry backoff | aws.s3.backoff.delay.ms | S3 default base sleep time (in milliseconds) for non-throttled exceptions. Default is 100. | | S3 maximum back-off | aws.s3.backoff.max.delay.ms | S3 maximum back-off time (in milliseconds) before retrying a request. Default is 20000. | | S3 max retries | aws.s3.backoff.max.retries | Maximum retry limit (if the value is greater than 30, there can be integer overflow issues during delay calculation). Default is 3. | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` (`org.apache.kafka.connect.json.JsonConverter`) when your messages are JSON-encoded. Select `Message JSON contains schema`, with the `schema` and `payload` fields. - `AVRO` (`io.confluent.connect.avro.AvroConverter`) when your messages contain AVRO-encoded messages, with schema stored in the Schema Registry. - `STRING` (`org.apache.kafka.connect.storage.StringConverter`) when your messages contain textual data. - `BYTES` (`org.apache.kafka.connect.converters.ByteArrayConverter`) when your messages contain arbitrary data. You can also select the output data format for your S3 files as follows: - `CSV` to produce data in the `CSV` format. For `CSV` only, you can set `STRING` and `BYTES` input formats. - `JSON` to produce data in the `JSON` format as an array of record objects. - `JSONL` to produce data in the `JSON` format, each message as a separate JSON, one per line. - `PARQUET` to produce data in the `PARQUET` format when your messages contain schema. - `AVRO` to produce data in the `AVRO` format when your messages contain schema. ## [](#test-the-connection)Test the connection After the connector is created, test the connection by writing to one of your topics, then checking the contents of the S3 bucket in the AWS management console. Files should appear after the file flush interval (default is 60 seconds). ## [](#troubleshoot)Troubleshoot If there are any connection issues, an error message is returned. Depending on the `AWS S3 bucket check` property value, the error results in a failed connector (`AWS S3 bucket check = true`) or a failed task (`AWS S3 bucket check = false`). Select **Show Logs** to view error details. Additional errors and corrective actions follow. | Message | Action | | --- | --- | | The AWS Access Key Id you provided does not exist in our records | AWS access key ID is invalid. Check to confirm that a valid existing AWS access key is specified. | | The authorization header is malformed; the region us-east-1 is wrong; expecting us-east-2 | The selected region (AWS S3 region) of the AWS bucket is incorrect. Check to confirm that you have specified the region in which the bucket was created. | | The specified bucket does not exist | Create the bucket specified in the AWS S3 bucket name property, or provide the correct name of the existing bucket. | | No files in the S3 bucket | Be sure to wait until the connector completes the first file flush (default 60 seconds). Verify that the topics specified are correct. Then verify that the topics contain messages to be pushed to S3. | --- # Page 542: Create a Snowflake Sink Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-snowflake-connector.md --- # Create a Snowflake Sink Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a Snowflake Sink Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-snowflake-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-snowflake-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-snowflake-connector.adoc description: Use the Redpanda Cloud UI to create a Snowflake Sink Connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use the Snowflake Sink connector to ingest and store Redpanda structured data into a Snowflake database for analytics and decision-making. ## [](#prerequisites)Prerequisites Before you can create a Snowflake Sink connector in the Redpanda Cloud, you must: 1. [Create a role](https://docs.snowflake.com/en/user-guide/kafka-connector-install#creating-a-role-to-use-the-kafka-connector) for use by Kafka Connect. 2. [Create a key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth#configuring-key-pair-authentication) for authentication. 3. [Create a database](https://docs.snowflake.com/en/user-guide/getting-started-tutorial-create-objects#creating-a-database) to hold the data you intend to stream from Redpanda Cloud messages. ## [](#limitations)Limitations Refer to the [Snowflake Kafka Connector Limitations](https://docs.snowflake.com/en/user-guide/kafka-connector-overview#kafka-connector-limitations) documentation for details. ## [](#create-a-snowflake-sink-connector)Create a Snowflake Sink connector To create a Snowflake Sink connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Export to Snowflake**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topics to export | topics | A comma-separated list of the cluster topics you want to export to Snowflake. | | Topics regex | topics.regex | Java regular expression of topics to replicate. For example: specify .* to replicate all available topics in the cluster. Applicable only when Use regular expressions is selected. | | Snowflake URL name | snowflake.url.name | The Snowflake URL to be used for the connection. | | Snowflake database name | snowflake.database.name | The Snowflake database name to be used for the exported data. | | Snowflake user name | snowflake.user.name | The name of the user who created the key pair. | | Snowflake private key | snowflake.private.key | The private key name for the Snowflake user. | | Snowflake private key passphrase | snowflake.private.key.passphrase | (Optional) If created and encrypted, the passphrase of the private key. | | Snowflake role name | snowflake.role.name | The name of the role created in Prerequisites. | | Kafka message value format | value.converter | The format of the value in the Redpanda topic. The default is SNOWFLAKE_JSON. | | Max Tasks | tasks.max | Maximum number of tasks to use for this connector. The default is 1. Each task replicates exclusive set of partitions assigned to it. | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ### [](#advanced-snowflake-sink-connector-configuration)Advanced Snowflake Sink connector configuration In most instances, the preceding basic configuration properties are sufficient. If you require additional property settings, then specify any of the following _optional_ advanced connector configuration properties by selecting **Show advanced options** on the **Create Connector** page: | Property name | Property key | Description | | --- | --- | --- | | Snowflake schema name | snowflake.schema.name | The Snowflake database schema name. The default is PUBLIC. | | Snowflake ingestion method | snowflake.ingestion.method | The default, SNOWPIPE, allows for structured data, while SNOWPIPE_STREAMING is lower latency option. | | Snowflake topic2table map | snowflake.topic2table.map | (Optional) Map of topics to tables. Format is comma-separated tuples. For example, :,:. | | Buffer count records | buffer.count.records | Number of records buffered in memory per partition before triggering Snowflake ingestion. Default is 10000. | | Buffer flush time | buffer.flush.time | The time in seconds to flush cached data. Default is 120. | | Buffer size bytes | buffer.size.bytes | Cumulative size of records buffered in memory per partition before triggering Snowflake ingestion. Default is 5000000. | | Error tolerance | errors.tolerance | Error tolerance response during connector operation. Default value is none and signals that any error will result in an immediate connector task failure. Value of all changes the behavior to skip over problematic records. | | Dead letter queue topic name | errors.deadletterqueue.topic.name | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, its transformations, or converters. The topic name is blank by default, which means that no messages are recorded in the DLQ. | | Dead letter queue topic replication factor | errors.deadletterqueue.topic .replication.factor | Replication factor used to create the dead letter queue topic when it doesn’t already exist. | | Enable error context headers | errors.deadletterqueue.context .headers.enable | When true, adds a header containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, start with __connect.errors. | ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - `JSON` formatted records should use `SNOWFLAKE_JSON` (`com.snowflake.kafka.connector.records.SnowflakeJsonConverter`). - `AVRO` formatted records that use Kafka’s Schema Registry Service should use `SNOWFLAKE_AVRO` (`com.snowflake.kafka.connector.records.SnowflakeAvroConverter`). - `AVRO` formatted records that contain the schema (and therefore do not need Kafka’s Schema Registry Service) should use `SNOWFLAKE_AVRO_WITHOUT_SCHEMA_REGISTRY` (`com.snowflake.kafka.connector.records.SnowflakeAvroConverterWithoutSchemaRegistry`). - Plain text formatted records should use `STRING` (`org.apache.kafka.connect.storage.StringConverter`). ## [](#test-the-connection)Test the connection After the connector is created, verify in your Snowflake worksheet that your table is populated: SELECT \* FROM TEST.PUBLIC.TABLE\_NAME; It may take a couple of minutes for the records to be visible in Snowflake. ## [](#troubleshoot)Troubleshoot After submitting the connector for creation in Redpanda Console, the Snowflake Sink connector attempts to authenticate to the Snowflake database to validate the configuration. This validation must be successful before the connector is created. It can take up 10 seconds or more to respond. If the connector fails, check the error message or select **Show Logs** to view error details. Additional errors and corrective actions follow. | Message | Action | | --- | --- | | snowflake.url.name is not a valid snowflake url | Check to make sure Snowflake URL name contains a valid Snowflake URL. | | snowflake.user.name: Cannot connect to Snowflake | Check to make sure Snowflake user name contains a valid Snowflake user. | | snowflake.private.key must be a valid PEM RSA private key / java.lang.IllegalArgumentException: Last encoded character (before the padding, if any) is a valid base 64 alphabet but not a possible value. Expect the discarded bits to be zero. | Snowflake private key is invalid. Provide a valid key. | | snowflake.database.name+ database does not exist | Specify a valid database name in snowflake.database.name. | | Object does not exist, or operation cannot be performed | Snowflake error that can have several causes: an invalid role is being used, there is no existing Snowflake table, or an incorrect schema name is specified. Verify that the connector configuration and Snowflake settings are valid. | | Config:value.converter has provided value:com.snowflake.kafka.connector.records.SnowflakeJsonConverter. If ingestionMethod is:snowpipe_streaming, Snowflake Custom Converters are not allowed. | Use STRING for the Kafka message value format. | ## [](#suggested-reading)Suggested reading - For more about limitations, see [Kafka Connector Limitations](https://docs.snowflake.com/en/user-guide/kafka-connector-overview#kafka-connector-limitations) - For testing the connection, see [Using Worksheets for Queries / DML / DDL](https://docs.snowflake.com/en/user-guide/ui-worksheet) - For details about all Snowflake Sink connector properties, see [Kafka Configuration Properties](https://docs.snowflake.com/en/user-guide/kafka-connector-install#required-properties) --- # Page 543: Create a SQL Server (Debezium) Source Connector **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-sqlserver-connector.md --- # Create a SQL Server (Debezium) Source Connector > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a SQL Server (Debezium) Source Connector latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/create-sqlserver-connector page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/create-sqlserver-connector.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/create-sqlserver-connector.adoc description: Use the Redpanda Cloud UI to create a SQL Server (Debezium) Source Connector. page-git-created-date: "2024-10-03" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can use an SQL Server (Debezium) Source connector to import updates to Redpanda from SQL Server. ## [](#prerequisites)Prerequisites Before you can create an SQL Server (Debezium) Source connector in the Redpanda Cloud, you must: - Make the SQL Server (Debezium) database accessible from the connector instance. - Create a SQL Server (Debezium) user with the necessary permissions. ## [](#limitations)Limitations The SQL Server (Debezium) Source connector has the following limitations: - Only `JSON`, `CloudEvents` or `AVRO` formats can be used for a Kafka message key and value format. - SQL Server (Debezium) connector can work with only a single task at a time per database name. ## [](#create-an-sql-server-debezium-source-connector)Create an SQL Server (Debezium) Source connector To create the SQL Server (Debezium) Source connector: 1. In Redpanda Cloud, click **Connectors** in the navigation menu, and then click **Create Connector**. 2. Select **Import from SQL Server (Debezium)**. 3. On the **Create Connector** page, specify the following required connector configuration options: | Property name | Property key | Description | | --- | --- | --- | | Topic prefix | topic.prefix | A topic prefix that identifies and provides a namespace for the particular database server/cluster that is capturing changes. The topic prefix should be unique across all other connectors because it is used as a prefix for all Kafka topic names that receive events emitted by this connector. Only alphanumeric characters, hyphens, dots, and underscores are accepted. | | Hostname | database.hostname | A resolvable hostname or IP address of the SQL Server database server. | | Port | database.port | Integer port number of the SQL Server database server. | | User | database.user | Name of the SQL Server user to be used when connecting to the SQL Server database. | | Password | database.password | The password of the SQL Server database user who will be connecting to the SQL Server database. | | Database instance | database.instance | Specifies the instance name of the SQL Server named instance. If both database.port and database.instance are specified, database.instance is ignored. | | Databases | database.names | The comma-separated list of the SQL Server database names from which to stream the changes. | | Kafka message key format | key.converter | Format of the key in the Redpanda topic. | | Message key JSON contains schema | key.converter.schemas.enable | Enable to specify that the message key contains schema in the schema field. | | Kafka message value format | value.converter | Format of the value in the Redpanda topic. | | Message value JSON contains schema | value.converter.schemas.enable | Enable to specify that the message value contains schema in the schema field. | | Max tasks | tasks.max | The maximum number of tasks that the connector can use to capture data from the database instance. If the Databases list contains more than one element, you can increase the value of this property to a number less than or equal to the number of elements in the list. Default: 1 | | Connector name | name | Globally-unique name to use for this connector. | 4. Click **Next**. Review the connector properties specified, then click **Create**. ## [](#map-data)Map data Use the appropriate key or value converter (input data format) for your data as follows: - Use the `Include Schemas`, `Include Tables`, and `Include Columns` properties to define lists of columns, tables, and schemas to read from. Alternatively, use `Exclude Schemas`, `Exclude Tables`, and `Exclude Columns` to define lists of columns, tables, and schemas to exclude from sources list. - Use only `JSON` (`org.apache.kafka.connect.json.JsonConverter`), `AVRO` (`io.confluent.connect.avro.AvroConverter`), and `CloudEvents` (`io.debezium.converters.CloudEventsConverter`) formats for the Kafka message key and value format. ## [](#test-the-connection)Test the connection After the connector is created: 1. Open Redpanda Console, click the **Topics** tab, and select a topic. Check to confirm that it contains data migrated from SQL Server. Alternatively, run `rpk consume` to check the topic. 2. Click the **Connectors** tab to confirm that no issues have been reported for the connector. ## [](#suggested-reading)Suggested reading - [Debezium connector for SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html) --- # Page 544: Disable Kafka Connect **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc.md --- # Disable Kafka Connect > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Disable Kafka Connect latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/disable-kc page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/disable-kc.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/disable-kc.adoc description: Learn how to disable Kafka Connect using the Cloud API. page-git-created-date: "2025-08-07" page-git-modified-date: "2026-05-26" --- Kafka Connect is disabled by default on new clusters. If you previously enabled Kafka Connect on a cluster and want to disable it, you can use the [Cloud API](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview). > 📝 **NOTE** > > Redpanda Support does not manage or monitor Kafka Connect, but Support can enable the feature for your account. ## [](#verify-kafka-connect-is-enabled)Verify Kafka Connect is enabled If Kafka Connect is enabled on your cluster, you will see it configured on the **Connect** page in the Redpanda Cloud UI. You can also verify with the Cloud API: ```bash curl -sX GET "https://api.redpanda.com/v1/clusters/{cluster.id}" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -H 'accept: application/json' | jq -r '.cluster.kafka_connect' ``` Replace `{cluster.id}` with your actual cluster ID. You can find the cluster ID in the Redpanda Cloud UI. Look in the **Details** section of the cluster overview. If Kafka Connect is enabled, the response will show: ```bash "enabled": true ``` ## [](#prerequisites)Prerequisites - You have the cluster ID of a cluster that has Kafka Connect enabled. - You have a valid bearer token for the Cloud API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). > ❗ **IMPORTANT** > > Make sure to stop any active connectors gracefully before disabling Kafka Connect to avoid data loss or incomplete processing. ## [](#disable-kafka-connect)Disable Kafka Connect After you are authenticated to the Cloud API, make a [`PATCH /v1/clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request, replacing `{cluster.id}` with your actual cluster ID. ```bash curl -X PATCH "https://api.redpanda.com/v1/clusters/{cluster.id}" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{"kafka_connect":{"enabled":false}}' ``` The `PATCH` request returns the ID of a long-running operation. You can check the status of the operation by polling the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint: ```bash curl -X GET "https://api.redpanda.com/v1/operations/\{operation.id\}" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -H "Content-Type: application/json" ``` When the operation is complete, the status will show `"state": "STATE_COMPLETED"`. You can verify that Kafka Connect has been disabled by running the verification command from the previous section. The response should show: ```bash "enabled": false ``` --- # Page 545: Monitor Kafka Connect **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/monitor-connectors.md --- # Monitor Kafka Connect > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Monitor Kafka Connect latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/monitor-connectors page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/monitor-connectors.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/monitor-connectors.adoc description: Use metrics to monitor the health of Kafka Connect. page-git-created-date: "2024-06-06" page-git-modified-date: "2026-05-26" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. You can monitor the health of Kafka Connect with metrics that Redpanda exports through a Prometheus HTTPS endpoint. You can use Grafana to visualize the metrics and set up alerts. The most important metrics to be monitored by alerts are: - connector failed tasks - connector lag / connector lag rate ## [](#view-connector-logs)View connector logs Connector logs are written to the system topic `__redpanda.connectors_logs`. You can view logs in Redpanda Cloud on the Topics page for your cluster, or you can download logs with `rpk`. For example: ```bash # Last 100 messages (most recent) rpk topic consume __redpanda.connectors_logs -o -100 -n 100 # Last 10 minutes rpk topic consume __redpanda.connectors_logs -o @-10m:end # Stream new logs only (like tail -f) rpk topic consume __redpanda.connectors_logs -o end # Filter by connector name rpk topic consume __redpanda.connectors_logs -o @-10m:end -O json \ | jq -r 'select(.message | test(""; "i"))' ``` > 📝 **NOTE** > > Access to system topics may be restricted by organization/project roles. Log retention follows cluster/system-topic policies and messages may expire. ## [](#limitations)Limitations The connectors dashboard renders metrics that are exported by managed connectors. However, when a connector does not create a task (for example, an empty topic list), the dashboard will not show metrics for that connector. --- # Page 546: Sizing Connectors **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/sizing-connectors.md --- # Sizing Connectors > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Sizing Connectors latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/sizing-connectors page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/sizing-connectors.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/sizing-connectors.adoc description: How to choose number of tasks to set for a connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. ## [](#connector-tasks)Connector tasks When you set up a connector, its main responsibility is to validate the configuration and spawn _connector tasks_, which perform the work. Setting up multiple tasks for a connector allows for parallelization of the work, resulting in higher throughputs. Before setting up connector tasks, consider the following: - For source connectors, the ability to add tasks to achieve higher throughput depends on the connector implementation and configuration. For many connectors, only a single connector task is allowed (for example, Debezium allows a single task only). When Redpanda Cloud does not offer an option to set the number of tasks, the source connector runs only one task. - For sink connectors, parallelism is achieved by evenly distributing configured topic partitions for the connector amongst connector tasks. The number of partitions must be equal to or greater than the number of tasks. ## [](#single-task-throughput)Single task throughput Connector throughput depends on many factors, including converters used, compression, message size, and the performance of external systems. As a rule of thumb, expect a single connector task to provide 1-2 MB/s of throughput. ## [](#specify-number-of-connector-tasks-for-a-sink-connector)Specify number of connector tasks for a sink connector It can be a challenge to determine the number of connector tasks to use for a given workload, so you must experiment to find the right number. Start with low number of connector tasks and wait a couple of minutes to view performance. Keep increasing the number of tasks until satisfactory throughput is achieved. Keep in mind that the underlying infrastructure must scale to provide room for additional connector tasks. Waiting roughly 10 minutes after each change should provide sufficient time for the system to scale up. --- # Page 547: Single Message Transforms **URL**: https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/transforms.md --- # Single Message Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Single Message Transforms latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: managed-connectors/transforms page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: managed-connectors/transforms.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/managed-connectors/transforms.adoc description: Single Message Transforms (SMTs) let you modify the data and its characteristics as it passes through a connector. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-05" --- > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. Single Message Transforms (SMTs) help you modify data and its characteristics as it passes through a connector, without needing additional stream processors. Prior to using an SMT with production data, test the configuration on a smaller subset of data to verify the behavior of the SMT. ## [](#cast)Cast Cast SMT lets you change the data type of fields in a Redpanda message, updating the schema if one is present. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.Cast$Key`) or value (`org.apache.kafka.connect.transforms.Cast$Value`). ### [](#configuration)Configuration | Property key | Description | | --- | --- | | spec | Comma-separated list of field names and the type to which they should be cast; for example: my-field1:int32,my-field2:string. Allowed types are: `int8, int16, int32, int64, float32, float64, boolean, and string. | ### [](#example)Example "transforms": "Cast", "transforms.Cast.type": "org.apache.kafka.connect.transforms.Cast$Value", "transforms.Cast.spec": "price:float64" Before: {"price": 1234, "product\_id": "9987"} After: {"price": 1234.0,"product\_id": "9987"} ## [](#dropheaders)DropHeaders DropHeaders SMT removes one or more headers from each record. ### [](#configuration-2)Configuration | Property key | Description | | --- | --- | | headers | Comma-separated list of header names to drop. | ### [](#example-2)Example Sample configuration: "transforms": "DropHeader", "transforms.DropHeader.type": "org.apache.kafka.connect.transforms.DropHeaders", "transforms.DropHeader.headers": "source-id,conv-id" ## [](#eventrouter-debezium)EventRouter (Debezium) The outbox pattern is a way to safely and reliably exchange data between multiple (micro) services. An outbox pattern implementation avoids inconsistencies between a service’s internal state (as typically persisted in its database) and state in events consumed by services that need the same data. To implement the outbox pattern in a Debezium application, configure a Debezium connector to: - Capture changes in an outbox table - Apply the Debezium outbox EventRouter Single Message Transformation > 📝 **NOTE** > > EventRouter SMT is available for managed Debezium connectors only. ### [](#configuration-3)Configuration | Property key | Description | | --- | --- | | route.by.field | Specifies the name of a column in the outbox table. The default behavior is that the value in this column becomes a part of the name of the topic to which the connector emits the outbox messages. | | route.topic.replacement | Specifies the name of the topic to which the connector emits outbox messages. The default topic name is outbox.event. followed by the aggregatetype column value in the outbox table record. | | table.expand.json.payload | Specifies whether the JSON expansion of a String payload should be done. If no content is found, or if there’s a parsing error, the content is kept "as is". | | fields.additional.placement | Specifies one or more outbox table columns to add to outbox message headers or envelopes. Specify a comma-separated list of pairs. In each pair, specify the name of a column and whether you want the value to be in the header or the envelope. | | table.field.event.key | Specifies the outbox table column that contains the event key. When this column contains a value, the SMT uses that value as the key in the emitted outbox message. This is important for maintaining the correct order in Kafka partitions. | ### [](#example-3)Example Sample JSON configuration: "transforms": "outbox", "transforms.outbox.route.by.field": "type", "transforms.outbox.route.topic.replacement": "my-topic.${routedByValue}", "transforms.outbox.table.expand.json.payload": "true", "transforms.outbox.table.field.event.key": "aggregate\_id", "transforms.outbox.table.fields.additional.placement": "before:envelope", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter" ### [](#suggested-reading)Suggested reading - [Debezium Outbox Event Router SMT](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) ## [](#extractfield)ExtractField ExtractField SMT pulls the specified field from a Struct when a schema is present, or a Map for schemaless data. Any null values are passed through unmodified. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.ExtractField$Key`) or value (`org.apache.kafka.connect.transforms.ExtractField$Value`). ### [](#configuration-4)Configuration | Property key | Description | | --- | --- | | field | Field name to extract. | ### [](#example-4)Example Sample configuration: "transforms": "ExtractField", "transforms.ExtractField.type": "org.apache.kafka.connect.transforms.ExtractField$Value", "transforms.ExtractField.field": "product\_id" Before: ```json {"product_id":9987,"price":1234} ``` After: ```json {"value":9987} ``` ## [](#filter)Filter Filter SMT drops all records, filtering them from subsequent transformations in the chain. This is intended to be used conditionally to filter out records matching (or not matching) a particular predicate. ### [](#configuration-5)Configuration | Property key | Description | | --- | --- | | predicate | Name of predicate filtering records. | ### [](#example-5)Example Sample configuration: "transforms": "Filter", "transforms.Filter.type": "org.apache.kafka.connect.transforms.Filter", "transforms.Filter.predicate": "IsMyTopic", "predicates": "IsMyTopic", "predicates.IsMyTopic.type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches", "predicates.IsMyTopic.pattern": "my-topic" ### [](#predicates)Predicates Managed connectors support the following predicates: #### [](#topicnamematches)TopicNameMatches `org.apache.kafka.connect.transforms.predicates.TopicNameMatches` - A predicate that is true for records with a topic name that matches the configured regular expression. | Property key | Description | | --- | --- | | pattern | A Java regular expression for matching against the name of a record’s topic. | #### [](#hasheaderkey)HasHeaderKey `org.apache.kafka.connect.transforms.predicates.HasHeaderKey` - A predicate that is true for records with at least one header with the configured name. | Property key | Description | | --- | --- | | name | The header name. | #### [](#recordistombstone)RecordIsTombstone `org.apache.kafka.connect.transforms.predicates.RecordIsTombstone` - A predicate that is true for records that are tombstones (that is, they have null values). ## [](#flatten)Flatten Flatten SMT flattens a nested data structure, generating names for each field by concatenating the field names at each level with a configurable delimiter character. Applies to Struct when a schema is present, or a Map for schemaless data. Array fields and their contents are not modified. The default delimiter is `.`. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.Flatten$Key`) or value (`org.apache.kafka.connect.transforms.Flatten$Value`). ### [](#configuration-6)Configuration | Property key | Description | | --- | --- | | delimiter | Delimiter to insert between field names from the input record when generating field names for the output record. | ### [](#example-6)Example "transforms": "flatten", "transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value", "transforms.flatten.delimiter": "." Before: ```json { "user": { "id": 10, "name": { "first": "Red", "last": "Panda" } } } ``` After: ```json { "user.id": 10, "user.name.first": "Red", "user.name.last": "Panda" } ``` ## [](#headerfrom)HeaderFrom HeaderFrom SMT moves or copies fields in the key or value of a record into that record’s headers. Corresponding elements of `fields` and `headers` together identify a field and the header it should be moved or copied to. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.HeaderFrom$Key`) or value (`org.apache.kafka.connect.transforms.HeaderFrom$Value`). ### [](#configuration-7)Configuration | Property key | Description | | --- | --- | | fields | Comma-separated list of field names in the record whose values are to be copied or moved to headers. | | headers | Comma-separated list of header names, in the same order as the field names listed in the fields configuration property. | | operation | Either move if the fields are to be moved to the headers (removed from the key/value), or copy if the fields are to be copied to the headers (retained in the key/value). | ### [](#example-7)Example "transforms": "HeaderFrom", "transforms.HeaderFrom.type": "org.apache.kafka.connect.transforms.HeaderFrom$Value", "transforms.HeaderFrom.fields": "id,last\_login\_ts", "transforms.HeaderFrom.headers": "user\_id,timestamp", "transforms.HeaderFrom.operation": "move" Before: - Record value: { "id": 11, "name": "Harry Wilson", "last\_login\_ts": 1715242380 } - Record header: { "conv\_id": "uier923" } After: - Record value: { "name": "Harry Wilson" } - Record header: { "conv\_id": "uier923", "user\_id": 11, "timestamp": 1715242380 } ## [](#hoistfield)HoistField HoistField SMT wraps data using the specified field name in a Struct when schema present, or a Map in the case of schemaless data. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.HoistField$Key`) or value (`org.apache.kafka.connect.transforms.HoistField$Value`). ### [](#configuration-8)Configuration | Property key | Description | | --- | --- | | field | Field name for the single field that will be created in the resulting Struct or Map. | ### [](#example-8)Example "transforms": "HoistField", "transforms.HoistField.type": "org.apache.kafka.connect.transforms.HoistField$Value", "transforms.HoistField.field": "name" Message: ```none Red Panda ``` After: ```none {"name":"Red"} {"name":"Panda"} ``` ## [](#insertfield)InsertField InsertField SMT inserts field(s) using attributes from the record metadata or a configured static value. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.InsertField$Key`) or value (`org.apache.kafka.connect.transforms.InsertField$Value`). ### [](#configuration-9)Configuration | Property key | Description | | --- | --- | | offset.field | Field name for Redpanda offset. | | partition.field | Field name for Redpanda partition. | | static.field | Field name for static data field. | | static.value | The static field value. | | timestamp.field | Field name for record timestamp. | | topic.field | Field name for Redpanda topic. | ### [](#example-9)Example Sample configuration: "transforms": "InsertField", "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.InsertField.static.field": "cluster\_id", "transforms.InsertField.static.value": "19423" Before: ```json {"product_id":9987,"price":1234} ``` After: ```json {"price":1234,"cluster_id":"19423","product_id":9987} ``` ## [](#maskfield)MaskField MaskField SMT replaces the contents of fields in a record. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.MaskField$Key`) or value (`org.apache.kafka.connect.transforms.MaskField$Value`). ### [](#configuration-10)Configuration | Property key | Description | | --- | --- | | fields | Comma-separated list of fields to mask. | | replacement | Custom value replacement used to mask field values. | ### [](#example-10)Example "transforms": "MaskField", "transforms.MaskField.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.MaskField.fields": "metadata", "transforms.MaskField.replacement": "\*\*\*" Before: {"product\_id":9987,"price":1234,"metadata":"test"} After: {"metadata":"\*\*\*","price":1234,"product\_id":9987} ## [](#regexrouter)RegexRouter RegexRouter SMT updates the record topic using the configured regular expression and replacement string. Under the hood, the regex is compiled to a `java.util.regex.Pattern`. If the pattern matches the input topic, `java.util.regex.Matcher#replaceFirst()` is used with the replacement string to obtain the new topic. ### [](#configuration-11)Configuration | Property key | Description | | --- | --- | | regex | Regular expression to use for matching. | | replacement | Replacement string. | ### [](#example-11)Example This configuration snippet shows how to add the prefix `prefix_` to the beginning of a topic. "transforms": "AppendPrefix", "transforms.AppendPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.AppendPrefix.regex": ".\*", "transforms.AppendPrefix.replacement": "prefix\_$0" Before: `topic-name` After: `prefix_topic-name` ## [](#replacefield)ReplaceField ReplaceField SMT filters or renames fields in a Redpanda record. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.ReplaceField$Key`) or value (`org.apache.kafka.connect.transforms.ReplaceField$Value`). ### [](#configuration-12)Configuration | Property key | Description | | --- | --- | | exclude | Fields to exclude. This takes precedence over the fields to include. | | include | Fields to include. If specified, only these fields are used. | | renames | List of comma-separated pairs. For example: foo:bar,abc:xyz | ### [](#example-12)Example Sample configuration: "transforms": "ReplaceField", "transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.ReplaceField.renames": "product\_id:item\_number" Before: ```json {"product_id":9987,"price":1234} ``` After: ```json {"item_number":9987,"price":1234} ``` ## [](#replacetimestamp-redpanda)ReplaceTimestamp (Redpanda) ReplaceTimestamp (Redpanda) SMT is designed to support using a record key/value field as a record timestamp, which then can be used to partition data with an S3 connector. Use the concrete transformation type designed for the record key (`com.redpanda.connectors.transforms.ReplaceTimestamp$Key`) or value (`com.redpanda.connectors.transforms.ReplaceTimestamp$Value`). > 📝 **NOTE** > > ReplaceTimestamp is available for Sink connector only. ### [](#configuration-13)Configuration | Property key | Description | | --- | --- | | field | Specifies the name of a field to be used as a source of timestamp. | ### [](#example-13)Example To use `my-timestamp` field as a source of the timestamp for the record, update a connector config with: "transforms": "ReplaceTimestamp", "transforms.ReplaceTimestamp.type": "com.redpanda.connectors.transforms.ReplaceTimestamp$Value", "transforms.ReplaceTimestamp.field": "my-timestamp" for messages in a format: { "name": "my-name", ... "my-timestamp": 1707928150868, ... } The SMT needs structured data to be able to extract the field from it, which means either a Map in the case of schemaless data, or a Struct when a schema is present. The timestamp value should be of a numeric type (epoch millis), or a Java Date object (which is the case when using `"connect.name":"org.apache.kafka.connect.data.Timestamp"` in schema). ## [](#schemaregistryreplicator-redpanda)SchemaRegistryReplicator (Redpanda) SchemaRegistryReplicator (Redpanda) SMT is a transform to replicate schemas. > 📝 **NOTE** > > SchemaRegistryReplicator SMT is designed to be used with the MirrorMaker2 connector only. To use it, remove the `_schema` topic from the topic exclude list. ### [](#example-14)Example Sample configuration: "transforms": "schema-replicator", "transforms.schema-replicator.type": "com.redpanda.connectors.transforms.SchemaRegistryReplicator" ## [](#setschemametadata)SetSchemaMetadata SetSchemaMetadata SMT sets the schema name, version, or both on the record’s key (`org.apache.kafka.connect.transforms.SetSchemaMetadata$Key`) or value (`org.apache.kafka.connect.transforms.SetSchemaMetadata$Value`) schema. ### [](#configuration-14)Configuration | Property key | Description | | --- | --- | | schema.name | Schema name to set. | | schema.version | Schema version to set. | ### [](#example-15)Example Sample configuration: "transforms": "SetSchemaMetadata", "transforms.SetSchemaMetadata.type": "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value", "transforms.SetSchemaMetadata.schema.name": "transaction-value" "transforms.SetSchemaMetadata.schema.version": "3" ## [](#timestampconverter)TimestampConverter TimestampConverter SMT converts timestamps between different formats, such as Unix epoch, strings, and Connect Date/Timestamp types. It applies to individual fields or to the entire value. Use the concrete transformation type designed for the record key (`org.apache.kafka.connect.transforms.TimestampConverter$Key`) or value (`org.apache.kafka.connect.transforms.TimestampConverter$Value`). ### [](#configuration-15)Configuration | Property key | Description | | --- | --- | | field | The field containing the timestamp, or empty if the entire value is a timestamp. Default: "". | | target.type | The desired timestamp representation: string, unix, Date, Time, or Timestamp. | | format | A SimpleDateFormat-compatible format for the timestamp. Used to generate the output when target.type=string or used to parse the input if the input is a string. Default: "". | | unix.precision | The desired Unix precision for the timestamp: seconds, milliseconds, microseconds, or nanoseconds. Used to generate the output when type=unix or used to parse the input if the input is a Long. Note: This SMT causes precision loss during conversions from, and to, values with sub-millisecond components. Default: milliseconds. | ### [](#example-16)Example Sample configuration: "transforms": "TimestampConverter", "transforms.TimestampConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value", "transforms.TimestampConverter.field": "last\_login\_date", "transforms.TimestampConverter.format": "yyyy-MM-dd", "transforms.TimestampConverter.target.type": "string" Before: `1702041416` After: `2023-12-08` ## [](#timestamprouter)TimestampRouter TimestampRouter SMT updates the record’s topic field as a function of the original topic value and the record timestamp. This is mainly useful for sink connectors, because the topic field is often used to determine the equivalent entity name in the destination system (for example, a database table or search index name). > 📝 **NOTE** > > TimestampRouter SMT should be used with sink connectors only. ### [](#configuration-16)Configuration | Property key | Description | | --- | --- | | topic.format | Format string that can contain ${topic} and ${timestamp} as placeholders for the topic and timestamp, respectively. | | timestamp.format | Format string for the timestamp that is compatible with java.text.SimpleDateFormat. | ### [](#example-17)Example Sample configuration: "transforms": "router", "transforms.router.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.router.topic.format": "${topic}\_${timestamp}", "transforms.router.timestamp.format": "YYYY-MM-dd" ## [](#valuetokey)ValueToKey ValueToKey SMT replaces the record key with a new key formed from a subset of fields in the record value. ### [](#configuration-17)Configuration | Property key | Description | | --- | --- | | fields | Comma-separated list of field names on the record value to extract as the record key. | ### [](#example-18)Example Sample configuration: "transforms": "valueToKey", "transforms.valueToKey.type": "org.apache.kafka.connect.transforms.ValueToKey", "transforms.valueToKey.fields": "txn-id" ## [](#error-handling)Error handling By default, `Error tolerance` is set to `NONE`, so SMTs fail for any exception (notably, data parsing or data processing errors). To avoid the connector crashing for data issues, set `Error tolerance` to `ALL`, and specify `Dead Letter Queue Topic Name` as a place where failed messages are redirected. --- # Page 548: Produce Data **URL**: https://docs.redpanda.com/cloud-data-platform/develop/produce-data.md --- # Produce Data > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Produce Data latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: produce-data/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: produce-data/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/produce-data/index.adoc description: Learn how to configure producers and idempotent producers. page-git-created-date: "2024-07-25" page-git-modified-date: "2024-08-01" --- - [Configure Producers](configure-producers/) Learn about configuration options for producers, including write caching and acknowledgment settings. - [Idempotent Producers](idempotent-producers/) Idempotent producers assign a unique ID to every write request, guaranteeing that each message is recorded only once in the order in which it was sent. - [Configure Leader Pinning](leader-pinning/) Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. --- # Page 549: Configure Producers **URL**: https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers.md --- # Configure Producers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Producers latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: produce-data/configure-producers page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: produce-data/configure-producers.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/produce-data/configure-producers.adoc description: Learn about configuration options for producers, including write caching and acknowledgment settings. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Producers are client applications that write data to Redpanda in the form of events. Producers communicate with Redpanda through the Kafka API. When a producer publishes a message to a Redpanda cluster, it sends it to a specific partition. Every event consists of a key and value. When selecting which partition to produce to, if the key is blank, then the producer publishes in a round-robin fashion between the topic’s partitions. If a key is provided, then the partition hashes the key using the murmur2 algorithm and modulates across the number of partitions. ## [](#producer-acknowledgment-settings)Producer acknowledgment settings The `acks` property sets the number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. Redpanda guarantees data safety with fsync, which means flushing to disk. - With `acks=all`, every write is fsynced by default. - With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. ### [](#acks0)`acks=0` The producer doesn’t wait for acknowledgments from the leader and doesn’t retry sending messages. This increases throughput and lowers latency of the system at the expense of durability and data loss. This option allows a producer to immediately consider a message acknowledged when it is sent to the Redpanda broker. This means that a producer does not have to wait for any response from the Redpanda broker. This is the least safe option, because a leader-broker crash can cause data loss if the data has not yet replicated to the other brokers in the replica set. However, this setting is useful when you want to optimize for the highest throughput and are willing to risk some data loss. Because of the lack of guarantees, this setting is the most network bandwidth-efficient. This is helpful for use cases like IoT/sensor data collection, where updates are periodic or stateless and you can afford some degree of data loss, but you want to gather as much data as possible in a given time interval. ### [](#acks1)`acks=1` The producer waits for an acknowledgment from the leader, but it doesn’t wait for the leader to get acknowledgments from followers. This setting doesn’t prioritize throughput, latency, or durability. Instead, `acks=1` attempts to provide a balance between all of them. Replication is not guaranteed with this setting because it happens in the background, after the leader broker sends an acknowledgment to the producer. This setting could result in data loss if the leader broker crashes before any followers manage to replicate the message or if a majority of replicas go down at the same time before fsyncing the message to the disk. ### [](#acksall)`acks=all` The producer receives an acknowledgment after the majority of (implicitly, all) replicas acknowledge the message. Redpanda guarantees data safety by fsyncing every message to disk before acknowledgement back to clients. This increases durability at the expense of lower throughput and increased latency. Sometimes referred to as `acks = -1`, this option instructs the broker that replication is considered complete when the message has been replicated (and fsynced) to the majority of the brokers responsible for the partition in the cluster. As soon as the fsync call is complete, the message is considered acknowledged and is made visible to readers. > 📝 **NOTE** > > This property has an important distinction compared to Kafka’s behavior. In Kafka, a message is considered acknowledged without the requirement that it has been fsynced. Messages that have not been fsynced to disk may be lost in the event of a broker crash. So when using `acks=all`, the Redpanda default configuration is more resilient than Kafka’s. You can also consider using write caching, which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. ### [](#retries)`retries` This property controls the number of times a message is re-sent to the broker if the broker fails to acknowledge it. This is essentially the same as if the client application resends the erroneous message after receiving an error response. The default value of `retries` in most client libraries is 0. This means that if the send fails, the message is not re-sent at all. If you increase this to a higher value, check the `max.in.flight.requests.per.connection` value as well, because leaving that property at its default value can potentially cause ordering issues in the target topic where the messages arrive. This occurs if two batches are sent to a single partition and the first fails and is retired, but the second succeeds so the records in the second batch may appear first. ### [](#max-in-flight-requests-per-connection)`max.in.flight.requests.per.connection` This property controls how many unacknowledged messages can be sent to the broker simultaneously at any given time. The default value is 5 in most client libraries. If you set this to 1, then the producer does not send any more messages until the previous one is either acknowledged or an error happens, which can prompt a retry. If you set this to a value higher than 1, then the producer sends more messages at the same time, which can help increase throughput but adds a risk of message reordering if retries are enabled. When you configure the producer to be [idempotent](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/idempotent-producers/), up to five requests can be guaranteed to be in flight with the order preserved. ### [](#enable-idempotence)`enable.idempotence` To enable idempotence, set `enable.idempotence` to `true` (the default) in your Redpanda configuration. When idempotence is enabled, the producer ensures that exactly one copy of every message is written to the broker. When set to `false`, the producer retries sending a message for any reason (such as transient errors like brokers not being available or not enough replicas exception), and it can lead to duplicates. In most client libraries `enable.idempotence` is set to true by default. Internally, this is implemented using a special identifier that is assigned to every producer (the producer ID or PID). This ID, along with a sequence number, is included in every message sent to the broker. The broker checks if the PID/sequence number combination is larger than the previous one and, if not, it discards the message. To guarantee true idempotent behavior, you must also set `acks=all` to ensure that all brokers record messages in order, even in the event of node failures. In this configuration, both the producer and the broker prefer safety and durability over throughput. Idempotence is only guaranteed within a session. A session starts after the producer is instantiated and a connection is established between the client and the Redpanda broker. When the connection is closed, the session ends. If your application code retries a request, the producer client assigns a new ID to that request, which may lead to duplicate messages. ## [](#message-batching)Message batching Batching is an efficient way to save on both network bandwidth and disk size, because messages can be compressed easier. When a producer prepares to send messages to a broker, it first fills up a buffer. When this buffer is full, the producer compresses (if instructed to do so) and sends out this batch of messages to the broker. The number of batches that can be sent in a single request to the broker is limited by the `max.request.size` property. The number of requests that can simultaneously be in this sending state is controlled by the `max.in.flight.requests.per.connection` value, which defaults to 5 in most client libraries. Tune the batching configuration with the following properties: ### [](#buffer-memory)`buffer.memory` This property controls the total amount of memory available to the producer for buffering. If messages are sent faster than they can be delivered to the broker, the producer application may run out of memory, which causes it to either block subsequent send calls or throw an exception. The `max.block.ms` property controls the amount of time the producer blocks before throwing an exception if it cannot immediately send messages to the broker. ### [](#batch-size)`batch.size` This property controls the maximum size of coupled messages that can be batched together in one request. The producer automatically puts messages being sent to the same partition into one batch. This configuration property is given in bytes, as opposed to the number of messages. When the producer is gathering messages to assign to a batch, at some point it hits this byte-size limit, which triggers it to send the batch to the broker. However, the producer does not necessarily wait (for as much time as set using `linger.ms`) until the batch is full. Sometimes, it can even send single-message batches. This means that setting the batch size too large is not necessarily undesirable, because it won’t cause throttling when sending messages; rather, it only causes increased memory usage. Conversely, setting the batch size too small can cause the producer to send batches of messages faster, which can cause network overhead, meaning a reduced throughput. The default value is usually 16384, but you can set this as low as 0, which turns off batching entirely. ### [](#linger-ms)`linger.ms` This property controls the maximum amount of time the producer waits before sending out a batch of messages, if it is not already full. This means you can somewhat force the producer to make sure that batches are filled as efficiently as possible. If you’re willing to tolerate some latency, setting this value to a number larger than the default of `0` causes the producer to send fewer, more efficient batches of messages. If you set the value to `0`, there is still a high chance messages arrive around the same time to be batched together. ## [](#common-producer-configurations)Common producer configurations ### [](#compression-type)`compression.type` This property controls how the producer should compress a batch of messages before sending it to the broker. The default is `none`, which means the batch of messages is not compressed at all. Compression occurs on full batches, so you can improve batching throughput by setting this property to use one of the available compression algorithms (along with increasing batch size). The available options are: `zstd`, `lz4`, `gzip`, and `snappy`. ### [](#serializers)Serializers Serializers are responsible for converting a message to a byte array. You can influence the speed/memory efficiency of your streaming setup by choosing one of the built-in serializers or writing a custom one. The performance consequences of using serializers is not typically significant. For example, if you opt for the JSON serializer, you have more data to transport with each message because every record contains its schema in a verbose format, which impacts your compression speeds and network throughput. Alternatively, going with AVRO or Protobuf allows you to only define the schema in one place, while also enabling features like schema evolution. ## [](#broker-timestamps)Broker timestamps Redpanda employs a unique strategy to help ensure the accuracy of retention operations. In this strategy, closed segments are only eligible for deletion when the age of all messages in the segment exceeds a configured threshold. However, when a producer sends a message to a topic, the timestamp set by the producer may not accurately reflect the time the message reaches the broker. To address this time skew, each time a producer sends a message to a topic, Redpanda records the broker’s system date and time in the `broker_timestamp` property of the message. This property helps maintain accurate retention policies, even when the message’s creation timestamp deviates from the broker’s time. > 📝 **NOTE** > > Clock synchronization should be monitored by the server owner, as Redpanda does not monitor clock synchronization. While Redpanda does not rely on clocks for correctness, if you are using `LogAppendTime` (server timestamp set by Redpanda), server clocks may affect the time your application sees. ## [](#producer-optimization-strategies)Producer optimization strategies You can optimize for speed (throughput and latency) or safety (durability and availability) by adjusting properties. Finding the optimal configuration depends on your use case. There are many configuration options within Redpanda. The configuration options mentioned here work best when combined with other broker and consumer configuration options. See also: - [Consumer Offsets](https://docs.redpanda.com/cloud-data-platform/develop/consume-data/consumer-offsets/) ### [](#optimize-for-speed)Optimize for speed To get data into Redpanda as quickly as possible, you can maximize latency and throughput in a variety of ways: - Experiment with [acks](#producer-acknowledgment-settings) settings. The quicker a producer receives a reply from the broker that the message has been committed, the sooner it can send the next message, which generally results in higher throughput. Hence, if you set `acks=1`, then the leader broker does not need to wait for replication to occur, and it can reply as soon as it finishes committing the message. This can result in less durability overall. - Enable [write caching](<#Write caching>), which acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. - Experiment with other component’s properties, like the topic partition size. - Explore how the producer batches messages. Increasing the value of `batch.size` and `linger.ms` can increase throughput by making the producer add more messages into one batch before sending it to the broker and waiting until the batches can properly fill up. This approach negatively impacts latency though. By contrast, if you set `linger.ms` to `0` and `batch.size` to `1`, you can achieve lower latency, but sacrifice throughput. ### [](#optimize-for-safety)Optimize for safety For applications where you must guarantee that there are no lost messages, duplicates, or service downtime, you can use higher durability `acks` settings. If you set `acks=all`, then the producer waits for a majority of replicas to acknowledge the message before it can send the next message, resulting in lower latency, because there is more communication required between brokers. This approach can guarantee higher durability because the message is replicated to all brokers. You can also increase durability by increasing the number of retries the broker can make in case messages are not delivered successfully. The trade-off is that duplicates may enter the system and potentially alter the ordering of messages. --- # Page 550: Idempotent Producers **URL**: https://docs.redpanda.com/cloud-data-platform/develop/produce-data/idempotent-producers.md --- # Idempotent Producers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Idempotent Producers latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: produce-data/idempotent-producers page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: produce-data/idempotent-producers.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/produce-data/idempotent-producers.adoc description: Idempotent producers assign a unique ID to every write request, guaranteeing that each message is recorded only once in the order in which it was sent. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- When a producer writes messages to a topic, each message should be recorded only once in the order in which it was sent. However, network issues such as a connection failure can result in a timeout, which prevents a write request from succeeding. In such cases, the client retries the write request until one of these events occurs: - The client receives an acknowledgment from the broker that the write was successful. - The retry limit is reached. - The message delivery timeout limit is reached. Since there is no way to tell if the initial write request succeeded before the disruption, a retry can result in a duplicate message. A retry can also cause subsequent messages to be written out of order. Idempotent producers prevent this problem by assigning a unique ID to every write request. The request ID consists of the producer ID and a sequence number. The sequence number identifies the order in which each write request was sent. If a retry results in a duplicate message, Redpanda detects and rejects the duplicate message and maintains the original order of the messages. If new write requests continue while a previous request is being retried, the new requests are stored in the client’s memory in the order in which they were sent. The client must also retry these requests once the previous request is successful. ## [](#enable-idempotence-for-producers)Enable idempotence for producers To make producers idempotent, the `enable.idempotence` property must be set to `true` in your producer configuration, as well as in the Redpanda cluster configuration, where it is set to `true` by default. Some Kafka clients have `enable.idempotence` set to `false` by default. In this case, set the property to `true` by following the instructions for your particular client. Idempotence is guaranteed within a session. A session starts once a producer is created and a connection is established between the client and the Kafka broker. > 📝 **NOTE** > > Idempotent producers retry unsuccessful write requests automatically. If you manually retry a write request, the client will assign a new ID to that request, which may lead to duplicate messages. --- # Page 551: Configure Leader Pinning **URL**: https://docs.redpanda.com/cloud-data-platform/develop/produce-data/leader-pinning.md --- # Configure Leader Pinning > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Leader Pinning latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: produce-data/leader-pinning page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: produce-data/leader-pinning.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/produce-data/leader-pinning.adoc description: Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. learning-objective-1: Configure preferred partition leader placement using rack labels learning-objective-2: Configure ordered rack preference for priority-based leader failover learning-objective-3: Identify conditions where Leader Pinning cannot place leaders in preferred racks page-git-created-date: "2024-12-04" page-git-modified-date: "2026-05-26" --- Produce requests that write data to Redpanda topics are routed through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic’s partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. If consumers are located in the same preferred region or AZ for Leader Pinning, and you have not set up [follower fetching](https://docs.redpanda.com/cloud-data-platform/develop/consume-data/follower-fetching/), Leader Pinning can also help reduce networking costs on consume requests. After reading this page, you will be able to: - Configure preferred partition leader placement using rack labels - Configure ordered rack preference for priority-based leader failover - Identify conditions where Leader Pinning cannot place leaders in preferred racks ## [](#set-leader-rack-preferences)Set leader rack preferences Configure Leader Pinning if you have Redpanda deployed in a multi-AZ or multi-region cluster and your ingress is concentrated in a particular AZ or region. Use the topic configuration property `redpanda.leaders.preference` to configure Leader Pinning for individual topics. The property accepts the following string values: - `none`: Disable Leader Pinning for the topic. - `racks:[,,…​]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more racks, and you can list the racks in any order. Spaces in the list are ignored, for example: `racks:rack1,rack2` and `racks: rack1, rack2` are equivalent. You cannot specify empty racks, for example: `racks: rack1,,rack2`. If you specify multiple racks, Redpanda tries to distribute the partition leader locations equally across brokers in these racks. - `ordered_racks:[,,…​]`: Supported in Redpanda v26.1 or later. Specify the preferred racks in priority order. Redpanda places leaders in the first listed rack when available, failing over to each subsequent rack when higher-priority racks are unavailable. If all listed racks are unavailable, leaders fall back to any other available brokers. Brokers with no rack assignment are treated as lowest priority. To find the rack identifiers of all brokers, run: ```bash rpk cluster info ``` Expected output ```bash CLUSTER ======= redpanda.be267958-279d-49cd-ae86-98fc7ed2de48 BROKERS ======= ID HOST PORT RACK 0* 54.70.51.189 9092 us-west-2a 1 35.93.178.18 9092 us-west-2b 2 35.91.121.126 9092 us-west-2c ``` To set the topic property: ```bash rpk topic alter-config --set redpanda.leaders.preference=ordered_racks:, ``` If there is more than one broker in the preferred AZ (or AZs), Leader Pinning distributes partition leaders uniformly across brokers in the AZ. ## [](#limitations)Limitations Leader Pinning controls which replica is elected as leader, and does not move replicas to different brokers. If all of a topic’s replicas are on brokers in non-preferred racks, no replica exists in the preferred racks to elect as leader, and Redpanda may elect a non-preferred leader indefinitely. For example, consider a cluster deployed across four racks (A, B, C, D) with Leader Pinning configured as `ordered_racks:A,B,C,D`. With a replication factor of 3, rack awareness can only place replicas in three of the four racks. If the highest-priority rack (A) does not receive a replica, no replica exists there to elect as leader, and Redpanda may elect a non-preferred leader indefinitely. To prevent this scenario, ensure the topic’s replication factor at least equals the total number of racks in the cluster, so every rack, including the highest-priority rack, receives a replica. ## [](#leader-pinning-failover-across-availability-zones)Leader Pinning failover across availability zones If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior with `racks` is as follows: - The topic with `A` as the preferred leader AZ will have its partition leaders uniformly distributed across B and C. - The topic with `A,B` as the preferred leader AZs will have its partition leaders in B. - The topic with `B` as the preferred leader AZ will have its partition leaders in B as well. ### [](#failover-with-ordered-rack-preference)Failover with ordered rack preference With `ordered_racks`, the failover order follows the configured priority list. Leaders move to the next available rack in the list when higher-priority racks become unavailable. For a topic configured with `ordered_racks:A,B,C`: - The topic with `A` as the first-priority rack will have its partition leaders in A. - If A becomes unavailable, leaders move to B. - If A and B become unavailable, leaders move to C. - If A, B, and C all become unavailable, leaders fall back to any available brokers. If a higher-priority rack recovers and the topic’s replication factor ensures that rack receives a replica, Redpanda automatically moves leaders back to the highest available preferred rack. ## [](#suggested-reading)Suggested reading - For latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major cost driver, also consider [Cloud Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/cloud-topics/) - [Follower Fetching](https://docs.redpanda.com/cloud-data-platform/develop/consume-data/follower-fetching/) --- # Page 552: Topics **URL**: https://docs.redpanda.com/cloud-data-platform/develop/topics.md --- # Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: topics/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: topics/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/topics/index.adoc description: Overview of standard topics in Redpanda Cloud. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" --- - [Topics Overview](create-topic/) Learn how to create a topic for a Redpanda Cloud cluster. - [Manage Topics](config-topics/) Learn how to create topics, update topic configurations, and delete topics or records. - [Manage Cloud Topics](cloud-topics/) Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. --- # Page 553: Manage Cloud Topics **URL**: https://docs.redpanda.com/cloud-data-platform/develop/topics/cloud-topics.md --- # Manage Cloud Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Cloud Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: topics/cloud-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: topics/cloud-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/topics/cloud-topics.adoc description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-05-29" --- Starting in v26.1, Redpanda provides [Cloud Topics](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#cloud-topic) to support multi-modal streaming workloads in the most cost-effective way possible: as a per-topic configuration running mixed latency workloads. While standard Redpanda [topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/config-topics/) that use local storage or Tiered Storage are ideal for latency-sensitive workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to both replicate data and serve it to consumers. This eliminates over 90% of the cost of replicating data over network links in multi-AZ clusters. The end-to-end latency experienced when using Cloud Topics can range from 500 ms to as high as a few seconds with different object stores. Lower latencies may be achievable in certain environments, but Cloud Topics is optimized for throughput rather than low latency or tightly constrained tail latency. This latency profile is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. With Cloud Topics, data from the client is not acknowledged until it is uploaded to object storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. You should generally expect end-to-end latencies of 1-2 seconds with public cloud stores. After reading this page, you will be able to: - Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics - Create a Cloud Topic using rpk on a cluster that has cloud storage enabled - Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs ## [](#prerequisites)Prerequisites - [Install rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) v26.1 or later. ## [](#limitations)Limitations - In Redpanda versions earlier than v26.2, shadow links do not support Cloud Topics. - Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local storage or Tiered Storage v1. Conversely, existing topics created as local or Tiered Storage v1 topics cannot be converted to Cloud Topics. Starting in Redpanda v26.2, Cloud Topics can be converted to and from Tiered Storage v2 topics. ## [](#create-cloud-topics)Create Cloud Topics Cloud Topics don’t require a separate cluster property to enable them. When cloud storage is enabled for your cluster, you can create Cloud Topics directly. To create a Cloud Topic, set the topic property `redpanda.storage.mode` to `cloud`: ```bash rpk topic create -c redpanda.storage.mode=cloud ``` ```console TOPIC STATUS audit.analytics.may2025 OK ``` You can make a topic a Cloud Topic only at topic creation time. In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking costs. When running multi-AZ clusters in general, Redpanda strongly recommends using [Follower Fetching](https://docs.redpanda.com/cloud-data-platform/develop/consume-data/follower-fetching/), which allows consumers to avoid crossing network zones. When possible, you can use [leader pinning](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/leader-pinning/), which positions a topic’s partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. For client-side tuning guidance, see [Configure producers for Cloud Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/configure-producers-for-cloud-topics/). --- # Page 554: Manage Topics **URL**: https://docs.redpanda.com/cloud-data-platform/develop/topics/config-topics.md --- # Manage Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: topics/config-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: topics/config-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/topics/config-topics.adoc description: Learn how to create topics, update topic configurations, and delete topics or records. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-05-26" --- Topics provide a way to organize events in a data streaming platform. ## [](#create-a-topic)Create a topic Creating a topic can be as simple as specifying a name for your topic on the command line. For example, to create a topic named `xyz`, run: ```bash rpk topic create xyz ``` This command creates a topic named `xyz` with one partition and three replicas, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. Redpanda Cloud supports 40,000 topics per cluster. ### [](#choose-the-number-of-partitions)Choose the number of partitions A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput. > 💡 **TIP** > > As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data. For example, suppose you plan to create a consumer group with 10 consumers. To create topic `xyz` with 10 partitions, run: ```bash rpk topic create xyz -p 10 ``` ## [](#update-topic-configurations)Update topic configurations After you create a topic, you can update the topic property settings for all new data written to it. For example, you can add partitions or change the cleanup policy. ### [](#add-partitions)Add partitions You can assign a certain number of partitions when you create a topic, and add partitions later. For example, suppose you add brokers to your cluster, and you want to take advantage of the additional processing power. To increase the number of partitions for existing topics, run: ```bash rpk topic add-partitions [TOPICS...] --num [#] ``` Note that `--num <#>` is the number of partitions to _add_, not the total number of partitions. > 📝 **NOTE** > > If a topic already has messages and you add partitions, the existing messages won’t be redistributed to the new partitions. If you require messages to be redistributed, then you must create a new topic with the new partition count, then stream the messages from the old topic to the new topic so they are appropriately distributed according to the new partition hashing. ### [](#reduce-the-number-of-partitions)Reduce the number of partitions You cannot reduce the number of partitions on an existing topic. The Kafka API does not support it: a record’s partition is chosen when the record is produced, and records that have already been written stay in the partition where they landed, so removing a partition would orphan its data. Redpanda assigns a record that has a key to a partition by hashing the key, and leaves a record without a key to the producer’s partitioner, which usually spreads such records across all available partitions. To move a topic’s data to fewer partitions, copy it to a new topic and switch your clients over. Before you start, check that your applications can tolerate the following: - **Duplicates**: Redpanda Connect delivers records at least once, so a restart or a retry during the copy can write the same record to the new topic twice. A lag of zero shows only how far the copy has committed, not that the new topic is free of duplicates. Either make your consumers idempotent, or deduplicate on a record ID after the copy. - **Ordering**: records keep their keys, so all records for a key still land on one partition and keep their order relative to each other, but the global order of records across partitions is not preserved. - **Retention**: the copy starts at the oldest record that is still retained. Records that retention or compaction has already removed cannot be copied, and both keep running during the copy, so complete the copy well within the topic’s retention period. - **Consumer offsets**: consumer group offsets are stored per topic, so the offsets your consumers committed on the original topic do not carry over. Each consumer starts from the beginning of the new topic and replays what the copy wrote, unless you set its offsets explicitly with `rpk group seek`. - **Write downtime**: producers must stop writing to the original topic before you switch clients over, so plan a window in which the topic accepts no writes. The following procedure reduces a topic named `orders` from three partitions to one. 1. Check the configuration of the original topic so that you can recreate it. Note the replication factor, and every row whose `SOURCE` is `DYNAMIC_TOPIC_CONFIG`, which is an override you must set on the new topic: ```bash rpk topic describe orders ``` Example output (abbreviated) ```bash SUMMARY ======= NAME orders PARTITIONS 3 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DYNAMIC_TOPIC_CONFIG segment.bytes 134217728 DEFAULT_CONFIG ``` Here, only `retention.ms` is an override. If the topic has Tiered Storage settings, a custom cleanup policy, or other overrides, carry all of them over: a new topic created without them silently falls back to the cluster defaults. 2. Create the new topic with the target number of partitions, the replication factor of the original topic, and each override from the previous step: ```bash rpk topic create orders-reduced --partitions 1 --replicas 1 --topic-config retention.ms=604800000 ``` Example output ```bash TOPIC STATUS orders-reduced OK ``` 3. Copy the data with a [Redpanda Connect](https://docs.redpanda.com/connect/get-started/about/) pipeline. This configuration reads all records that are still available in the original topic and preserves record keys, so records for the same key land on the same partition of the new topic: `reduce-partitions.yaml` ```yaml input: redpanda: seed_brokers: [""] topics: ["orders"] consumer_group: orders-to-orders-reduced start_offset: earliest output: redpanda: seed_brokers: [""] topic: orders-reduced key: ${! @kafka_key } ``` Give the consumer group a name that is unique to this copy, such as `-to-`. `start_offset: earliest` applies only when the group has no committed offset, so a group name that has been used before resumes from where it left off and skips records. ```bash rpk connect run reduce-partitions.yaml ``` Example output ```bash level=info msg="Launching a Redpanda Connect instance, use CTRL+C to close" level=info msg="Output type redpanda is now active" level=info msg="Input type redpanda is now active" ``` 4. Stop the producers that write to the original topic. Leave the pipeline running so that it copies the last records they wrote. 5. Wait for the copy to drain. It is complete when the consumer group reports a `LAG` of `0` for every partition of the original topic: ```bash rpk group describe orders-to-orders-reduced ``` Example output ```bash GROUP orders-to-orders-reduced COORDINATOR-NODE 0 COORDINATOR-PARTITION __consumer_offsets/0 STATE Stable BALANCER cooperative-sticky MEMBERS 1 TOTAL-LAG 0 TOPIC PARTITION CURRENT-OFFSET LOG-START-OFFSET LOG-END-OFFSET LAG MEMBER-ID CLIENT-ID HOST orders 0 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 orders 1 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 orders 2 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 ``` 6. Compare the record counts of the two topics. For each topic, the number of available records is the sum of `HIGH-WATERMARK` minus `LOG-START-OFFSET` across its partitions. A higher count on the new topic means the copy wrote duplicates: ```bash rpk topic describe orders -p rpk topic describe orders-reduced -p ``` Example output ```bash PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 0 1 [0] 0 3 1 0 1 [0] 0 3 2 0 1 [0] 0 3 PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 0 1 [0] 0 9 ``` Nine records across the three original partitions, and the same nine on the single partition of the new topic. 7. Point your producers and consumers at the new topic. Consumers start from the beginning of the new topic unless you set their offsets with `rpk group seek`. 8. Stop the pipeline with Ctrl+C. 9. When you no longer need the original topic, delete it to reclaim storage. See [Delete a topic](#delete-a-topic). > ⚠️ **CAUTION** > > Do not delete the original topic until the new topic holds the data you expect and your consumers are running against it. Deleting a topic deletes its data. ### [](#change-the-cleanup-policy)Change the cleanup policy The cleanup policy determines how to clean up the partition log files when they reach a certain size: - `delete` deletes data based on age or log size. Topics retain all records until then. - `compact` compacts the data by only keeping the latest values for each KEY. - `compact,delete` combines both methods. Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics. > ⚠️ **WARNING** > > All topic properties take effect immediately after being set. Do not modify properties on internal Redpanda topics (such as `__consumer_offsets`, `_schemas`, or other system topics) as this can cause cluster instability. For example, to change a topic’s policy to `compact`, run: ```bash rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact ``` ### [](#configure-write-caching)Configure write caching Write caching is a relaxed mode of [`acks=all`](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers/#acksall) that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. Write caching applies to user topics. It does not apply to transactions or consumer offsets: data written in the context of a transaction and consumer offset commits is always written to disk and fsynced before being acknowledged to the client. Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures. #### [](#configure-at-topic-level)Configure at topic level To override the cluster-level setting at the topic level, set the topic-level property `write.caching`: `rpk topic alter-config my_topic --set write.caching=true` With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. ### [](#remove-a-configuration-setting)Remove a configuration setting You can remove a configuration that overrides the default setting, and the setting will use the default value again. For example, suppose you altered the cleanup policy to use `compact` instead of the default, `delete`. Now you want to return the policy setting to the default. To remove the configuration setting `cleanup.policy=compact`, run `rpk topic alter-config` with the `--delete` flag: ```bash rpk topic alter-config [TOPICS...] --delete cleanup.policy ``` ## [](#list-topic-configuration-settings)List topic configuration settings To display all the configuration settings for a topic, run: ```bash rpk topic describe -c ``` The `-c` flag limits the command output to just the topic configurations. This command is useful for checking the default configuration settings before you make any changes and for verifying changes after you make them. The following command output displays after running `rpk topic describe test-topic`, where `test-topic` was created with default settings: ```bash rpk topic describe test_topic SUMMARY ======= NAME test_topic PARTITIONS 1 REPLICAS 3 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG max.message.bytes 20971520 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG ``` ## [](#delete-a-topic)Delete a topic To delete a topic, run: ```bash rpk topic delete ``` When a topic is deleted, its underlying data is deleted, too. To delete multiple topics at a time, provide a space-separated list. For example, to delete two topics named `topic1` and `topic2`, run: ```bash rpk topic delete topic1 topic2 ``` You can also use the `-r` flag to specify one or more regular expressions; then, any topic names that match the pattern you specify are deleted. For example, to delete topics with names that start with “f” and end with “r”, run: ```bash rpk topic delete -r '^f.*' '.*r$' ``` Note that the first regular expression must start with the `^` symbol, and the last expression must end with the `$` symbol. This requirement helps prevent accidental deletions. ## [](#delete-records-from-a-topic)Delete records from a topic Redpanda allows you to delete data from the beginning of a partition up to a specific offset (a monotonically increasing sequence number for records in a partition). Deleting records frees up disk space, which is especially helpful if your producers are pushing more data than anticipated in your retention plan. Delete records when you know that all consumers have read up to that given offset, and the data is no longer needed. There are different ways to delete records from a topic, including using the [`rpk topic trim-prefix`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-topic/rpk-topic-trim-prefix/) command, using the `DeleteRecords` Kafka API with Kafka clients, or using Redpanda Cloud. > 📝 **NOTE** > > - To delete records, `cleanup.policy` must be set to `delete` or `compact,delete`. > > - Object storage is deleted asynchronously. After messages are deleted, the partition’s start offset will have advanced, but garbage collection of deleted segments may not be complete. > > - Similar to Kafka, after deleting records, local storage and object storage may still contain data for deleted offsets. (Redpanda does not truncate segments. Instead, it bumps the start offset, then it attempts to delete as many whole segments as possible.) Data before the new start offset is not visible to clients but could be read by someone with access to the local disk of a Redpanda node. > ⚠️ **WARNING** > > When you delete records from a topic with a timestamp, Redpanda advances the partition start offset to the first record whose timestamp is after the threshold. If record timestamps are not in order with respect to offsets, this may result in unintended deletion of data. Before using a timestamp, verify that timestamps increase in the same order as offsets in the topic to avoid accidental data loss. For example: > > ```bash > rpk topic consume -n 50 --format '%o %d{go[2006-01-02T15:04:05Z07:00]} %k %v' > ``` ## [](#next-steps)Next steps [Configure Producers](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers/) --- # Page 555: Configure Producers for Cloud Topics **URL**: https://docs.redpanda.com/cloud-data-platform/develop/topics/configure-producers-for-cloud-topics.md --- # Configure Producers for Cloud Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Producers for Cloud Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: topics/configure-producers-for-cloud-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: topics/configure-producers-for-cloud-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/topics/configure-producers-for-cloud-topics.adoc description: Learn about producer configuration considerations for Cloud Topics. page-git-created-date: "2026-05-20" page-git-modified-date: "2026-05-26" --- This page describes how to tune the client producer for Cloud Topics (note that general producer configuration guidance still applies). See [Configure Producers](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers/). With idempotency enabled, Kafka’s protocol allows only five in-flight requests at a time on a single broker connection. This imposes a tight limit on how much data can be in flight at a single time. Over high-latency links, this is already a problem for standard topics. Cloud Topics use a 250ms batching interval at the broker to reduce cloud storage costs, which effectively makes every producer connection a high-latency link. Compared to standard topics, this changes producer tuning priorities. Configuring batch size, linger time, and request size correctly is critical for achieving good single-producer throughput. This section covers the key settings and example values for the most common client libraries. Calculate the maximum throughput for a single producer as follows: ```text broker_count * max_in_flight * request_size / latency_seconds ``` Throughput formula definitions: - `broker_count`: The number of brokers in the cluster. - `max_in_flight`: The maximum number of in-flight requests on a single connection. With an idempotent producer, this is usually five. - `latency_seconds`: The latency of a single produce request. Assuming normal operation, this can be equated to the 250ms as earlier. In practice, it is typically lower, as the timer starts with the first byte arriving. - `request_size`: The size of a single produce request. This is the primary tuning lever because the other factors are effectively constants. Another way to increase throughput in the system is to increase producer count, as doing so naturally increases available parallelism in the system as more connections are opened to the brokers. ## [](#producer-settings)Producer settings Most Kafka client libraries offer an assortment of tunables, with many properties impacting request size. The most important ones are described here, along with an example for the Kafka Java client, along with caveats from other libraries. ### [](#batches-and-requests)Batches and requests A Kafka produce request consists of one or more batches. A batch contains multiple records. Records mostly correspond to application-level messages as passed to the Kafka client API. At the broker level, the unit that matters is the batch, as everything happens at the batch level. Hence, creating the largest possible batches is the most important factor for performance in general, and equally applies for Cloud Topics. A request can contain multiple batches, this can sometimes alleviate the need for massive batches. However, this requires that a producer is producing to multiple partitions and that there are enough partitions per broker to fill the request with batches. As explained in [Java client](#java-client), there are also exceptions to this in some client libraries, so you should not blindly rely on them. ### [](#java-client)Java client For the Java client, Redpanda Data recommends the following as minimal settings: | Setting | Recommended value | | --- | --- | | linger.ms | 10ms | | batch.size | 131072 | | max.request.size | 1048576 (default) | As shown in the preceding formula, with a basic three-broker setup and enough partitions, you can achieve a maximum throughput of ~65 MB/s per producer. If more throughput is needed on a single producer, increase batch and max request size. Setting `linger.ms` to a higher value is recommended, but it’s less critical for Cloud Topics because as soon as there are five requests in flight, messages are force-batched even after crossing the `linger.ms` threshold. ### [](#librdkafka)librdkafka librdkafka is a commonly-used Kafka C library. However, it’s also the backing library for many other Kafka clients like confluent-kafka-python. librdkafka only allows a single batch in a produce request, unlike most other Kafka client libraries. This significantly cuts down how much data is packed into a single request. Thus, it’s important to increase `batch.size` to even higher values, as it is effectively the limiting factor. librdkafka has a default of 1MB, which typically allows for decent throughput. ### [](#idempotency-considerations)idempotency considerations Disabling idempotency allows you to avoid the in-flight limitation and vastly increases the number of concurrent in-flight requests and throughput. However, running without idempotency can result in duplicate and out-of-order messages, which for most applications is a problem, and is not recommended. In cases where message requirements are fairly lax, it can be a viable alternative. --- # Page 556: Topics Overview **URL**: https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic.md --- # Topics Overview > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Topics Overview latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: topics/create-topic page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: topics/create-topic.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/topics/create-topic.adoc description: Learn how to create a topic for a Redpanda Cloud cluster. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-07-15" --- Topics provide a way to organize events. After creating a cluster, you can create a topic in it. Each cluster can have up to 40,000 topics. Topic properties are populated from information stored in the broker. Redpanda features, such as Tiered Storage, are enabled and configured by default in Redpanda Cloud. You can optionally overwrite some settings. > ⚠️ **WARNING** > > Modifying the properties of topics that are created and managed by Redpanda applications can cause unexpected errors. This may lead to connector and cluster failures. | Property | Description | | --- | --- | | Partitions | The number of partitions for the topic. | | Replication factor | The number of partition replicas for the topic.Redpanda Cloud requires a minimum of 3 topic replicas. If a topic is created with a replication factor of 1, Redpanda resets the replication factor to 3. | | Cleanup policy | The policy that determines how to clean up old log segments.The default is delete. | | Retention time | The maximum length of time to keep messages in a topic.The default is 7 days. | | Retention size | The maximum size of each partition. If a partition reaches this size and more messages are added, the oldest messages are deleted.The default is infinite. | | Message size | The maximum size of a message or batch for a newly-created topic.The default is 20 MiB for BYOC and Dedicated clusters, and 8 MiB for Serverless clusters. You can increase this value up to 32 MiB for BYOC and Dedicated clusters, and 20 MiB for Serverless clusters, with the max.message.bytes topic property. | | Segment size | The maximum size of a log segment. When a segment reaches this size, Redpanda closes it and starts a new one.Redpanda Cloud sets the segment size for your cluster automatically. You can override it for a topic with the segment.bytes property. | ## [](#next-steps)Next steps - [Manage Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/config-topics/) - [Manage Cloud Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/cloud-topics/) --- # Page 557: Transactions **URL**: https://docs.redpanda.com/cloud-data-platform/develop/transactions.md --- # Transactions > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Transactions latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: transactions page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: transactions.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/develop/pages/transactions.adoc description: Learn how to use transactions; for example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Redpanda supports Apache Kafka®-compatible transaction semantics and APIs. For example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. A transaction can span partitions from different topics, and a topic can be deleted while there are active transactions on one or more of its partitions. In-flight transactions can detect deletion events, remove the deleted partitions (and related messages) from the transaction scope, and commit changes to the remaining partitions. If a producer is sending multiple messages to the same or different partitions, and network connectivity or broker failure cause the transaction to fail, then it’s guaranteed that either all messages are written to the partitions or none. This is important for applications that require strict guarantees, like financial services transactions. Transactions guarantee both exactly-once semantics (EOS) and atomicity: - EOS helps developers avoid the anomalies of at-most-once processing (with potential lost events) and at-least-once processing (with potential duplicated events). Redpanda supports EOS when transactions are used in combination with [idempotent producers](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/idempotent-producers/). - Atomicity additionally commits a set of messages across partitions as a unit: either all messages are committed or none. Encapsulated data received or sent across multiple topics in a single operation can only succeed or fail globally. ## [](#use-transactions)Use transactions By default, the `[enable_transactions](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#enable_transactions)` cluster configuration property is set to true. However, in the following use cases, clients must explicitly use the Transactions API to perform operations within a transaction: - [Atomic (all or nothing) publishing of multiple messages](#atomic-publishing-of-multiple-messages) - [Exactly-once stream processing](#exactly-once-stream-processing) When you use transactions, you must set the [`transactional.id`](https://kafka.apache.org/documentation/#producerconfigs_transactional.id) property in the producer configuration. This property uniquely identifies the producer and enables reliable semantics across multiple producer sessions. It ensures that all transactions issued by a given producer are completed before any new transactions are started. ### [](#atomic-publishing-of-multiple-messages)Atomic publishing of multiple messages A banking IT system with an event-sourcing microservice architecture illustrates why transactions are necessary. In this system, each bank branch is implemented as an independent microservice that manages its own distinct set of accounts. Every branch maintains its own transaction history, stored as a Redpanda partition. When a branch starts, it replays the transaction history to reconstruct its current state. Financial transactions such as money transfers require the following guarantees: - A sender can’t withdraw more than the account withdrawal limit. - A recipient receives exactly the same amount sent. - A transaction is fast and is run at most once. - If a transaction fails, the system rolls back to the initial state. - Without withdrawals and deposits, the amount of money in the system remains constant with any history of money transfers. These requirements are easy to satisfy when the sender and the recipient of a financial transaction are hosted by the same branch. The operation doesn’t leave the consistency domain, and all checks and locks can be performed within a single service (ledger). Things get more complex with cross-branch financial transactions, because they involve several ledgers, and the operations should be performed atomically (all or nothing). The default approach (saga pattern) breaks a transaction into a sequence of reversible idempotent steps; however, this violates the isolation principle and adds complexity, making the application responsible for orchestrating the steps. Redpanda natively supports transactions, so it’s possible to atomically update several ledgers at the same time. For example: Show multi-ledger transaction example: ```java Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); props.put(ProducerConfig.ACKS_CONFIG, "all"); props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "app-id"); Producer producer = null; while (true) { // waiting for somebody to initiate a financial transaction var sender_branch = ...; var sender_account = ...; var recipient_branch = ...; var recipient_account = ...; var amount = 42; if (producer == null) { try { producer = new KafkaProducer<>(props); producer.initTransactions(); } catch (Exception e1) { // TIP: log error for further analysis try { if (producer != null) { producer.close(); } } catch(Exception e2) { } producer = null; // TIP: notify the initiator of a transaction about the failure continue; } } producer.beginTransaction(); try { var f1 = producer.send(new ProducerRecord("ledger", sender_branch, sender_account, "" + (-amount))); var f2 = producer.send(new ProducerRecord("ledger", recipient_branch, recipient_account, "" + amount)); f1.get(); f2.get(); } catch (Exception e1) { // TIP: log error for further analysis try { producer.abortTransaction(); } catch (Exception e2) { // TIP: log error for further analysis try { producer.close(); } catch (Exception e3) { } producer = null; } // TIP: notify the initiator of a transaction about the failure continue; } try { producer.commitTransaction(); } catch (Exception e1) { try { producer.close(); } catch (Exception e3) {} producer = null; // TIP: notify the initiator of a transaction about the failure continue; } // TIP: notify the initiator of a transaction about the success } ``` When a transaction fails before a `commitTransaction` attempt completes, you can assume that it is not executed. When a transaction fails after a `commitTransaction` attempt completes, the true transaction status is unknown. Redpanda only guarantees that there isn’t a partial result: either the transaction is committed and complete, or it is fully rolled back. ### [](#exactly-once-stream-processing)Exactly-once stream processing Redpanda is commonly used as a pipe connecting different applications and storage systems. An application could use an OLTP database and then rely on change data capture to deliver the changes to a data warehouse. Redpanda transactions let you use streams as a smart pipe in your applications, building complex atomic operations that transform, aggregate, or otherwise process data transiting between external applications and storage systems. For example, here is the regular pipe flow: Postgresql -> topic -> warehouse Here is the smart pipe flow, with a transformation in `topic(1) -> topic(2)`: Postgresql -> topic(1) transform topic(2) -> warehouse The transformation reads a record from `topic(1)`, processes it, and writes it to `topic(2)`. Without transactions, an intermittent error can cause a message to be lost or processed several times. With transactions, Redpanda guarantees exactly-once semantics. For example: Show exactly-once processing example: ```java var source = "source-topic"; var target = "target-topic"; Properties pprops = new Properties(); pprops.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); pprops.put(ProducerConfig.ACKS_CONFIG, "all"); pprops.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); pprops.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, UUID.randomUUID().toString()); Properties cprops = new Properties(); cprops.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); cprops.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); cprops.put(ConsumerConfig.GROUP_ID_CONFIG, "app-id"); cprops.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); cprops.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); Consumer consumer = null; Producer producer = null; boolean should_reset = false; while (true) { if (should_reset) { should_reset = false; if (consumer != null) { try { consumer.close(); } catch(Exception e) {} consumer = null; } if (producer != null) { try { producer.close(); } catch (Exception e2) {} producer = null; } } try { if (consumer == null) { consumer = new KafkaConsumer<>(cprops); consumer.subscribe(Collections.singleton(source)); } } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } try { if (producer == null) { producer = new KafkaProducer<>(pprops); producer.initTransactions(); } } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } ConsumerRecords records = null; try { records = consumer.poll(Duration.ofMillis(10000)); } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } var it = records.iterator(); while (it.hasNext()) { var record = it.next(); // transformation var old_value = record.value(); var new_value = old_value.toUpperCase(); try { producer.beginTransaction(); producer.send(new ProducerRecord(target, record.key(), new_value)); var offsets = new HashMap(); offsets.put(new TopicPartition(source, record.partition()), new OffsetAndMetadata(record.offset() + 1)); producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata()); } catch (Exception e1) { // TIP: log error for further analysis try { producer.abortTransaction(); } catch (Exception e2) { } should_reset = true; break; } try { producer.commitTransaction(); } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; break; } } } ``` #### [](#exactly-once-processing-configuration-requirements)Exactly-once processing configuration requirements Redpanda’s default configuration supports exactly-once processing. To preserve this capability, ensure the following settings are maintained: - `enable_idempotence = true` - `enable_transactions = true` - `transaction_coordinator_delete_retention_ms` is greater than or equal to `transactional_id_expiration_ms` ## [](#best-practices)Best practices To help avoid common pitfalls and optimize performance, consider the following when configuring transactional workloads in Redpanda: ### [](#tune-producer-id-limits)Tune producer ID limits For production environments with heavy producer usage, configure both [`max_concurrent_producer_ids`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#max_concurrent_producer_ids) and [`transactional_id_expiration_ms`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#transactional_id_expiration_ms) to prevent out-of-memory (OOM) crashes. Setting limits on producer IDs helps manage memory usage in high-throughput environments, particularly when using transactions or idempotent producers. If you have\`kafka\_connections\_max\` configured, you can determine an appropriate value for `max_concurrent_producer_ids` based on your connection patterns. - Lower bound: `kafka_connections_max` / `number_of_shards`, assuming each producer connects to only one shard. - Upper bound: `topic_partitions_per_shard` \* `kafka_connections_max`, assuming producers connect to all shards. If `kafka_connections_max` is not configured, estimate the value for `max_concurrent_producer_ids` based on your application patterns. A conservative approach is to start with 1000-5000 per shard, then monitor and adjust as needed. Applications with many partitions per producer typically require higher values, such as 10000 or more per shard. Tune `transactional_id_expiration_ms` based on your application’s transaction patterns. Calculate this value by taking your longest expected transaction time and adding a safety buffer. For example, if transactions typically run for 30 minutes, consider setting this to 2-4 hours. Short-lived transactions can use values between 1-4 hours, while batch processing applications should match their batch interval plus buffer time. Interactive applications may benefit from shorter values to free up memory faster. Client applications should minimize producer ID churn. Reuse producer instances when possible, instead of creating new ones for each operation. Avoid using random transactional IDs, as some Flink configurations do, because this creates excessive producer ID churn. Instead, use consistent transactional IDs that can be resumed across application restarts. ### [](#configure-transaction-timeouts-and-limits)Configure transaction timeouts and limits - If a consumer is configured to use the read\_committed isolation level, it can only process successfully committed transactions. As a result, an ongoing transaction with a large timeout that becomes stuck could prevent the consumer from processing other committed transactions. To avoid this, don’t set the transaction timeout client setting (`transaction.timeout.ms` in the Kafka Java client implementation) to a value that is too high. The longer the timeout, the longer consumers may be blocked. ## [](#handle-transaction-failures)Handle transaction failures Different transactions require different approaches to handling failures within the application. Consider the approaches to failed or timed-out transactions in the provided use cases: - Publishing of multiple messages: The request came from outside the system, and it is the application’s responsibility to discover the true status of a timed-out transaction. (This example doesn’t use consumer groups to distribute partitions between consumers.) - Exactly-once streaming (consume-transform-loop): This is a closed system. Upon re-initialization of the consumer and producer, the system automatically discovers the moment it was interrupted and continues from that place. Additionally, this automatically scales by the number of partitions. Run another instance of the application, and it starts processing its share of partitions in the source topic. ## [](#transactions-with-compacted-segments)Transactions with compacted segments Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches). ## [](#suggested-reading)Suggested reading - [Kafka-compatible fast distributed transactions](https://redpanda.com/blog/fast-transactions) --- # Page 558: Get Started **URL**: https://docs.redpanda.com/cloud-data-platform/get-started.md --- # Get Started > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Get Started latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/index.adoc description: Get Started index page. page-git-created-date: "2024-06-06" page-git-modified-date: "2024-06-07" --- - [What’s New in Redpanda Cloud](whats-new-cloud/) Summary of new features in Redpanda Cloud. - [Redpanda Cloud Overview](cloud-overview/) Learn about Redpanda Cloud deployment options including BYOC, Dedicated, and Serverless clusters. - [BYOC Architecture](byoc-arch/) Learn about the control plane - data plane architecture in BYOC. --- # Page 559: How Redpanda Works **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/architecture.md --- # How Redpanda Works > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: How Redpanda Works latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: architecture page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: architecture.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/architecture.adoc description: Learn specifics about Redpanda architecture. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- At its core, Redpanda is a fault-tolerant transaction log for storing event streams. Producers and consumers interact with Redpanda using the Kafka API. To achieve high scalability, producers and consumers are fully decoupled. Redpanda provides strong guarantees to producers that events are stored durably within the system, and consumers can subscribe to Redpanda and read the events asynchronously. Redpanda achieves this decoupling by organizing events into topics. Topics represent a logical grouping of events that are written to the same log. A topic can have multiple producers writing events to it and multiple consumers reading events from it. This page provides details about how Redpanda works. For a high-level overview, see [Introduction to Redpanda](https://docs.redpanda.com/cloud-data-platform/get-started/intro-to-events/). ## [](#tiered-storage)Tiered Storage Redpanda Tiered Storage is a multi-tiered object storage solution that provides the ability to offload log segments to object storage in near real time. Tiered Storage can be combined with local storage to provide long-term data retention and disaster recovery on a per-topic basis. Consumers that read from more recent offsets continue to read from local storage, and consumers that read from historical offsets read from object storage, all with the same API. Consumers can read and reread events from any point within the maximum retention period, whether the events reside on local or object storage. As data in object storage grows, the metadata for it grows. To support efficient long-term data retention, Redpanda splits the metadata in object storage, maintaining metadata of only recently-updated segments in memory or local disk, while safely archiving the remaining metadata in object storage and caching it locally on disk. Archived metadata is then loaded only when historical data is accessed. This allows Tiered Storage to handle partitions of virtually any size or retention length. ## [](#partitions)Partitions To scale topics, Redpanda shards them into one or more partitions that are distributed across the nodes in a cluster. This allows for concurrent writing and reading from multiple nodes. When producers write to a topic, they route events to one of the topic’s partitions. Events with the same key (like a stock ticker) are always routed to the same partition, and Redpanda guarantees the order of events at the partition level. Consumers read events from a partition in the order that they were written. If a key is not specified, then events are sent to all topic partitions in a round-robin fashion. ## [](#raft-consensus-algorithm)Raft consensus algorithm Redpanda provides strong guarantees for data safety and fault tolerance. Events written to a topic partition are appended to a log file on disk. They can be replicated to other nodes in the cluster and appended to their copies of the log file on disk to prevent data loss in the event of failure. The [Raft consensus algorithm](https://raft.github.io/) is used for data replication. Every topic partition forms a Raft group consisting of a single elected leader and zero or more followers (as specified by the topic’s replication factor). A Raft group can tolerate ƒ failures given 2ƒ+1 nodes. For example, in a cluster with five nodes and a topic with a replication factor of five, the topic remains fully operational if two nodes fail. Raft is a majority vote algorithm. For a leader to acknowledge that an event has been committed to a partition, a majority of its replicas must have written that event to their copy of the log. When a majority (quorum) of responses have been received, the leader can make the event available to consumers and acknowledge receipt of the event when `acks=all (-1)`. [Producer acknowledgement settings](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers/#producer-acknowledgement-settings) define how producers and leaders communicate their status while transferring data. As long as the leader and a majority of the replicas are stable, Redpanda can tolerate disturbances in a minority of the replicas. If [gray failures](https://blog.acolyer.org/2017/06/15/gray-failure-the-achilles-heel-of-cloud-scale-systems/) cause a minority of replicas to respond slower than normal, then the leader does not have to wait for their responses to progress, and any additional latency is not passed on to the clients. The result is that Redpanda is less sensitive to faults and can deliver predictable performance. ## [](#partition-leadership-elections)Partition leadership elections [Raft](https://raft.github.io/) uses a heartbeat mechanism to maintain leader authority and to trigger leader elections. The partition leader sends a heartbeat to all followers every 150 milliseconds to assert its leadership in the current term (an election cycle). If a follower does not receive a heartbeat within the election timeout, it triggers an election to choose a new partition leader. Configure the election timeout with the [`election_timeout_ms`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#election_timeout_ms) cluster property (default: 1500 milliseconds). The follower increments its term and votes for itself to be the leader for that term. It then sends a vote request to the other nodes and waits for one of the following scenarios: - It receives a majority of votes and becomes the leader. Raft guarantees that at most one candidate can be elected the leader for a given term. - Another follower establishes itself as the leader. While waiting for votes, the candidate may receive communication from another node in the group claiming to be the leader. The candidate only accepts the claim if its term is greater than or equal to the candidate’s term; otherwise, the communication is rejected and the candidate continues to wait for votes. - No leader is elected over a period of time. If multiple followers timeout and become election candidates at the same time, it’s possible that no candidate gets a majority of votes. When this happens, each candidate increments its term and triggers a new election round. Raft uses a random timeout between 150-300 milliseconds to ensure that split votes are rare and resolved quickly. As long as there is a timing inequality between heartbeat time, election timeout, and mean time between node failures (MTBF), then Raft can elect and maintain a steady leader and make progress. A leader can maintain its position as long as one of the ten heartbeat messages it sends to all of its followers every 1.5 seconds is received; otherwise, a new leader is elected. If a follower triggers an election, but the incumbent leader subsequently springs back to life and starts sending data again, then it’s too late. As part of the election process, the follower (now an election candidate) incremented the term and rejects requests from the previous term, essentially forcing a leadership change. If a cluster is experiencing wider network infrastructure problems that result in latencies above the heartbeat timeout, then back-to-back election rounds can be triggered. During this period, unstable Raft groups may not be able to form a quorum. This results in partitions rejecting writes, but data previously written to disk is not lost. Redpanda has a Raft-priority implementation that allows the system to settle quickly after network outages. ## [](#controller-partition-and-snapshots)Controller partition and snapshots Redpanda stores metadata update commands (such as creating and deleting topics or users) in a system partition called the controller partition. A new snapshot is created after each controller command is added, or, with rapid updates, after a set period of time (default is 60 seconds). Controller snapshots save the current cluster metadata state to disk, so startup is fast. For example, with a partition that has moved several times, a snapshot can restore the latest state without replaying every move command. Each broker has a snapshot file stored in the controller log directory, such as `/var/lib/redpanda/data/redpanda/controller/0_0/snapshot`. The controller partition is replicated by a Raft group that includes all cluster brokers, and the controller snapshot is the Raft snapshot for this group. Snapshots are hydrated when a broker joins the cluster or restarts. Snapshots are enabled by default for all clusters, both new and upgraded. ## [](#optimized-platform-performance)Optimized platform performance Redpanda is designed to exploit advances in modern hardware, from the network down to the disks. Network bandwidth has increased considerably, especially in object storage, and spinning disks have been replaced by SSD devices that deliver better I/O performance. CPUs are faster too, but this is largely due to the increased core counts as opposed to the increase in single-core speeds. Redpanda has tuners that detect your hardware configuration to automatically optimize itself. Examples of platform and kernel features that Redpanda uses to optimize its performance: - Direct Memory Access (DMA) for disk I/O - Sparse file system support with XFS - Distribution of interrupt request (IRQ) processing between CPU cores - Isolated processes with control groups (cgroups) - Disabled CPU power-saving modes - Upfront memory allocation, partitioned and pinned to CPU cores ## [](#tpc)Thread-per-core model Redpanda implements a thread-per-core programming model through its use of the [Seastar](https://seastar.io/) library. This allows Redpanda to pin each of its application threads to a CPU core to avoid context switching and blocking. It combines this with message passing to asynchronously communicate between the pinned threads. With this, Redpanda avoids the overhead of context switching and expensive locking operations to improve processing performance and efficiency. From a sizing perspective, Redpanda’s ability to efficiently use all available hardware enables it to scale up to get the most out of your infrastructure, before you’re forced to scale out to meet the demands of your workload. Redpanda delivers better performance with a smaller footprint, resulting in reduced operational costs and complexity. --- # Page 560: BYOC Architecture **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch.md --- # BYOC Architecture > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: BYOC Architecture latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: byoc-arch page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: byoc-arch.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/byoc-arch.adoc description: Learn about the control plane - data plane architecture in BYOC. page-git-created-date: "2025-04-01" page-git-modified-date: "2026-04-07" --- With Bring Your Own Cloud (BYOC) clusters, you deploy Redpanda in your own cloud (AWS, Azure, or GCP), and all data is contained in your own environment. This provides an additional layer of security and isolation. Redpanda handles provisioning, operations, and maintenance of the underlying infrastructure, including Kubernetes. ## [](#control-plane-data-plane)Control plane - data plane For high availability, Redpanda Cloud uses the following control plane - data plane architecture: ![Control plane and data plane](https://docs.redpanda.com/cloud-data-platform/shared/_images/control_d_plane.png) - **Control plane**: This is a Redpanda Cloud managed service that manages provisioning, operations, and maintenance of clusters with Kubernetes under the hood, including Kubernetes version upgrades and infrastructure maintenance. The control plane enforces rules in the data plane. You can use [RBAC](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/) or [GBAC](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/gbac/) in the control plane to manage access to organization-level resources like clusters, resource groups, and networks. - **Data plane**: This is where your cluster lives. The term _data plane_ is sometimes used interchangeably with _cluster_. The data plane is where you manage topics, consumer groups, connectors, and schemas. You can use [RBAC](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac_dp/) or [GBAC](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/gbac_dp/) in the data plane to configure cluster-level permissions for provisioned users at scale. IAM permissions allow the Redpanda Cloud agent to access the cloud provider API to create and manage cluster resources. The permissions follow the principle of least privilege, limiting access to only what is necessary. Clusters are configured and maintained in the control plane, but they remain available even if the network connection to the control plane is lost. > 💡 **TIP** > > In the Redpanda Cloud UI, you can identify which plane you’re in by the side navigation: > > - **Control Plane:** Visible after login at the organization level. Here you can select, create, and delete clusters, networks, and resource groups. > > - **Data Plane:** Visible after selecting a specific cluster. Here you can work with topics, consumer groups, connectors, and schemas. ## [](#byoc-setup)BYOC setup In a BYOC architecture, you deploy the data plane in your own VPC. All network connections into the data plane take place through either a public endpoint, or for private clusters, through Redpanda Cloud network connections such as VPC peering, AWS PrivateLink, Azure Private Link, or GCP Private Service Connect. Customer data never leaves the data plane. A BYOC cluster is initially set up from the control plane. This is a two-step process performed by `rpk cloud byoc apply`: 1. You bootstrap a virtual machine (VM) in your VPC. This VM launches the agent and bootstraps the necessary infrastructure. Redpanda then assigns fine-grained IAM policies following least privilege, creating dedicated IAM roles per workload with only the permissions each requires. 2. The agent communicates with the control plane to pull the cluster specifications. After the agent is up and running, it connects to the control plane and starts dequeuing and applying cluster specifications that provision, configure, and maintain clusters. The agent is in constant communication with the control plane, receiving and applying cluster specifications and exchanging cluster metadata. Agents are authenticated and authorized through opaque and ephemeral tokens, and they have dedicated job queues in the control plane. Agents also manage VPC peering networks. ![cloud\_byoc\_apply](https://docs.redpanda.com/cloud-data-platform/shared/_images/byoc_apply.png) > 📝 **NOTE** > > To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply` with your cloud provider. > 📝 **NOTE** > > Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. --- # Page 561: Redpanda Cloud Overview **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview.md --- # Redpanda Cloud Overview > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud Overview latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cloud-overview page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cloud-overview.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cloud-overview.adoc description: Learn about Redpanda Cloud deployment options including BYOC, Dedicated, and Serverless clusters. page-git-created-date: "2024-06-06" page-git-modified-date: "2026-08-13" --- Redpanda Cloud is a complete data streaming platform delivered as a fully-managed service. It provides automated upgrades and patching, data balancing, and support while continuously monitoring your data to meet strict performance, availability, reliability, and security requirements. All Redpanda Cloud clusters are deployed with an integrated [Redpanda Console](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#redpanda-console), and all clusters have access to unlimited retention and 300+ data connectors with [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#redpanda-connect). ## [](#redpanda-cloud-deployment-options)Redpanda Cloud deployment options Redpanda Cloud applications are supported by three fully-managed deployment options: - **[Serverless](#serverless)**: Fastest way to get started with automatic scaling - **[Dedicated](#dedicated)**: Production clusters in Redpanda’s cloud with enhanced isolation - **[Bring Your Own Cloud (BYOC)](#bring-your-own-cloud-byoc)**: Maximum control and security by deploying in your own cloud environment ### [](#quick-comparison)Quick comparison | | Serverless | Dedicated | BYOC | | --- | --- | --- | --- | | Best for | Starter projects and applications with low or variable traffic | Production clusters requiring cloud hosting, higher throughput, and extra isolation | Production clusters requiring data sovereignty, the highest throughput, and added security | | Deployment | Redpanda’s cloud (AWS/GCP) | Redpanda’s cloud (AWS/Azure/GCP) | Your cloud account (AWS/Azure/GCP) | | Tenancy | Multi-tenant | Single-tenant | Single-tenant | | Cloud SLA | 99.9% | 99.99%, multi-AZ | 99.99%, multi-AZ | | Max throughput (write, read) | Up to 100 MB/s, 300 MB/s | Up to 400 MB/s, 800 MB/s | Up to 2 GB/s, 4 GB/s | | Partitions, pre-replication | Up to 5,000 | Up to 45,600 | Up to 112,500 | | Max message size (MiB) | 8 (default), 20 (max) | 20 (default), 32 (max) | 20 (default), 32 (max) | | Private networking | ✓ | ✓ | ✓ | | SSO authentication | ✓ (GitHub, Google) | ✓ (GitHub, Google, OIDC) | ✓ (GitHub, Google, OIDC) | | Redpanda Connect | ✓ | ✓ | ✓ | | Role-based access control (RBAC) & audit logs | ✗ | ✓ | ✓ | | Group-based access control (GBAC) | ✗ | ✓ | ✓ | | Prometheus/OpenMetrics endpoint for cluster metrics | ✓ | ✓ | ✓ | | Multiple availability zones (AZs) | ✗ | ✓ | ✓ | | Cluster properties editing | ✗ | ✓ (AWS/GCP) | ✓ (AWS/GCP) | | Kafka Connect | ✗ | ✓ (disabled by default) | ✓ (disabled by default) | | Redpanda Support | Enterprise support with annual contracts | Enterprise support | Enterprise support for BYOC; Premium support required for BYOVPC/BYOVNet | > 📝 **NOTE** > > - The partition limit is the number of logical partitions before replication occurs. Redpanda Cloud uses a replication factor of three. > > - Enterprise support provides access to streaming experts 24/5, with 24/7 priority escalation for production outages. Premium support provides an enhanced Support SLA. > > - See also: [Serverless vs BYOC/Dedicated](#serverless-vs-byocdedicated) ### [](#serverless)Serverless Serverless is the fastest and easiest way to start data streaming. With Serverless clusters, you host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. This is a production-ready deployment option with a cluster available instantly, and you only pay for what you consume. > 📝 **NOTE** > > - Serverless on GCP is currently in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release. #### [](#sign-up-for-serverless)Sign up for Serverless ##### Free trial A [free trial on AWS](https://www.redpanda.com/try-redpanda) is the fastest way to get started with Serverless. Each free-trial customer qualifies for $100 (USD) in credits to spend in the first 30 days. This should be enough to run Redpanda with reasonable throughput. No credit card is required. To continue using Serverless after your trial expires, you can enter a credit card and pay as you go. Any remaining credit balance is used before you are charged. When either the credits expire or the days in the trial expire, the clusters move into a suspended state, and you won’t be able to access your data in either the Redpanda Cloud Console or with the Kafka API. There is a seven-day grace period following the end of the trial when you can add your credit card and restore service. After that, the data is permanently deleted. For questions about the trial, use the **#serverless** [Community Slack](https://redpandacommunity.slack.com/) channel. After you start a trial, Redpanda instantly prepares an account for you. The first time you sign in, you can answer a few quick questions about your project so Redpanda can tailor your experience. Your account includes a `welcome` cluster with a `hello-world` demo topic you can explore. It includes sample data so you can see how real-time messaging works before sending your own data. On that first visit, the **Overview** page shows a **Get started** button with guided ways to [interact with your cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#interact-with-your-cluster): create a Redpanda Connect [pipeline](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#pipeline), use `rpk` from the command line, or connect with your own Kafka client. To get started with `rpk`: 1. Log in with `rpk cloud login`. 2. Consume from the `hello-world` topic with `rpk topic consume hello-world`. 3. In the [Redpanda Cloud Console](https://cloud.redpanda.com), navigate to the **Topics** page and open the `hello-world` topic to see the included messages. ##### Redpanda Sales To request a private offer with possible discounts for annual committed use, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). When you subscribe to Serverless through Redpanda Sales, you gain immediate access to Enterprise support. Redpanda creates a cloud organization for you and sends you a welcome email. ##### AWS Marketplace New subscriptions to Redpanda Cloud through [AWS Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/aws-pay-as-you-go/) receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. > 📝 **NOTE** > > When you subscribe to Redpanda through AWS Marketplace, you do not have immediate access to Enterprise support, only the [Community Slack](https://redpandacommunity.slack.com/) channel. For Enterprise support, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). Redpanda creates a cloud organization for you and sends you a welcome email. ##### Google Cloud Marketplace New subscriptions to Redpanda Cloud through [Google Cloud Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/gcp-pay-as-you-go/) receive $300 (USD) in free credits to spend in the first 30 days. Google Cloud Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. > 📝 **NOTE** > > When you subscribe to Redpanda through Google Cloud Marketplace, you do not have immediate access to Enterprise support, only the [Community Slack](https://redpandacommunity.slack.com/) channel. For Enterprise support, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). Redpanda creates a cloud organization for you and sends you a welcome email. ### [](#dedicated)Dedicated With Dedicated clusters, you host your data on Redpanda Cloud resources (AWS, GCP, or Azure), and Redpanda handles provisioning, operations, and maintenance. When you create a Dedicated cluster, you select the supported [tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/) that meets your compute and storage needs. #### [](#sign-up-for-dedicated)Sign up for Dedicated ##### Redpanda Sales To request a private offer with possible discounts for monthly or annual committed use, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). With a usage-based billing commitment, you sign up for a minimum spend amount through [AWS Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/aws-commit/), [Azure Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/azure-commit/), or [Google Cloud Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/gcp-commit/). Redpanda creates a cloud organization for you and sends you a welcome email. You can then provision Dedicated clusters in Redpanda Cloud, and you can view invoices and manage your subscription in the marketplace. ##### AWS Marketplace New subscriptions to Redpanda Cloud through [AWS Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/aws-pay-as-you-go/) receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. Redpanda creates a cloud organization for you and sends you a welcome email. ### [](#bring-your-own-cloud-byoc)Bring Your Own Cloud (BYOC) With BYOC clusters, you deploy the Redpanda [data plane](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#data-plane) into your existing VPC (for AWS and GCP) or VNet (for Azure), and all data is contained in your own environment. This provides an additional layer of security and isolation. (See [BYOC Architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/).) Redpanda manages provisioning, monitoring, upgrades, and security policies, including the underlying infrastructure and Kubernetes used to run the cluster. Redpanda also manages required resources in your VPC or VNet, including subnets (subnetworks in GCP), IAM roles, and object storage resources (for example, S3 buckets or Azure Storage accounts). For full details, see [Upgrades and Maintenance](https://docs.redpanda.com/cloud-data-platform/manage/maintenance/). #### [](#bring-your-own-vpcvnet-byovpcbyovnet)Bring Your Own VPC/VNet (BYOVPC/BYOVNet) With BYOVPC/BYOVNet clusters, you take full control of the networking lifecycle. Compared to standard BYOC, BYOVPC/BYOVNet provides more security, but the configuration is more complex. See the [shared responsibility model](#shared-responsibility-model) to understand what you manage versus what Redpanda manages. The BYOC infrastructure that Redpanda manages should not be used to deploy any other workloads. For details about the control plane - data plane framework in BYOC, see [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). #### [](#sign-up-for-byoc)Sign up for BYOC To start using BYOC, contact [Redpanda sales](https://redpanda.com/try-redpanda?section=enterprise-trial) to request a private offer with possible discounts. You are billed directly or through Google Cloud Marketplace or AWS Marketplace. ### [](#serverless-vs-byocdedicated)Serverless vs BYOC/Dedicated Serverless clusters are a good fit for the following use cases: - Quick setup for development or testing - Variable or unpredictable traffic patterns - No upfront cost commitment - Isolated environments for different applications Consider BYOC or Dedicated if you need more control over the deployment or if you have workloads with consistently-high throughput. BYOC and Dedicated clusters offer the following features: - Multiple availability zones (AZs). A multi-AZ cluster provides higher resiliency in the event of a failure in one of the zones. - Role-based access control (RBAC) in the data plane - Group-based access control (GBAC) - Kafka Connect - Higher limits and quotas. See [BYOC usage tiers](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/) and [Dedicated usage tiers](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/) compared to [Serverless limits](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#serverless-usage-limits). ## [](#redpanda-cloud-architecture)Redpanda Cloud architecture When you sign up for a Redpanda account, Redpanda creates an organization for you. Your organization contains all your Redpanda resources, including your clusters and networks. Within your organization, Redpanda creates a default resource group to contain your resources. You can rename this resource group, and you can create more resource groups. For example, you may want different resource groups for production and testing. > 💡 **TIP** > > For more detailed information about the Redpanda platform, see [Introduction to Redpanda](https://docs.redpanda.com/cloud-data-platform/get-started/intro-to-events/) and [How Redpanda Works](https://docs.redpanda.com/cloud-data-platform/get-started/architecture/). ## [](#shared-responsibility-model)Shared responsibility model The Redpanda Cloud shared responsibility model lists the security areas owned by Redpanda and the security areas owned by customers. Responsibilities depend on the type of deployment. ### BYOC | Resource | Redpanda responsibility | Customer responsibility | | --- | --- | --- | | Redpanda upgrades and hotfixes | ✓ | | | Cost management and attribution | ✓ | ✓ | | Software vulnerability remediation | ✓ | | | Infrastructure vulnerability remediation | ✓ | | | IAM (roles, service accounts, access segmentation) | ✓ | ✓ | | Compute | ✓ | | | Redpanda agent VM maintenance | ✓ | | | VPC (subnets, routing, firewall) | ✓ | ✓ | | VPC peering | | ✓ | | VPC private links (service endpoint) | ✓ | | | VPC private links (consumer endpoint) | | ✓ | | Local storage | ✓ | | | Tiered Storage | ✓ | | | Control plane | ✓ | | | Access controls and audit | ✓ | ✓ | | Managed disaster recovery | | ✓ | | Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) | ✓ | | | Availability service-level agreement (SLA) | ✓ (subject to required access to customer resources) | | | Proactive threat detection | ✓ | ✓ | | Static secret rotation | ✓ | | | Incident response | ✓ | | | Resilience verification | ✓ | | | Kafka Connect infrastructure | ✓ | ✓ | | Kafka Connect tasks state | | ✓ | ### BYOVPC/BYOVNet | Resource | Redpanda responsibility | Customer responsibility | | --- | --- | --- | | Redpanda upgrades and hotfixes | ✓ | | | Cost management and attribution | ✓ | ✓ | | Software vulnerability remediation | ✓ | | | Infrastructure vulnerability remediation | ✓ | ✓ | | IAM (roles, service accounts, access segmentation) | | ✓ | | Compute | ✓ | | | Redpanda agent VM maintenance | ✓ | | | VPC (subnets, routing, firewall) | | ✓ | | VPC peering | | ✓ | | VPC private links (service endpoint) | ✓ | | | VPC private links (consumer endpoint) | | ✓ | | Local storage | ✓ | | | Tiered Storage | | ✓ | | Control plane | ✓ | | | Access controls and audit | ✓ | ✓ | | Managed disaster recovery | | ✓ | | Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) | ✓ | ✓ (for VPC components and cloud storage buckets/containers managed by customer) | | Availability SLA | ✓ (subject to required access to customer resources) | ✓ | | Proactive threat detection | ✓ | ✓ | | Static secret rotation | ✓ | ✓ | | Incident response | ✓ | | | Resilience verification | ✓ | | | Kafka Connect infrastructure | ✓ | ✓ | | Kafka Connect tasks state | | ✓ | ### Dedicated | Resource | Redpanda responsibility | Customer responsibility | | --- | --- | --- | | Redpanda upgrades and hotfixes | ✓ | | | Cost management and attribution | ✓ | | | Software vulnerability remediation | ✓ | | | Infrastructure vulnerability remediation | ✓ | | | IAM (roles, service accounts, access segmentation) | ✓ | | | Compute | ✓ | | | Redpanda agent VM maintenance | ✓ | | | VPC (subnets, routing, firewall) | ✓ | | | VPC peering | ✓ | | | VPC private links (service endpoint) | ✓ | | | VPC private links (consumer endpoint) | | ✓ | | Local storage | ✓ | | | Tiered Storage | ✓ | | | Control plane | ✓ | | | Access controls and audit | ✓ | | | Managed disaster recovery | | ✓ | | Observability and monitoring (SLOs, SLIs, tracing, alerting, runbooks) | ✓ | | | Availability SLA | ✓ | | | Proactive threat detection | ✓ | | | Static secret rotation | ✓ | | | Incident response | ✓ | | | Resilience verification | ✓ | | | Kafka Connect infrastructure | ✓ | | | Kafka Connect tasks state | | ✓ | ## [](#redpanda-connect-and-kafka-connect)Redpanda Connect and Kafka Connect [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) lets you compose pipelines from a rich library of inputs, processors, and outputs with strong metrics, logging, and per-pipeline scaling. To try it, see the [quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/). [Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/) is disabled by default on all new clusters. To unlock this feature for your BYOC or Dedicated cluster, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). When enabled, a Kafka Connect node runs even if no connectors are configured. | | Data transforms | Redpanda Connect | | --- | --- | --- | | Best for | Simple, stateless, per-record normalization inside Redpanda | Enrichment/lookup with external services; multi-stage flows | | External I/O | Not permitted (sandboxed) | Native (HTTP/database/object storage) | | Topology | 1:1 or 1:N (no cross-topic fan-in) | Fan-in and fan-out; multi-step pipelines | | Ordering | Preserves per-partition order | Per-partition order can be preserved; configure parallelism and batching accordingly | | Scale & isolation | Shares broker CPU/memory; best for lightweight operations | Scales independently; isolates heavy work from brokers | | Failure handling | You code routing/error behavior | Built-in retries/backoff and DLQ patterns | > 💡 **TIP** > > - Use data transforms for simple, in-broker, per-record changes with minimal latency. > > - Use Redpanda Connect if your pipeline must talk to external systems (HTTP services, databases, cloud storage), or when you need advanced flow control, such as batching and windowed processing. ### [](#redpanda-connect-vs-data-transforms)Redpanda Connect vs data transforms [Data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/) (Wasm) provide lightweight, per-record changes between Redpanda topics with minimal latency. Transforms run inside the broker, map one input topic to one or more output topics, and are intentionally sandboxed (no external network or disk access). They’re ideal for validation, redaction, format/schema conversion, and simple routing. ## [](#redpanda-cloud-vs-self-managed-feature-compatibility)Redpanda Cloud vs Self-Managed feature compatibility Because Redpanda Cloud is a fully-managed service that provides maintenance, data and partition balancing, upgrades, and recovery, much of the cluster maintenance required for Self-Managed users is not necessary for Redpanda Cloud users. Also, Redpanda Cloud is opinionated about Kafka configurations. For example, automatic topic creation is disabled. Some systems expect the Kafka service to automatically create topics when a message is produced to a topic that doesn’t exist. (You can enable this for BYOC and Dedicated clusters with the `auto_create_topics_enabled` cluster property.) New clusters in Redpanda Cloud generally include functionality added in Self-Managed versions immediately. Existing clusters include new functionality when they get upgraded to the latest version. Redpanda Cloud deployments do not support the following functionality available in Redpanda Self-Managed deployments: - Kafka API OIDC authentication. However, Redpanda Cloud does support [SSO to the Redpanda Cloud UI](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#single-sign-on). - Admin API. - FIPS-compliance mode. - Kerberos authentication. - Redpanda debug bundles. - Redpanda Console topic documentation. - Manual deserialization of Schema Registry - Configuring access to object storage with customer-managed encryption key. - Kubernetes Helm chart and Redpanda Operator functionality. - The following `rpk` commands: - `rpk cluster health` - `rpk cluster license` - `rpk cluster maintenance` - `rpk cluster partitions` - `rpk cluster self-test` - `rpk cluster storage restore` (But `rpk cluster storage` and subcommands for mountable topics are supported in BYOC and Dedicated clusters) - `rpk connect` - `rpk container` - `rpk debug` - `rpk generate app` (This is supported in Serverless clusters only.) - `rpk iotune` - `rpk redpanda` - `rpk topic describe-storage` (All other `rpk topic` commands are supported on both Redpanda Cloud and Self Managed.) > 📝 **NOTE** > > The `rpk cloud` commands are not supported in Self-Managed deployments. ## [](#suggested-videos)Suggested videos - [YouTube - What is Redpanda BYOC? (3 mins)](https://www.youtube.com/watch?v=gVlzsJAYT64&ab_channel=RedpandaData) ## [](#next-steps)Next steps - [Learn about upgrades and maintenance](https://docs.redpanda.com/cloud-data-platform/manage/maintenance/) - [Create a Serverless cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) - [Create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) --- # Page 562: Redpanda Cloud Deployment **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types.md --- # Redpanda Cloud Deployment > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud Deployment latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/index.adoc description: Learn about Redpanda Cloud deployments. page-git-created-date: "2024-06-06" page-git-modified-date: "2024-08-01" --- - [Serverless](serverless/) Learn how to create a Serverless cluster and start streaming. - [BYOC](byoc/) Learn how to create a Bring Your Own Cloud (BYOC), Bring Your Own Virtual Private Cloud (BYOVPC), or Bring Your Own Virtual Network (BYOVNet) cluster. - [Dedicated](create-dedicated-cloud-cluster/) Learn how to create a Dedicated cluster and start streaming. --- # Page 563: BYOC **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc.md --- # BYOC > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: BYOC latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/index.adoc description: Learn how to create a Bring Your Own Cloud (BYOC), Bring Your Own Virtual Private Cloud (BYOVPC), or Bring Your Own Virtual Network (BYOVNet) cluster. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-16" --- Bring Your Own Cloud (BYOC) lets you run Redpanda in your own cloud environment while using managed services provided by Redpanda. With BYOC clusters, Redpanda deploys into your existing cloud network: - AWS and GCP: Virtual Private Cloud (VPC) - Azure: Virtual Network (VNet) Your data never leaves your environment, giving you extra security and control. See [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/) for details. Redpanda manages provisioning, monitoring, upgrades, and security policies, and it manages required resources in your VPC or VNet, including subnets (subnetworks in GCP), IAM roles, and object storage resources (for example, S3 buckets or Azure Storage accounts). You get hands-off operations with a 99.99% uptime guarantee while keeping full control of your data. If you want to manage the networking infrastructure yourself, create a Bring Your Own Virtual Private Cloud (BYOVPC) or Bring Your Own Virtual Network (BYOVNet) cluster. With BYOVPC/BYOVNet, the Redpanda agent does not create or change resources in your account. This is ideal for organizations with stringent compliance requirements or existing network configurations, when you need full control over the network lifecycle. Compared to standard BYOC, BYOVPC/BYOVNet provides more security, but the configuration is more complex. See the [shared responsibility model](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/#shared-responsibility-model) to understand what you manage versus what Redpanda manages. > ❗ **IMPORTANT** > > Don’t deploy other workloads on the BYOC infrastructure that Redpanda manages. - [BYOC: AWS](aws/) Learn how to create a BYOC or BYOVPC cluster on AWS. - [BYOC: Azure](azure/) Learn how to create a BYOC or BYOVNet cluster on Azure. - [BYOC: GCP](gcp/) Learn how to create a BYOC or BYOVPC cluster on GCP. - [Create Remote Read Replicas](remote-read-replicas/) Learn how to create a remote read replica topic with BYOC, which is a read-only topic that mirrors a topic on a different cluster. --- # Page 564: BYOC: AWS **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws.md --- # BYOC: AWS > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: "BYOC: AWS" latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/aws/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/aws/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/aws/index.adoc description: Learn how to create a BYOC or BYOVPC cluster on AWS. page-git-created-date: "2024-10-24" page-git-modified-date: "2025-05-07" --- - [Create a BYOC Cluster on AWS](create-byoc-cluster-aws/) Use the Redpanda Cloud UI to create a BYOC cluster on AWS. - [Create a BYOVPC Cluster on AWS](vpc-byo-aws/) Use the Redpanda BYOVPC Terraform module to deploy a BYOVPC cluster on AWS. --- # Page 565: Create a BYOC Cluster on AWS **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/create-byoc-cluster-aws.md --- # Create a BYOC Cluster on AWS > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOC Cluster on AWS latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/aws/create-byoc-cluster-aws page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/aws/create-byoc-cluster-aws.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc description: Use the Redpanda Cloud UI to create a BYOC cluster on AWS. page-git-created-date: "2024-10-24" page-git-modified-date: "2026-05-21" --- To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). > 📝 **NOTE** > > With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a [BYOVPC cluster on AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/). ## [](#prerequisites)Prerequisites Before you deploy a BYOC cluster on AWS, check that the user creating the cluster has the following prerequisites: - A minimum version of Redpanda `rpk` v24.1. See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - The user authenticating to AWS has `AWSAdministratorAccess` access to create the IAM policies specified in [AWS IAM policies](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies/). - The user has the AWS variables necessary to authenticate. Use either: - `AWS_PROFILE` or - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` To verify access, you should be able to successfully run `aws sts get-caller-identity` for your region. For more information, see the [AWS CLI reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sts/get-caller-identity.html). ## [](#create-a-byoc-cluster)Create a BYOC cluster 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. On the Clusters page, click **Create cluster**, then click **Create** for BYOC. 3. Enter a cluster name, then select the resource group, provider (AWS), [region, tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/), availability, and Redpanda version. > 📝 **NOTE** > > - If you plan to create a private network in your own VPC, select the region where your VPC is located. > > - Three availability zones provide two backups in case one availability zone goes down. Optionally, click **Advanced settings** to specify up to five key-value custom tags. After the cluster is created, the tags are applied to all AWS resources associated with this cluster. For more information, see the [AWS documentation](https://docs.aws.amazon.com/mediaconnect/latest/ug/tagging-restrictions.html). After the cluster is created, you can [specify more tags with the Cloud API](#manage-custom-tags). 4. Click **Next**. 5. On the Network page, select the connection type: either public or private. For BYOC clusters, private is best-practice. - Your network name is used to identify this network. - For a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/), choose one that does not overlap with your existing VPCs or your Redpanda network. - Clusters with private networking include a setting for API Gateway network access. Public access exposes endpoints for Redpanda Console, the Data Plane API, and the MCP Server API, but they remain protected by your authentication and authorization controls. Private access restricts endpoint access to your VPC only. > 📝 **NOTE** > > After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. > 💡 **TIP** > > To route all cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway, set the **Transit Gateway ID** field on this page. The field is only available on clusters with a private connection type, and is only visible if centralized egress is enabled for your organization. This option is in beta. See [Configure Centralized Egress with AWS Transit Gateway](https://docs.redpanda.com/cloud-data-platform/networking/byoc/aws/nat-free-egress/). 6. Click **Next**. 7. On the Deploy page, follow the steps to log in to Redpanda Cloud and deploy the agent. As part of agent deployment: - Redpanda assigns the permission required to run the agent. For details about these permissions, see [AWS IAM policies](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies/). - Redpanda allocates one Elastic IP (EIP) address in AWS for each BYOC cluster. > 📝 **NOTE** > > Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. ## [](#manage-custom-tags)Manage custom tags Your organization might require custom tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage tags with the [Cloud Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/). The Control Plane API allows up to 16 custom tags in AWS. Make sure you have: - The cluster ID. You can find this in the Redpanda Cloud UI, in the **Details** section of the cluster overview. - A valid bearer token for the Cloud Control Plane API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). Then complete the following steps: 1. To refresh agent permissions so the Redpanda agent can update tags, run: ```bash export CLUSTER_ID="" rpk cloud byoc aws apply --redpanda-id="$CLUSTER_ID" ``` This step is required because tag management requires additional IAM permissions that may not have been granted during initial cluster creation: - `ec2:DescribeTags` - `ec2:DescribeVolumes` - `ec2:DescribeNetworkInterfaces` - `ec2:CreateTags` - `ec2:DeleteTags` - `iam:TagPolicy` - `iam:UntagPolicy` - `iam:TagInstanceProfile` - `iam:UntagInstanceProfile` 2. To update tags, invoke the Cloud API. First, set your authentication token: ```bash export AUTH_TOKEN="" ``` The `PATCH` call sets the tags specified under `"cloud_provider_tags"`. It replaces the existing tags with the specified tags. Include all desired tags in the request. To remove a single entry, omit it from the map you send. ```bash cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "Environment": "production", "CostCenter": "engineering" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` To remove all tags, send an empty `cloud_provider_tags` object: ```bash cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` ## [](#next-steps)Next steps [Configure private networking](https://docs.redpanda.com/cloud-data-platform/networking/byoc/aws/) --- # Page 566: Create a BYOVPC Cluster on AWS **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws.md --- # Create a BYOVPC Cluster on AWS > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOVPC Cluster on AWS latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/aws/vpc-byo-aws page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/aws/vpc-byo-aws.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc description: Use the Redpanda BYOVPC Terraform module to deploy a BYOVPC cluster on AWS. page-topic-type: how-to personas: platform_admin learning-objective-1: Deploy a BYOVPC cluster on AWS using the Redpanda Terraform module learning-objective-2: Configure the Redpanda network and cluster resources using module outputs learning-objective-3: Enable PrivateLink on a BYOVPC cluster page-git-created-date: "2024-12-02" page-git-modified-date: "2026-08-03" --- > ❗ **IMPORTANT** > > BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or [Redpanda Sales](https://www.redpanda.com/price-estimator). A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda [data plane](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#data-plane) into your existing VPC and manage the networking lifecycle yourself. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more control. For background on the architecture, see [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). When you create a BYOVPC cluster, you specify your VPC and the IAM role (instance profile) that the Redpanda agent will assume. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With BYOVPC: - You provide your own VPC in your AWS account. - You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters. - You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets. The [Redpanda BYOVPC Terraform Module](https://registry.terraform.io/modules/redpanda-data/redpanda-byovpc/aws/latest) contains [Terraform](https://developer.hashicorp.com/terraform) code that deploys the resources required for a BYOVPC cluster on AWS. You need to create these resources in advance and provide them to Redpanda during cluster creation. Variables are provided in the code so you can exclude resources that already exist in your environment, such as the VPC. > 📝 **NOTE** > > Secrets management is enabled by default with the Terraform module. It allows you to store and read secrets in your cluster, for example to integrate a REST catalog with Iceberg-enabled topics. > > For existing BYOVPC clusters, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new) to enable secrets management. ## [](#prerequisites)Prerequisites - Access to an AWS account in which you create your cluster. - Minimum permissions in that AWS account. For the actions required by the user who will create the cluster with `terraform apply`, see [`iam_rpk_user.tf`](https://github.com/redpanda-data/terraform-aws-redpanda-byovpc/blob/main/iam_rpk_user.tf). - Each BYOVPC cluster requires one allocated Elastic IP (EIP) address in AWS. - [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) version 1.8.5 or later. - The [Redpanda Terraform provider](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs) configured with valid credentials. For setup details, see the provider documentation. ## [](#limitations)Limitations - Existing clusters cannot be converted to BYOVPC clusters. - After creating a BYOVPC cluster, you cannot change to a different VPC. - Only primary CIDR ranges are supported for the VPC. > 📝 **NOTE** > > For simplicity, the instructions are based on the assumption that Terraform is configured to use local state. You may want to configure [remote state](https://developer.hashicorp.com/terraform/language/state/remote). ## [](#configure-the-redpanda-byovpc-terraform-module)Configure the Redpanda BYOVPC Terraform module The following example uses the Redpanda BYOVPC Terraform Module to create the resources required to create a BYOVPC cluster. > 📝 **NOTE** > > Redpanda recommends using a VPC in AWS with a CIDR block (10.0.0.0/16) to allow for enough address space. The subnets must be set to /24. ```hcl locals { common_prefix = "abc-stg" region = "us-east-2" zones = ["use2-az1", "use2-az2", "use2-az3"] enable_private_link = false # see example below for enabling private link force_destroy_cloud_storage = false # see example below if using pre-existing VPC and subnets, # otherwise when provided with these cidrs the module will # attempt to create the VPC and subnets vpc_cidr_block = "10.0.0.0/16" public_subnet_cidrs = [ "10.0.1.0/24", "10.0.3.0/24", "10.0.5.0/24", "10.0.7.0/24", "10.0.9.0/24", "10.0.11.0/24" ] private_subnet_cidrs = [ "10.0.0.0/24", "10.0.2.0/24", "10.0.4.0/24", "10.0.6.0/24", "10.0.8.0/24", "10.0.10.0/24" ] # condition_tags restrict the IAM permissions granted by the # module to only those resources with these tags, when using # condition_tags these tags must also be provided to the # redpanda_cluster so that all resources created are given # these tags condition_tags = { "redpanda-managed" : "true" } # default_tags are applied to all resources created by the # module or redpanda_cluster resource default_tags = { "env" : "staging" } # when using a brand new AWS account that has never hosted an # EKS cluster before the EKS node group service linked role # must be created, if it already exists this may be set to false create_eks_nodegroup_service_linked_role = true } module "redpanda_byovpc" { source = "redpanda-data/redpanda-byovpc/aws" common_prefix = local.common_prefix region = local.region zones = local.zones create_rpk_user = false enable_private_link = local.enable_private_link force_destroy_cloud_storage = local.force_destroy_cloud_storage enable_redpanda_connect = true vpc_cidr_block = local.vpc_cidr_block private_subnet_cidrs = local.private_subnet_cidrs public_subnet_cidrs = local.public_subnet_cidrs condition_tags = local.condition_tags default_tags = local.default_tags create_eks_nodegroup_service_linked_role = local.create_eks_nodegroup_service_linked_role } ``` > 📝 **NOTE** > > - To send telemetry back to the Redpanda control plane, the cluster needs outbound internet access. You can provide this through at least one public subnet, or through network peering or a transit gateway to another VPC that routes traffic through a public subnet. The example configuration includes multiple public subnets to allow for future scaling. Standard BYOC clusters can also route egress through a customer-owned hub VPC and Transit Gateway, eliminating the per-VPC NAT Gateway entirely. See [Configure Centralized Egress with AWS Transit Gateway](https://docs.redpanda.com/cloud-data-platform/networking/byoc/aws/nat-free-egress/). > > - The example creates an Internet Gateway and an associated Route Table rule that routes traffic into the VPC, which allows the Redpanda control plane to access the cluster. To disable creation of the Internet Gateway, either remove the configuration and value for `create_internet_gateway` or set `"create_internet_gateway": false`. > > - When using a pre-existing VPC, at least one public subnet must already exist in that VPC. Setting `public_subnet_cidrs = []` only prevents the module from creating new ones. > 💡 **TIP** > > See the full list of zones and tiers available with each provider in the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers). ## [](#configure-the-redpanda-network-and-cluster)Configure the Redpanda network and cluster After provisioning the AWS infrastructure, configure the Redpanda network and cluster resources using the module outputs. ```hcl locals { resource_group_name = "staging" throughput_tier = "tier-1-aws-v3-arm" } data "redpanda_resource_group" "staging" { name = local.resource_group_name } resource "redpanda_network" "network" { name = "${local.common_prefix}-network" resource_group_id = data.redpanda_resource_group.staging.id cloud_provider = "aws" region = local.region cluster_type = "byoc" customer_managed_resources = { aws = { management_bucket = { arn = module.redpanda_byovpc.management_bucket_arn } dynamodb_table = { arn = module.redpanda_byovpc.dynamodb_table_arn } vpc = { arn = module.redpanda_byovpc.vpc_arn } private_subnets = { arns = module.redpanda_byovpc.private_subnet_arns } } } depends_on = [module.redpanda_byovpc] } resource "redpanda_cluster" "cluster" { name = "${local.common_prefix}-cluster" resource_group_id = data.redpanda_resource_group.staging.id cloud_provider = "aws" region = redpanda_network.network.region zones = local.zones network_id = redpanda_network.network.id cluster_type = "byoc" connection_type = "private" throughput_tier = local.throughput_tier allow_deletion = false tags = merge(local.condition_tags, local.default_tags) customer_managed_resources = { aws = { agent_instance_profile = { arn = module.redpanda_byovpc.agent_instance_profile_arn } cloud_storage_bucket = { arn = module.redpanda_byovpc.cloud_storage_bucket_arn } cluster_security_group = { arn = module.redpanda_byovpc.cluster_security_group_arn } connectors_node_group_instance_profile = { arn = module.redpanda_byovpc.connectors_node_group_instance_profile_arn } connectors_security_group = { arn = module.redpanda_byovpc.connectors_security_group_arn } k8s_cluster_role = { arn = module.redpanda_byovpc.k8s_cluster_role_arn } node_security_group = { arn = module.redpanda_byovpc.node_security_group_arn } permissions_boundary_policy = { arn = module.redpanda_byovpc.permissions_boundary_policy_arn } redpanda_agent_security_group = { arn = module.redpanda_byovpc.redpanda_agent_security_group_arn } redpanda_node_group_instance_profile = { arn = module.redpanda_byovpc.redpanda_node_group_instance_profile_arn } redpanda_node_group_security_group = { arn = module.redpanda_byovpc.redpanda_node_group_security_group_arn } utility_node_group_instance_profile = { arn = module.redpanda_byovpc.utility_node_group_instance_profile_arn } utility_security_group = { arn = module.redpanda_byovpc.utility_security_group_arn } redpanda_connect_node_group_instance_profile = { arn = module.redpanda_byovpc.redpanda_connect_node_group_instance_profile_arn } redpanda_connect_security_group = { arn = module.redpanda_byovpc.redpanda_connect_security_group_arn } } } depends_on = [redpanda_network.network] } ``` ## [](#apply-the-terraform-configuration)Apply the Terraform configuration Initialize, plan, and apply Terraform to set up the AWS infrastructure: ```bash terraform init && terraform plan && terraform apply ``` Cluster provisioning can take up to 45 minutes. When provisioning completes, the cluster status updates to `Running`. If the cluster stays in `Creating` status, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). ### [](#validation-checks)Validation checks The `redpanda_cluster` resource performs validation checks before proceeding with provisioning: - RPK user: Checks if the user running the command has sufficient privileges to provision the agent. Any missing permissions are displayed in the output. - IAM instance profile: Checks that `agent_instance_profile`, `connectors_node_group_instance_profile`, `redpanda_node_group_instance_profile`, `redpanda_connect_node_group_instance_profile`, `utility_node_group_instance_profile`, and `k8s_cluster_role` have the minimum required permissions. Any missing permissions are displayed in the output. - Storage: Checks that the `management_bucket` exists and is versioned, checks that the `cloud_storage_bucket` exists and is not versioned, and checks that the `dynamodb_table` exists. - Network: Checks that the VPC exists, checks that the subnets exist and have the expected tags, and checks that the security groups exist and have the desired ingress and egress rules. ## [](#delete-the-cluster)Delete the cluster To delete the cluster and all associated resources, run `terraform destroy`. > ⚠️ **WARNING** > > This also deletes the customer-managed resources created by the module. ```bash terraform destroy ``` ## [](#enable-privatelink)Enable PrivateLink PrivateLink can be enabled during cluster creation or on an already existing cluster. Start by enabling PrivateLink in the Redpanda BYOVPC Terraform module. This adds the permissions required for PrivateLink. ```hcl module "redpanda_byovpc" { # ... enable_private_link = true # ... } ``` Enable PrivateLink on the `redpanda_cluster` resource: ```hcl resource "redpanda_cluster" "cluster" { # ... aws_private_link = { allowed_principals = ["arn:aws:iam::${var.aws_account_id}:root"] enabled = true connect_console = false } # ... } ``` ## [](#deploy-with-pre-existing-vpc-and-subnets)Deploy with pre-existing VPC and subnets If you already have a VPC and subnets in your AWS account, provide their IDs to the module instead of CIDR blocks. ```hcl module "redpanda_byovpc" { # ... # vpc_cidr_block = local.vpc_cidr_block # private_subnet_cidrs = local.private_subnet_cidrs # public_subnet_cidrs = local.public_subnet_cidrs vpc_id = "vpc-0c79b236047faa1ab" private_subnet_ids = [ "subnet-0e58df59b5eb037c3", "subnet-0c74559ab372f5123", "subnet-0525df35c467cad1c", "subnet-09c301e004e96c803", "subnet-0f67e76738572cb8e", "subnet-0cca6892cf789f6ec", ] public_subnet_cidrs = [] # when empty the module will not create any public subnets # ... } ``` ## [](#next-steps)Next steps - [Configure AWS PrivateLink](https://docs.redpanda.com/cloud-data-platform/networking/aws-privatelink/) - [Review AWS IAM policies](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies/) - [Learn about `rpk` commands](https://docs.redpanda.com/cloud-data-platform/reference/rpk/) - [Enable Redpanda SQL on a BYOVPC Cluster on AWS](https://docs.redpanda.com/cloud-data-platform/sql/get-started/enable-sql-byovpc-aws/) --- # Page 567: BYOC: Azure **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure.md --- # BYOC: Azure > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: "BYOC: Azure" latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/azure/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/azure/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/azure/index.adoc description: Learn how to create a BYOC or BYOVNet cluster on Azure. page-git-created-date: "2024-10-24" page-git-modified-date: "2025-07-30" --- - [Create a BYOC Cluster on Azure](create-byoc-cluster-azure/) Use the Redpanda Cloud UI to create a BYOC cluster on Azure. - [Create a BYOVNet Cluster on Azure](vnet-azure/) Use Terraform to deploy a BYOVNet cluster on Azure. --- # Page 568: Create a BYOC Cluster on Azure **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/create-byoc-cluster-azure.md --- # Create a BYOC Cluster on Azure > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOC Cluster on Azure latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/azure/create-byoc-cluster-azure page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/azure/create-byoc-cluster-azure.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/azure/create-byoc-cluster-azure.adoc description: Use the Redpanda Cloud UI to create a BYOC cluster on Azure. page-git-created-date: "2024-10-24" page-git-modified-date: "2026-07-30" --- To create a Redpanda cluster in your virtual network (VNet), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). > 📝 **NOTE** > > With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the most security, you can manage these resources yourself with a [BYOVNet cluster on Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/vnet-azure/). ## [](#prerequisites)Prerequisites Before you deploy a BYOC cluster on Azure, check all prerequisites to ensure that your Azure subscription meets requirements. ### [](#configure-azure-cli)Configure Azure CLI - [Install the Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). - [Sign in](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli) with the Azure CLI: ```none az login ``` - Set the desired subscription for the Azure CLI: ```none az account set --subscription ``` ### [](#verify-rpk-version)Verify rpk version Confirm you have a minimum version of Redpanda `rpk` v24.1. See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). ### [](#prepare-your-azure-subscription)Prepare your Azure subscription In the [Azure Portal](https://login.microsoftonline.com/), confirm that the dedicated subscription you intend to use with Redpanda includes the following: - **Role**: The Azure user must have the _Owner_ role in the subscription. - **Resources**: The subscription must be registered for the following resource providers (AKS + common dependencies). See the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types). - Microsoft.Compute - Microsoft.ManagedIdentity - Microsoft.Storage - Microsoft.KeyVault - Microsoft.Network - Microsoft.ContainerService To check if a resource provider is registered, run the following command using the Azure CLI or in the Azure Cloud Shell: ```none az provider show -n Microsoft.Compute --query registrationState -o tsv az provider show -n Microsoft.ManagedIdentity --query registrationState -o tsv az provider show -n Microsoft.Storage --query registrationState -o tsv az provider show -n Microsoft.KeyVault --query registrationState -o tsv az provider show -n Microsoft.Network --query registrationState -o tsv az provider show -n Microsoft.ContainerService --query registrationState -o tsv ``` If a resource provider is not registered, run: ```none az provider register --namespace Microsoft.Compute az provider register --namespace Microsoft.ManagedIdentity az provider register --namespace Microsoft.Storage az provider register --namespace Microsoft.KeyVault az provider register --namespace Microsoft.Network az provider register --namespace Microsoft.ContainerService ``` - **Feature**: The subscription must be registered for Microsoft.Compute/EncryptionAtHost. See the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/disks-enable-host-based-encryption-cli#prerequisites). To register it, run: ```none az feature register --namespace Microsoft.Compute --name EncryptionAtHost # (optional) Wait and verify it shows as Registered az feature show --namespace Microsoft.Compute --name EncryptionAtHost --query properties.state -o tsv # Refresh the provider after enabling a feature az provider register --namespace Microsoft.Compute ``` - **Monitoring**: The subscription must have Azure Network Watcher enabled in the NetworkWatcherRG resource group and the region where you will use Redpanda. Network Watcher lets you monitor and diagnose conditions at a network level. See the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-create?tabs=portaly). To enable it, run: ```none # Create the NetworkWatcherRG resource group az group create --name 'NetworkWatcherRG' --location '' # Enable Network Watcher in az network watcher configure --resource-group 'NetworkWatcherRG' --locations '' --enabled ``` ### [](#check-azure-quota)Check Azure quota Confirm that the Azure subscription has enough virtual CPUs (vCPUs) per instance family and total regional vCPUs in the region where you will use Redpanda: - Standard Ddv5-series vCPUs: 12 (3 Redpanda broker nodes + extra capacity for 3 more nodes that could be utilized temporarily during tier 1 maintenance) - Standard Dadsv5-series vCPUs: 8 (2 Redpanda utility nodes) - Standard Dv3-series vCPUs: 2 (1 Redpanda agent node) See the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/quotas/view-quotas). ### [](#check-azure-sku-restrictions)Check Azure SKU restrictions Ensure your subscription has access to the required VM sizes in the region where you will use Redpanda. For example, using the Azure CLI or in the Azure Cloud Shell, run: ```bash # Replace eastus2 with your target region az vm list-skus -l eastus2 --zone --size Standard_D2d_v5 --output table ``` Example output (no restrictions: good) ```bash ResourceType Locations Name Zones Restrictions --------------- ----------- --------------- ------- ------------ virtualMachines eastus2 Standard_D2d_v5 1,2,3 None ``` Example output (with restrictions: needs attention) ```bash ResourceType Locations Name Zones Restrictions --------------- ----------- --------------- ------- ------------ virtualMachines eastus2 Standard_D2d_v5 1,2,3 NotAvailableForSubscription ``` If you see restrictions, [open a Microsoft support request](https://learn.microsoft.com/en-us/troubleshoot/azure/general/region-access-request-process) to remove them. ### [](#prerequisite-checklist)Prerequisite checklist - Verified `rpk` version - Verified Azure user has Owner role - Registered all required resource providers - Registered EncryptionAtHost feature - Enabled Network Watcher - Verified vCPU quota - Verified no SKU restrictions ## [](#create-a-byoc-cluster)Create a BYOC cluster To create a Redpanda cluster in your Azure VNet, follow the [prerequisites](#prerequisites) then follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. On the Clusters page, click **Create cluster**, then click **Create** for BYOC. 3. Enter a cluster name, then select the resource group, provider (Azure), [region, tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/), availability, and Redpanda version. > 📝 **NOTE** > > - If you plan to create a private network in your own VNet, select the region where your VNet is located. > > - Multi-AZ is the default configuration. Three AZs provide two backups in case one availability zone goes down. Optionally, click **Advanced settings** to specify up to five key-value custom tags. After the cluster is created, the tags are applied to all Azure resources associated with this cluster. For details, see the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/tag-resources). After the cluster is created, you can [specify more tags with the Cloud API](#manage-custom-tags). 4. Click **Next**. 5. On the Network page, select the connection type: either public or private. For BYOC clusters, private using Azure Private Link is best-practice. - Your network name is used to identify this network. - For a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/), choose one that does not overlap with your existing VPCs or your Redpanda network. - Clusters with private networking include a setting for API Gateway network access. Public access exposes endpoints for Redpanda Console, the Data Plane API, and the MCP Server API, but they remain protected by your authentication and authorization controls. Private access restricts endpoint access to your VNet only. Private access incurs an additional cost, since it involves deploying two network load balancers, instead of one. > 📝 **NOTE** > > After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. > 💡 **TIP** > > To route all cluster egress through your own Azure Firewall and hub VNet instead of a per-cluster NAT Gateway, enter the **Hub egress VNet ID** and **Firewall private IP** on this page, or set `egress_spec.azure.hub_vnet_id` and `egress_spec.azure.firewall_private_ip` when you create the network with the Cloud API. This option is only available on clusters with a private connection type and a Redpanda-managed VNet, and only if centralized egress is enabled for your organization. This option is in beta. See [Configure Centralized Egress with Azure Firewall](https://docs.redpanda.com/cloud-data-platform/networking/byoc/azure/nat-free-egress/). 6. Click **Next**. 7. On the Deploy page, follow the steps to log in to Redpanda Cloud and deploy the agent. As part of agent deployment, Redpanda assigns the permissions required to run the agent. For details about these permissions, see [Azure IAM policies](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies-azure/). ## [](#manage-custom-tags)Manage custom tags Your organization might require custom tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage tags with the [Cloud Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/). The Control Plane API allows up to 16 custom tags in Azure. Make sure you have: - The cluster ID. You can find this in the Redpanda Cloud UI, in the **Details** section of the cluster overview. - A valid bearer token for the Cloud Control Plane API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). Then complete the following steps: 1. To refresh Redpanda agent permissions in the target subscription, run: ```bash export CLUSTER_ID="" export SUBSCRIPTION_ID="" rpk cloud byoc azure apply --redpanda-id="$CLUSTER_ID" --subscription-id="$SUBSCRIPTION_ID" ``` 2. To update tags, invoke the Cloud API. First, set your authentication token: ```bash export AUTH_TOKEN="" ``` The `PATCH` call sets the tags specified under `"cloud_provider_tags"`. It replaces the existing tags with the specified tags. Include all desired tags in the request. To remove a single entry, omit it from the map you send. ```bash cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "Environment": "production", "CostCenter": "engineering" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` To remove all tags, send an empty `cloud_provider_tags` object: ```bash cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` ### [](#limitations)Limitations - Nodepool Application Security Groups (ASG): Custom tags are set only when the cluster is created. Tags cannot be updated on these resources after cluster creation. - Private Link network interfaces (Kubernetes API server, Tiered Storage, and Private Link service): Custom tags are set only during cluster creation and cannot be changed later. --- # Page 569: Create a BYOVNet Cluster on Azure **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/vnet-azure.md --- # Create a BYOVNet Cluster on Azure > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOVNet Cluster on Azure latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/azure/vnet-azure page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/azure/vnet-azure.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/azure/vnet-azure.adoc description: Use Terraform to deploy a BYOVNet cluster on Azure. page-topic-type: how-to personas: platform_admin learning-objective-1: Deploy a BYOVNet cluster on Azure using Terraform learning-objective-2: Configure the Redpanda network and cluster resources using the Cloud API learning-objective-3: Manage the lifecycle of a BYOVNet cluster, including creation and deletion page-git-created-date: "2024-11-15" page-git-modified-date: "2026-08-13" --- > ❗ **IMPORTANT** > > BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or [Redpanda Sales](https://www.redpanda.com/price-estimator). A Bring Your Own Virtual Network (BYOVNet) cluster allows you to deploy the Redpanda [data plane](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#data-plane) into your existing VNet and manage the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVNet provides more control. For background on the architecture, see [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). When you create a BYOVNet cluster, you specify your VNet and managed identities. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With a customer-managed VNet: - You provide your own VNet in your Azure account. - You maintain more control over your account, because Redpanda requires fewer permissions than standard BYOC clusters. - You control your security resources and policies, including subnets, user-assigned identities, IAM roles and assignments, security groups, storage accounts, and key vaults. The [Redpanda Cloud Examples repository](https://github.com/redpanda-data/cloud-examples/tree/main/customer-managed/azure/README.md) contains [Terraform](https://developer.hashicorp.com/terraform) code that deploys the resources required for a BYOVNet cluster on Azure. You need to create these resources in advance and provide them to Redpanda during cluster creation. Variables are provided in the code so you can exclude resources that already exist in your environment, such as the VNet. See the code for the complete list of resources required to create and deploy a Redpanda cluster. Customer-managed resources can be broken down into the following groups: - Resource group resources - User-assigned identities - IAM roles and assignments - Network - Storage - Key vaults ## [](#prerequisites)Prerequisites - Access to an Azure subscription where you want to create your cluster - Knowledge of your internal VNet and subnet configuration - Permission to call the [Redpanda Cloud API](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview) - Permission to create, modify, and delete the resources described by Terraform - [Terraform](https://developer.hashicorp.com/terraform/install) version 1.8.5 or later - [jq](https://jqlang.org/download/), which is used to parse JSON values from API responses ## [](#limitations)Limitations - Existing clusters cannot be moved to a BYOVNet cluster. - After creating a BYOVNet cluster, you cannot change to a different VNet. - Only primary CIDR ranges are supported for the VNet. ## [](#set-environment-variables)Set environment variables Set environment variables for the resource group, VNet name, and Azure region. For example: ```bash export AZURE_RESOURCE_GROUP_NAME=sample-redpanda-rg export AZURE_VNET_NAME="sample-vnet" export AZURE_REGION=centralus ``` ## [](#create-azure-resource-group-and-vnet)Create Azure resource group and VNet 1. Create a resource group to contain all resources, and then create a VNet with your address and subnet prefixes. The following example uses the environment variables to create the `sample-redpanda-rg` resource group and the `sample-vnet` virtual network with an address space of `10.0.0.0/16`. ```bash az group create --name ${AZURE_RESOURCE_GROUP_NAME} --location ${AZURE_REGION} az network vnet create \ --name ${AZURE_VNET_NAME} \ --resource-group ${AZURE_RESOURCE_GROUP_NAME} \ --location ${AZURE_REGION} \ --address-prefix 10.0.0.0/16 ``` 2. Set additional environment variables for Azure resources. For example: ```bash export AZURE_SUBSCRIPTION_ID= export AZURE_TENANT_ID= export AZURE_ZONES='["centralus-az1", "centralus-az2", "centralus-az3"]' export AZURE_RESOURCE_PREFIX=sample- export REDPANDA_CLUSTER_NAME= export REDPANDA_RG_ID= export REDPANDA_THROUGHPUT_TIER=tier-1-azure-v3-x86 export REDPANDA_VERSION=25.2 export REDPANDA_MANAGEMENT_STORAGE_ACCOUNT_NAME=rpmgmtsa export REDPANDA_MANAGEMENT_STORAGE_CONTAINER_NAME=rpmgmtsc export REDPANDA_0_PODS_SUBNET_NAME=snet-rp-0-pods export REDPANDA_0_VNET_SUBNET_NAME=snet-rp-0-vnet export REDPANDA_1_PODS_SUBNET_NAME=snet-rp-1-pods export REDPANDA_1_VNET_SUBNET_NAME=snet-rp-1-vnet export REDPANDA_2_PODS_SUBNET_NAME=snet-rp-2-pods export REDPANDA_2_VNET_SUBNET_NAME=snet-rp-2-vnet export REDPANDA_CONNECT_PODS_SUBNET_NAME=snet-connect-pods export REDPANDA_CONNECT_VNET_SUBNET_NAME=snet-connect-vnet export KAFKA_CONNECT_PODS_SUBNET_NAME=snet-kafka-connect-pods export KAFKA_CONNECT_VNET_SUBNET_NAME=snet-kafka-connect-vnet export SYSTEM_PODS_SUBNET_NAME=snet-system-pods export SYSTEM_VNET_SUBNET_NAME=snet-system-vnet export REDPANDA_AGENT_SUBNET_NAME=snet-agent-private export REDPANDA_EGRESS_SUBNET_NAME=snet-agent-public export REDPANDA_MANAGEMENT_KEY_VAULT_NAME=redpanda-vault export REDPANDA_CONSOLE_KEY_VAULT_NAME=rp-console-vault export REDPANDA_AKS_SUBNET_CIDR="10.0.15.0/24" export REDPANDA_IAM_RESOURCE_GROUP_NAME=sample-redpanda-rg export REDPANDA_NETWORK_RESOURCE_GROUP_NAME=sample-redpanda-rg export REDPANDA_RESOURCE_GROUP_NAME=sample-redpanda-rg export REDPANDA_STORAGE_RESOURCE_GROUP_NAME=sample-redpanda-rg export REDPANDA_SECURITY_GROUP_NAME=redpanda-nsg export REDPANDA_TIERED_STORAGE_ACCOUNT_NAME=tieredsa export REDPANDA_TIERED_STORAGE_CONTAINER_NAME=tieredsc export REDPANDA_AGENT_USER_ASSIGNED_IDENTITY_NAME=agent-uai export REDPANDA_AKS_USER_ASSIGNED_IDENTITY_NAME=aks-uai export REDPANDA_CERT_MANAGER_USER_ASSIGNED_IDENTITY_NAME=cert-manager-uai export REDPANDA_EXTERNAL_DNS_USER_ASSIGNED_IDENTITY_NAME=external-dns-uai export REDPANDA_CLUSTER_USER_ASSIGNED_IDENTITY_NAME=cluster-uai export REDPANDA_CONSOLE_USER_ASSIGNED_IDENTITY_NAME=console-uai export KAFKA_CONNECT_USER_ASSIGNED_IDENTITY_NAME=kafka-connect-uai export REDPANDA_CONNECT_USER_ASSIGNED_IDENTITY_NAME=redpanda-connect-uai export REDPANDA_CONNECT_API_USER_ASSIGNED_IDENTITY_NAME=redpanda-connect-api-uai export REDPANDA_OPERATOR_USER_ASSIGNED_IDENTITY_NAME=redpanda-operator-uai ``` ## [](#configure-terraform)Configure Terraform > 📝 **NOTE** > > For simplicity, these instructions assume that Terraform is configured to use local state. You may want to configure [remote state](https://developer.hashicorp.com/terraform/language/state/remote). Create a JSON file called `byovnet.auto.tfvars.json` inside the Terraform directory to configure variables for your specific needs: Show script ```bash cat > byovnet.auto.tfvars.json < 💡 **TIP** > > To get the Redpanda authentication credentials, follow the [authentication guide](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/authentication). ## [](#create-the-network)Create the network To create the Redpanda network: 1. Define a JSON file called `redpanda-network.json` to configure the network for Redpanda with details about VNet, subnets, and storage. Show script ```bash cat > redpanda-network.json < redpanda-cluster.json < 💡 **TIP** > > See the full list of zones and tiers available with each provider in the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers). 2. Make a Cloud API call to create a Redpanda cluster and get the network ID from the response in JSON `.operation.metadata.network_id`. ```bash export REDPANDA_ID=$(curl -X POST "https://api.redpanda.com/v1/clusters" \ -H "accept: application/json"\ -H "content-type: application/json" \ -H "authorization: Bearer ${BEARER_TOKEN}" \ --data-binary @redpanda-cluster.json | jq -r '.operation.resource_id') ``` ## [](#create-the-cluster-resources)Create the cluster resources To create the initial cluster resources, first log in to Redpanda Cloud, then run `rpk cloud byoc azure apply`: ```bash rpk cloud login \ --save \ --client-id=${REDPANDA_CLIENT_ID} \ --client-secret=${REDPANDA_CLIENT_SECRET} \ --no-profile ``` ```bash rpk cloud byoc azure apply --redpanda-id="${REDPANDA_ID}" --subscription-id="${AZURE_SUBSCRIPTION_ID}" ``` The Redpanda Cloud agent now is running and handles the remaining steps. This can take up to 45 minutes. When provisioning completes, the cluster status updates to `Running`. If the cluster remains in `Creating` status after 45 minutes, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). ## [](#check-the-cluster-status)Check the cluster status Cluster creation is an example of an operation that can take a longer period of time to complete. You can check the operation state with the Cloud API, or check the Redpanda Cloud UI for cluster status. Example using the returned `operation_id`: ```bash curl -X GET "https://api.redpanda.com/v1/operations/" \ -H "accept: application/json"\ -H "content-type: application/json" \ -H "authorization: Bearer ${BEARER_TOKEN}" ``` Example retrieving cluster: ```bash curl -X GET "https://api.redpanda.com/v1/clusters/" \ -H "accept: application/json"\ -H "content-type: application/json" \ -H "authorization: Bearer ${BEARER_TOKEN}" ``` ## [](#delete-the-cluster)Delete the cluster To delete the cluster, first send a DELETE request to the Cloud API, and retrieve the `resource_id` of the DELETE operation. Then run the `rpk` command to destroy the cluster identified by the `resource_id`. ```bash export REDPANDA_ID=$(curl -X DELETE "https://api.redpanda.com/v1/clusters/${REDPANDA_ID}" \ -H "accept: application/json"\ -H "content-type: application/json" \ -H "authorization: Bearer ${BEARER_TOKEN}" | jq -r '.operation.resource_id') ``` After that completes, run: ```bash rpk cloud byoc azure destroy --redpanda-id ${REDPANDA_ID} ``` > 📝 **NOTE** > > Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. ## [](#manage-custom-tags)Manage custom tags Your organization might require custom tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage tags with the [Cloud Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/). The Control Plane API allows up to 16 custom tags in Azure. Make sure you have: - The cluster ID. You can find this in the Redpanda Cloud UI, in the **Details** section of the cluster overview. - A valid bearer token for the Cloud Control Plane API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). Then complete the following steps: 1. To refresh Redpanda agent permissions in the target subscription, run: ```bash export CLUSTER_ID="" export SUBSCRIPTION_ID="" rpk cloud byoc azure apply --redpanda-id="$CLUSTER_ID" --subscription-id="$SUBSCRIPTION_ID" ``` 2. To update tags, invoke the Cloud API. First, set your authentication token: ```bash export AUTH_TOKEN="" ``` The `PATCH` call sets the tags specified under `"cloud_provider_tags"`. It replaces the existing tags with the specified tags. Include all desired tags in the request. To remove a single entry, omit it from the map you send. ```bash cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "Environment": "production", "CostCenter": "engineering" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` To remove all tags, send an empty `cloud_provider_tags` object: ```bash cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` ### [](#limitations-2)Limitations - Nodepool Application Security Groups (ASG): Custom tags are set only when the cluster is created. Tags cannot be updated on these resources after cluster creation. - Private Link network interfaces (Kubernetes API server, Tiered Storage, and Private Link service): Custom tags are set only during cluster creation and cannot be changed later. > 📝 **NOTE** > > For BYOVNet clusters, custom tags are not applied to the customer-managed resources that are deployed by the customer. ## [](#next-steps)Next steps - [Configure Azure Private Link](https://docs.redpanda.com/cloud-data-platform/networking/azure-private-link/) - [Review Azure IAM policies](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies-azure/) - [Learn about `rpk` commands](https://docs.redpanda.com/cloud-data-platform/reference/rpk/) --- # Page 570: BYOC: GCP **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp.md --- # BYOC: GCP > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: "BYOC: GCP" latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/gcp/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/gcp/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/gcp/index.adoc description: Learn how to create a BYOC or BYOVPC cluster on GCP. page-git-created-date: "2024-10-24" page-git-modified-date: "2025-05-07" --- - [Create a BYOC Cluster on GCP](create-byoc-cluster-gcp/) Use the Redpanda Cloud UI to create a BYOC cluster on GCP. - [Create a BYOVPC Cluster on GCP](vpc-byo-gcp/) Connect Redpanda Cloud to your existing VPC for additional security. - [Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP](enable-rpcn-byovpc-gcp/) Add Redpanda Connect to your existing BYOVPC cluster. - [Enable Secrets Management on an Existing BYOVPC Cluster on GCP](enable-secrets-byovpc-gcp/) Store and read secrets in your existing BYOVPC cluster. --- # Page 571: Create a BYOC Cluster on GCP **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/create-byoc-cluster-gcp.md --- # Create a BYOC Cluster on GCP > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOC Cluster on GCP latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/gcp/create-byoc-cluster-gcp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc description: Use the Redpanda Cloud UI to create a BYOC cluster on GCP. page-git-created-date: "2024-10-24" page-git-modified-date: "2026-07-08" --- To create a Redpanda cluster in your virtual private cloud (VPC), follow the instructions in the Redpanda Cloud UI. The UI contains the parameters necessary to successfully run `rpk cloud byoc apply`. See also: [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). > 📝 **NOTE** > > With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a [BYOVPC cluster on GCP](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp/). If your clients need to connect from different GCP regions than where your cluster will be deployed, you must enable global access during cluster creation using the Cloud API. To create a BYOC cluster with global access enabled, see [Enable Global Access](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/enable-global-access/). ## [](#prerequisites)Prerequisites Before you deploy a BYOC cluster on GCP, verify the following prerequisites: - A minimum version of Redpanda `rpk` v24.1. See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - Assign the `roles/editor` role (or higher, such as `roles/owner`) to the GCP user or service account that runs the bootstrap on the target GCP project. This grants the permissions needed to create VPC networks, GKE clusters, service accounts, and other infrastructure during the initial bootstrap. These bootstrap permissions are separate from the [agent permissions](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies-gcp/) that Redpanda assigns after bootstrap. - The user has the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) installed and authenticated, with the target project selected. To verify, run: ```bash gcloud auth list gcloud config get-value project ``` ### [](#gcp-quotas)GCP quotas Ensure at least three nodes of headroom in the relevant GCP quotas in the same region as your cluster. During maintenance, Redpanda may temporarily create extra nodes. Quotas such as vCPUs per VM family (for example, N2D) and Local SSD total per VM family (quota key: `LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY`) are listed for each tier on the **Create BYOC cluster** page in the Redpanda Cloud UI. Headroom formulas: - vCPU spare = `3 x (vCPUs per node)` - Local SSD spare (GB) = `3 x (Storage size per node in GB)` For example, with per-node storage **1500 GB** (4 × 375 GB Local SSD) and machine type **n2d-standard-4** (4 vCPUs), keep **4500 GB** Local SSD and **12 vCPUs** of spare quota. ## [](#create-a-byoc-cluster)Create a BYOC cluster 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. On the Clusters page, click **Create cluster**, then click **Create** for BYOC. Enter a cluster name, then select the resource group, provider (GCP), [region, tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/), availability, and Redpanda version. > 📝 **NOTE** > > - If you plan to create a private network in your own VPC, select the region where your VPC is located. > > - Three availability zones provide two backups in case one availability zone goes down. Optionally, click **Advanced settings** to specify up to five key-value custom GCP labels. If a label key starts with `gcp.network-tag.`, then the agent interprets it as a request to apply the `` [network tag](https://cloud.google.com/vpc/docs/add-remove-network-tags) to GCE instances in the cluster. Use labels for organization/metadata; use network tags to target firewall rules and routes. After the cluster is created, labels are applied to applicable GCP resources (for example, instances and disks), and network tags are applied to instances. For more information, see the [GCP documentation](https://cloud.google.com/compute/docs/labeling-resources). You can also [specify more labels and network tags on the Dataplane settings page or with the Control Plane API](#manage-custom-resource-labels-and-network-tags). 3. Click **Next**. 4. On the Network page, select the connection type: either public or private. For BYOC clusters, private is best-practice. - Your network name is used to identify this network. - For a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/), choose one that does not overlap with your existing VPCs or your Redpanda network. - Clusters with private networking include a setting for API Gateway network access. Public access exposes endpoints for Redpanda Console, the Data Plane API, but they remain protected by your authentication and authorization controls. Private access restricts endpoint access to your VPC only. > 📝 **NOTE** > > After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. > 💡 **TIP** > > To route all cluster egress through your own GCP hub VPC and NAT VM instead of a per-cluster Cloud NAT, enter the **Hub VPC name** and **Hub project ID** on this page. These fields are only available on clusters with a private connection type, and are only visible if centralized egress is enabled for your organization. This option is in beta. See [Configure Centralized Egress with GCP VPC Peering](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/nat-free-egress/). 5. Click **Next**. 6. On the Deploy page, follow the steps to log in to Redpanda Cloud and deploy the agent. As part of agent deployment, Redpanda assigns the permissions required to run the agent. For details about these permissions, see [GCP IAM permissions](https://docs.redpanda.com/cloud-data-platform/security/authorization/cloud-iam-policies-gcp/). > 📝 **NOTE** > > Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. ## [](#manage-custom-resource-labels-and-network-tags)Manage custom resource labels and network tags Your organization might require custom resource labels and network tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage labels and network tags on your cluster’s **Dataplane settings** page in the Redpanda Cloud UI, or with the [Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/). > ⚠️ **CAUTION** > > Do not add labels or network tags directly to the GCP node pools of a BYOC cluster, either in the GCP console or with commands such as `gcloud container node-pools update`. Google Kubernetes Engine (GKE) treats a node pool label or tag update as a node replacement and aggressively replaces all nodes, rather than performing a controlled rolling upgrade. This bypasses the failover process that Redpanda requires for safe node cycling. It can leave persistent volume claims (PVCs) in a pending state and may require recovering the entire cluster. > > The Dataplane settings page and the Control Plane API are the supported methods. With both, Redpanda applies labels directly to the Google Compute Engine (GCE) instances and disks in your cluster, and network tags directly to the instances. Because nothing is applied to the node pools, GKE does not replace any nodes. ### [](#use-the-dataplane-settings-page)Use the Dataplane settings page To manage labels and network tags in the Redpanda Cloud UI: 1. In the Redpanda Cloud UI, open your [cluster](https://cloud.redpanda.com/clusters), and click **Dataplane settings**. 2. Under **Manage resource labels and network tags for your cluster**, click **Add**, and enter a key and value for each label. Keys and values are case-sensitive. To apply a network tag to the GCE instances in the cluster, use a key that starts with `gcp.network-tag.`. For example, the key `gcp.network-tag.web-servers` applies the `web-servers` [network tag](https://cloud.google.com/vpc/docs/add-remove-network-tags). You can add up to 10 entries on this page. To manage up to 16, use the [Control Plane API](#use-the-control-plane-api). 3. Click **Save**. ### [](#use-the-control-plane-api)Use the Control Plane API The Control Plane API allows up to 16 custom resource labels and network tags in GCP. Make sure you have: - The cluster ID. You can find this in the Redpanda Cloud UI, in the **Details** section of the cluster overview. - A valid bearer token for the Control Plane API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). Then complete the following steps: 1. To refresh agent permissions so the Redpanda agent can update labels and network tags, run: ```bash export CLUSTER_ID="" export PROJECT_ID="" rpk cloud byoc gcp apply --redpanda-id="$CLUSTER_ID" --project-id="$PROJECT_ID" ``` This step is required because label/tag management requires additional IAM permissions that may not have been granted during initial cluster creation: - `compute.disks.get` - `compute.disks.list` - `compute.disks.setLabels` - `compute.instances.setLabels` 2. To update labels and network tags, invoke the Control Plane API. First, set your authentication token: ```bash export AUTH_TOKEN="" ``` The `PATCH` call sets the labels and network tags specified under `"cloud_provider_tags"`. It replaces the existing labels and tags with the specified labels and tags. Include all desired labels and tags in the request. To remove a single entry, omit it from the map you send. ```bash cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "environment": "production", "cost-center": "engineering", "gcp.network-tag.web-servers": "true", "gcp.network-tag.database-access": "true" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` To remove all labels and network tags, send an empty `cloud_provider_tags` object: ```bash cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` ## [](#service-account-credential-rotation)Service account credential rotation To rotate service account credentials for your BYOC cluster, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new) with your cluster ID, the service accounts that require rotation, and your target timeline. > ⚠️ **WARNING** > > GCP service account credential rotation for BYOC clusters is not self-service. Rotating these credentials without coordinating with Redpanda can disrupt agent connectivity, monitoring, and Tiered Storage uploads, and can leave the cluster stuck and unable to complete future operations. ## [](#next-steps)Next steps [Configure private networking](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/) --- # Page 572: Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.md --- # Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc description: Add Redpanda Connect to your existing BYOVPC cluster. page-git-created-date: "2025-04-04" page-git-modified-date: "2026-05-26" --- > ❗ **IMPORTANT** > > BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or [Redpanda Sales](https://www.redpanda.com/price-estimator). To enable Redpanda Connect on an existing BYOVPC cluster, you must update your configuration. You can also create [a new BYOVPC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp/) with Redpanda Connect already enabled. Replace all `` with your own values. 1. Create two new service accounts with the necessary permissions and roles. Show commands ```bash # Account used to check for and read secrets, which are required to create Redpanda Connect pipelines. gcloud iam service-accounts create redpanda-connect-api \ --display-name="Redpanda Connect API Service Account" cat << EOT > redpanda-connect-api.role { "name": "redpanda_connect_api_role", "title": "Redpanda Connect API Role", "description": "Redpanda Connect API Role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_connect_api_role --project= --file redpanda-connect-api.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-connect-api@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_connect_api_role" ``` ```bash # Account used to retrieve secrets and create Redpanda Connect pipelines. gcloud iam service-accounts create redpanda-connect \ --display-name="Redpanda Connect Service Account" cat << EOT > redpanda-connect.role { "name": "redpanda_connect_role", "title": "Redpanda Connect Role", "description": "Redpanda Connect Role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_connect_role --project= --file redpanda-connect.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-connect@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_connect_role" ``` 2. Bind the service accounts. The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `my-gcp-sa@my-project.iam.gserviceaccount.com`, then the account ID is `my-gcp-sa`. Show commands ```none gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-connect/]" ``` ```none gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-connect/]" ``` 3. Make a [`PATCH /v1/clusters/{cluster-id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request to update the cluster configuration. Show request ```bash export CLUSTER_PATCH_BODY=`cat << EOF { "customer_managed_resources": { "gcp": { "redpanda_connect_api_service_account": { "email": "@.iam.gserviceaccount.com" }, "redpanda_connect_service_account": { "email": "@.iam.gserviceaccount.com" } } } } EOF` curl -v -X PATCH \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/ ``` 4. Check Redpanda Connect is available in the Cloud UI. 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the **Connect** page and you should see Redpanda Connect. ## [](#next-steps)Next steps - Choose [connectors for your use case](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/). - Learn how to [configure, test, and run a data pipeline locally](https://docs.redpanda.com/connect/get-started/quickstarts/rpk/). - Try the [Redpanda Connect quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/). - Try one of our [Redpanda Connect cookbooks](https://docs.redpanda.com/cloud-data-platform/develop/connect/cookbooks/). - Learn how to [add secrets to your pipeline](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/). --- # Page 573: Enable Secrets Management on an Existing BYOVPC Cluster on GCP **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.md --- # Enable Secrets Management on an Existing BYOVPC Cluster on GCP > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Enable Secrets Management on an Existing BYOVPC Cluster on GCP page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/gcp/enable-secrets-byovpc-gcp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc description: Store and read secrets in your existing BYOVPC cluster. # Beta release status page-beta: "true" page-git-created-date: "2025-06-06" page-git-modified-date: "2025-08-20" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- > ❗ **IMPORTANT** > > BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or [Redpanda Sales](https://www.redpanda.com/price-estimator). Storing secrets in your cluster allows you to keep your cloud infrastructure secure as you integrate your data across different systems, for example, REST catalogs with your Iceberg-enabled topics. If you do not have secrets management enabled on an existing BYOVPC cluster, you can do so by following the steps on this page to update your cluster configuration. You can also create [a new BYOVPC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp/) with secrets management already enabled. Replace all `` with your own values. 1. Create one new service account with the necessary permissions and roles. Show commands ```bash # Account used to check for and read secrets gcloud iam service-accounts create redpanda-operator \ --display-name="Redpanda Operator Service Account" cat << EOT > redpanda-operator.role { "name": "redpanda_operator_role", "title": "Redpanda Operator Role", "description": "Redpanda Operator Role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_operator_role --project= --file redpanda-operator.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-operator@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_operator_role" ``` 2. Update the existing Redpanda cluster service account with the necessary permissions to read secrets. Show commands ```bash cat << EOT > redpanda-cluster.role { "name": "redpanda_cluster_role", "title": "Redpanda Cluster Role", "description": "Redpanda Cluster Role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_cluster_role --project= --file redpanda-cluster.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-cluster@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_cluster_role" ``` 3. Bind the new service account. The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `my-gcp-sa@my-project.iam.gserviceaccount.com`, then the account ID is `my-gcp-sa`. Show commands ```none gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-system/]" ``` 4. Make a [`PATCH /v1/clusters/{cluster-id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request to update the cluster configuration. Show request ```bash export CLUSTER_PATCH_BODY=`cat << EOF { "customer_managed_resources": { "gcp": { "redpanda_operator_service_account": { "email": "@.iam.gserviceaccount.com" } } } } EOF` curl -v -X PATCH \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/ ``` 5. Check secrets management is available in the Cloud UI. 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Go to the **Secrets Store** page of your cluster. You should be able to create a new secret. ## [](#next-steps)Next steps - [Reference a secret in a cluster property](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties). - [Integrate a catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) for querying Iceberg topics in your cluster. --- # Page 574: Create a BYOVPC Cluster on GCP **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp.md --- # Create a BYOVPC Cluster on GCP > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create a BYOVPC Cluster on GCP latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/gcp/vpc-byo-gcp page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/gcp/vpc-byo-gcp.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc description: Connect Redpanda Cloud to your existing VPC for additional security. page-git-created-date: "2024-10-24" page-git-modified-date: "2026-08-03" --- > ❗ **IMPORTANT** > > BYOVPC/BYOVNet is an add-on feature that requires Premium support. To unlock this feature for your account, contact your Redpanda account team or [Redpanda Sales](https://www.redpanda.com/price-estimator). A Bring Your Own Virtual Private Cloud (BYOVPC) cluster allows you to deploy the Redpanda [data plane](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#data-plane) into your existing VPC and manage the networking lifecycle. Compared to a standard Bring Your Own Cloud (BYOC) setup, where Redpanda manages the networking lifecycle for you, BYOVPC provides more control. See also: [BYOC architecture](https://docs.redpanda.com/cloud-data-platform/get-started/byoc-arch/). When you create a BYOVPC cluster, you specify your VPC and service account. The Redpanda Cloud agent doesn’t create any new resources or alter any settings in your account. With BYOVPC: - You provide your own VPC in your Google Cloud account. - You maintain more control of your Google Cloud account, because Redpanda requires fewer permissions than standard BYOC clusters. - You control your security resources and policies, including subnets, service accounts, IAM roles, firewall rules, and storage buckets. If your clients need to connect from different GCP regions than where your cluster will be deployed, you must enable global access during cluster creation. To create a BYOVPC cluster with global access enabled, see [Enable Global Access](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/enable-global-access/). ## [](#prerequisites)Prerequisites - A standalone GCP project is recommended. If your host project (where your VPC project is created) and your service project (where your Redpanda cluster is created) are in different projects, you must first provision a shared VPC in Google Cloud. For more information, see the [Google shared VPC documentation](https://cloud.google.com/vpc/docs/provisioning-shared-vpc). - Redpanda creates a private Google Kubernetes Engine (GKE) cluster in your VPC. The subnet and secondary IP ranges you provide must allow public internet access. The configuration requires you to provide reserved CIDR ranges for the subnet and GKE Pods, Services, and master IP addresses. See the [GKE service account documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/service-accounts) and [Configure your VPC](#configure-your-vpc). - Only primary CIDR ranges are supported for the VPC. - Redpanda requires access to certain Google APIs, storage buckets, and service accounts. See [Configure the service project](#configure-the-service-project). ### [](#gcp-quotas)GCP quotas Ensure at least three nodes of headroom in the relevant GCP quotas in the same region as your cluster. During maintenance, Redpanda may temporarily create extra nodes. Quotas such as vCPUs per VM family (for example, N2D) and Local SSD total per VM family (quota key: `LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY`) are listed for each tier on the **Create BYOC cluster** page in the Redpanda Cloud UI. Headroom formulas: - vCPU spare = `3 x (vCPUs per node)` - Local SSD spare (GB) = `3 x (Storage size per node in GB)` For example, with per-node storage **1500 GB** (4 × 375 GB Local SSD) and machine type **n2d-standard-4** (4 vCPUs), keep **4500 GB** Local SSD and **12 vCPUs** of spare quota. ## [](#limitations)Limitations - Existing clusters cannot be moved to a BYOVPC cluster. - After creating a BYOVPC cluster, you cannot change to a different VPC. ## [](#configure-your-vpc)Configure your VPC 1. Create the primary and secondary subnets in your VPC using CIDR notation. Redpanda clusters require one subnet, and that subnet should have two secondary IP ranges: - Subnet IP range should be at least /24 CIDR, such as 10.0.0.0/24. - Secondary IP for GKE Pods is a /21 CIDR, such as 10.0.8.0/21. - Secondary IP for GKE Services is a /24 CIDR, such as 10.0.1.0/24. Replace all `` with your own values. ```bash gcloud compute networks subnets create \ --project \ --network \ --range 10.0.0.0/24 \ --region \ --secondary-range =10.0.8.0/21,=10.0.1.0/24 ``` Additionally, a /28 CIDR is required for the GKE master IP addresses. This CIDR is not used in the GCP networking configuration, but is input into the Redpanda UI; for example, 10.0.7.240/28. 2. To enable egress, create a cloud router and NAT at the host project: ```bash gcloud compute routers create \ --project \ --region \ --network gcloud compute addresses create --region gcloud compute routers nats create \ --project \ --router \ --region \ --nat-all-subnet-ip-ranges \ --nat-external-ip-pool \ --enable-endpoint-independent-mapping ``` 3. Create VPC firewall rules. - Redpanda ingress: ```bash gcloud compute firewall-rules create redpanda-ingress \ --description="Allow access to Redpanda cluster" \ --network="" \ --project="" \ --direction="INGRESS" \ --target-tags="redpanda-node" \ --source-ranges="10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10" \ --allow="tcp:9092-9094,tcp:30081,tcp:30082,tcp:30092" ``` - Master webhooks: ```bash gcloud compute firewall-rules create gke-redpanda-cluster-webhooks \ --description="Allow master to hit pods for admission controllers/webhooks" \ --network="" \ --project="" \ --direction="INGRESS" \ --source-ranges="" \ --allow="tcp:9443,tcp:8443,tcp:6443" ``` Replace `` with a /28 CIDR. For example: 172.16.0.32/28. For information about the master CIDR, and how to set it using `--master-ipv4-cidr`, see the **gcloud** tab in [Creating a private cluster with no client access to the public endpoint](https://cloud.google.com/kubernetes-engine/docs/how-to/legacy/network-isolation#private_cp) 4. Grant permission to read the VPC and related resources. If the host project and service project are in different projects, it’s helpful for the Redpanda team to have read access to the VPC and related resources in the host project. If your host project and service project are the same, you can skip this step. - Redpanda Agent custom role: ```bash cat << EOT > redpanda-agent.role { "name": "redpanda_agent_role", "title": "Redpanda Agent Role", "description": "A role granting the redpanda agent permissions to view network resources in the project of the vpc.", "includedPermissions": [ "compute.firewalls.get", "compute.subnetworks.get", "resourcemanager.projects.get", "compute.networks.getRegionEffectiveFirewalls", "compute.networks.getEffectiveFirewalls" ] } EOT gcloud iam roles create redpanda_agent_role --project= --file redpanda-agent.role ``` ## [](#configure-the-service-project)Configure the service project 1. Enable Google APIs in the service project: ```bash gcloud services enable cloudresourcemanager.googleapis.com --project gcloud services enable dns.googleapis.com --project gcloud services enable secretmanager.googleapis.com --project gcloud services enable compute.googleapis.com --project gcloud services enable iam.googleapis.com --project gcloud services enable storage-api.googleapis.com --project gcloud services enable container.googleapis.com --project gcloud services enable serviceusage.googleapis.com --project ``` 2. Create storage buckets at the service project in the same region as the cluster: ```bash gcloud storage buckets create gs:// \ --location="" \ --uniform-bucket-level-access gcloud storage buckets create gs:// \ --location="" \ --uniform-bucket-level-access gcloud storage buckets update gs:// --versioning ``` - Redpanda uses the tiered storage bucket for writing log segments. This should not be versioned. - Redpanda uses the management storage bucket to store cluster metadata. This can have versioning enabled. 3. Create service accounts with necessary permissions and roles. - Redpanda Cloud agent service account Show commands ```bash gcloud iam service-accounts create redpanda-agent \ --display-name="Redpanda Agent Service Account" cat << EOT > redpanda-agent.role { "name": "redpanda_agent_role", "title": "Redpanda Agent Role", "description": "A role comprising general permissions allowing the agent to manage Redpanda cluster resources.", "includedPermissions": [ "compute.firewalls.get", "compute.disks.get", "compute.globalOperations.get", "compute.instanceGroupManagers.get", "compute.instanceGroupManagers.delete", "compute.instanceGroups.delete", "compute.instances.list", "compute.instanceTemplates.delete", "compute.networks.getRegionEffectiveFirewalls", "compute.networks.getEffectiveFirewalls", "compute.projects.get", "compute.subnetworks.get", "compute.zoneOperations.get", "compute.zoneOperations.list", "compute.zones.get", "compute.zones.list", "dns.changes.create", "dns.changes.get", "dns.changes.list", "dns.managedZones.create", "dns.managedZones.delete", "dns.managedZones.get", "dns.managedZones.list", "dns.managedZones.update", "dns.projects.get", "dns.resourceRecordSets.create", "dns.resourceRecordSets.delete", "dns.resourceRecordSets.get", "dns.resourceRecordSets.list", "dns.resourceRecordSets.update", "iam.roles.get", "iam.roles.list", "iam.serviceAccounts.actAs", "iam.serviceAccounts.get", "iam.serviceAccounts.getIamPolicy", "resourcemanager.projects.get", "resourcemanager.projects.getIamPolicy", "serviceusage.services.list", "storage.buckets.get", "storage.buckets.getIamPolicy", "compute.subnetworks.use", "compute.instances.use", "compute.networks.use", "compute.regionOperations.get", "compute.serviceAttachments.create", "compute.serviceAttachments.delete", "compute.serviceAttachments.get", "compute.serviceAttachments.list", "compute.serviceAttachments.update", "compute.forwardingRules.use", "compute.forwardingRules.create", "compute.forwardingRules.delete", "compute.forwardingRules.get", "compute.forwardingRules.setLabels", "compute.forwardingRules.setTarget", "compute.forwardingRules.pscCreate", "compute.forwardingRules.pscDelete", "compute.forwardingRules.pscSetLabels", "compute.forwardingRules.pscSetTarget", "compute.forwardingRules.pscUpdate", "compute.regionBackendServices.create", "compute.regionBackendServices.delete", "compute.regionBackendServices.get", "compute.regionBackendServices.use", "compute.regionNetworkEndpointGroups.create", "compute.regionNetworkEndpointGroups.delete", "compute.regionNetworkEndpointGroups.get", "compute.regionNetworkEndpointGroups.use", "compute.regionNetworkEndpointGroups.attachNetworkEndpoints", "compute.regionNetworkEndpointGroups.detachNetworkEndpoints", "compute.disks.list", "compute.disks.setLabels", "compute.instanceGroupManagers.update", "compute.instances.delete", "compute.instances.get", "compute.instances.setLabels" ] } EOT gcloud iam roles create redpanda_agent_role --project= --file redpanda-agent.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-agent@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_agent_role" gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-agent@.iam.gserviceaccount.com" \ --role="roles/container.admin" gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:redpanda-agent@.iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" # skip this step if host project and service project are the same gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-agent@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_agent_role" ``` - Redpanda cluster service account Show commands ```bash cat << EOT > redpanda-cluster.role { "name": "redpanda_cluster_role", "title": "Redpanda Cluster Role", "description": "Redpanda Cluster role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam service-accounts create redpanda-cluster \ --display-name="Redpanda Cluster Service Account" gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:redpanda-cluster@.iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" gcloud iam roles create redpanda_cluster_role --project= --file redpanda-cluster.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-cluster@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_cluster_role" ``` - Redpanda operator service account Show commands ```bash gcloud iam service-accounts create redpanda-operator \ --display-name="Redpanda Operator Service Account" cat << EOT > redpanda-operator.role { "name": "redpanda_operator_role", "title": "Redpanda Operator Role", "description": "Redpanda Operator role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_operator_role --project= --file redpanda-operator.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-operator@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_operator_role" ``` - Redpanda Connect service accounts Show commands ```bash # Account used to check for and read secrets, which are required to create Redpanda Connect pipelines. gcloud iam service-accounts create redpanda-connect-api \ --display-name="Redpanda Connect API Service Account" cat << EOT > redpanda-connect-api.role { "name": "redpanda_connect_api_role", "title": "Redpanda Connect API Role", "description": "Redpanda Connect API role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.secrets.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_connect_api_role --project= --file redpanda-connect-api.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-connect-api@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_connect_api_role" ``` ```bash # Account used to retrieve secrets and create Redpanda Connect pipelines. gcloud iam service-accounts create redpanda-connect \ --display-name="Redpanda Connect Service Account" cat << EOT > redpanda-connect.role { "name": "redpanda_connect_role", "title": "Redpanda Connect Role", "description": "Redpanda Connect role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_connect_role --project= --file redpanda-connect.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-connect@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_connect_role" ``` - Redpanda Cloud secret manager Show commands ```bash gcloud iam service-accounts create redpanda-console \ --display-name="Redpanda Cloud Secret Manager" cat << EOT > redpanda-console.role { "name": "redpanda_console_secret_manager_role", "title": "Redpanda Cloud Secret Manager Writer", "description": "Redpanda Cloud Secret Manager Writer", "includedPermissions": [ "secretmanager.secrets.get", "secretmanager.secrets.create", "secretmanager.secrets.delete", "secretmanager.secrets.list", "secretmanager.secrets.update", "secretmanager.versions.add", "secretmanager.versions.destroy", "secretmanager.versions.disable", "secretmanager.versions.enable", "secretmanager.versions.list", "iam.serviceAccounts.getAccessToken" ] } EOT gcloud iam roles create redpanda_console_secret_manager_role --project= --file redpanda-console.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-console@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_console_secret_manager_role" ``` - Kafka Connect service account Show commands ```bash gcloud iam service-accounts create redpanda-connectors \ --display-name="Kafka Connect Service Account" cat << EOT > redpanda-connectors.role { "name": "redpanda_connectors_role", "title": "Kafka Connect Custom Role", "description": "Kafka Connect custom role", "includedPermissions": [ "resourcemanager.projects.get", "secretmanager.versions.access" ] } EOT gcloud iam roles create redpanda_connectors_role --project= --file redpanda-connectors.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-connectors@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_connectors_role" ``` - Redpanda GKE cluster service account Show commands ```bash gcloud iam service-accounts create redpanda-gke \ --display-name="Redpanda GKE cluster default node service account" cat << EOT > redpanda-gke.role { "name": "redpanda_gke_utility_role", "title": "Redpanda cluster utility node role", "description": "Redpanda cluster utility node role", "includedPermissions": [ "artifactregistry.dockerimages.get", "artifactregistry.dockerimages.list", "artifactregistry.files.get", "artifactregistry.files.list", "artifactregistry.locations.get", "artifactregistry.locations.list", "artifactregistry.mavenartifacts.get", "artifactregistry.mavenartifacts.list", "artifactregistry.npmpackages.get", "artifactregistry.npmpackages.list", "artifactregistry.packages.get", "artifactregistry.packages.list", "artifactregistry.projectsettings.get", "artifactregistry.pythonpackages.get", "artifactregistry.pythonpackages.list", "artifactregistry.repositories.downloadArtifacts", "artifactregistry.repositories.get", "artifactregistry.repositories.list", "artifactregistry.repositories.listEffectiveTags", "artifactregistry.repositories.listTagBindings", "artifactregistry.repositories.readViaVirtualRepository", "artifactregistry.tags.get", "artifactregistry.tags.list", "artifactregistry.versions.get", "artifactregistry.versions.list", "logging.logEntries.create", "logging.logEntries.route", "monitoring.metricDescriptors.create", "monitoring.metricDescriptors.get", "monitoring.metricDescriptors.list", "monitoring.monitoredResourceDescriptors.get", "monitoring.monitoredResourceDescriptors.list", "monitoring.timeSeries.create", "cloudnotifications.activities.list", "monitoring.alertPolicies.get", "monitoring.alertPolicies.list", "monitoring.dashboards.get", "monitoring.dashboards.list", "monitoring.groups.get", "monitoring.groups.list", "monitoring.notificationChannelDescriptors.get", "monitoring.notificationChannelDescriptors.list", "monitoring.notificationChannels.get", "monitoring.notificationChannels.list", "monitoring.publicWidgets.get", "monitoring.publicWidgets.list", "monitoring.services.get", "monitoring.services.list", "monitoring.slos.get", "monitoring.slos.list", "monitoring.snoozes.get", "monitoring.snoozes.list", "monitoring.timeSeries.list", "monitoring.uptimeCheckConfigs.get", "monitoring.uptimeCheckConfigs.list", "opsconfigmonitoring.resourceMetadata.list", "resourcemanager.projects.get", "stackdriver.projects.get", "stackdriver.resourceMetadata.list", "dns.changes.create", "dns.changes.get", "dns.changes.list", "dns.managedZones.list", "dns.resourceRecordSets.create", "dns.resourceRecordSets.delete", "dns.resourceRecordSets.get", "dns.resourceRecordSets.list", "dns.resourceRecordSets.update", "secretmanager.versions.access", "stackdriver.resourceMetadata.write", "storage.objects.get", "storage.objects.list", "compute.instances.use", "iam.serviceAccounts.getAccessToken", "compute.regionNetworkEndpointGroups.create", "compute.regionNetworkEndpointGroups.delete", "compute.regionNetworkEndpointGroups.get", "compute.regionNetworkEndpointGroups.use", "compute.regionNetworkEndpointGroups.attachNetworkEndpoints", "compute.regionNetworkEndpointGroups.detachNetworkEndpoints" ] } EOT gcloud iam roles create redpanda_gke_utility_role --project= --file redpanda-gke.role gcloud projects add-iam-policy-binding \ --member="serviceAccount:redpanda-gke@.iam.gserviceaccount.com" \ --role="projects//roles/redpanda_gke_utility_role" ``` 4. Bind the service accounts. The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `my-gcp-sa@my-project.iam.gserviceaccount.com`, then the account ID is `my-gcp-sa`. - Redpanda cluster service account Show command ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda/rp-]" ``` - Redpanda operator service account Show command ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-system/]" ``` - Redpanda Console service account Show command ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda/console-]" ``` - Redpanda Connect service accounts Show command ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-connect/]" ``` ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-connect/]" ``` - Kafka Connect service account Show command ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-connectors/connectors-]" ``` - Cert-manager and external-DNS service accounts Show commands ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[cert-manager/cert-manager]" gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[external-dns/external-dns]" ``` - Private Service Connect Controller service account Show commands ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:.svc.id.goog[redpanda-psc/psc-controller]" ``` ## [](#create-cluster)Create cluster Log in to the [Redpanda Cloud UI](https://cloud.redpanda.com), and follow the steps to [create a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/create-byoc-cluster-gcp/), with the following exceptions: 1. On the **Network** page, select the **BYOVPC** connection type, and enter the network, service account, storage bucket information, and GKE master CIDR range you created. 2. With customer-managed networks, you must grant yourself (the user deploying the cluster with `rpk`) the following permissions: Expand permissions - `compute.disks.create` - `compute.disks.setLabels` - `compute.instanceGroupManagers.create` - `compute.instanceGroupManagers.delete` - `compute.instanceGroupManagers.get` - `compute.instanceGroups.create` - `compute.instanceGroups.delete` - `compute.instanceTemplates.create` - `compute.instanceTemplates.delete` - `compute.instanceTemplates.get` - `compute.instanceTemplates.useReadOnly` - `compute.instances.create` - `compute.instances.setLabels` - `compute.instances.setMetadata` - `compute.instances.setTags` - `compute.subnetworks.get` - `compute.subnetworks.use` - `compute.zones.list` - `iam.roles.get` - `iam.serviceAccounts.actAs` - `iam.serviceAccounts.get` - `resourcemanager.projects.get` - `resourcemanager.projects.getIamPolicy` - `serviceusage.services.list` - `storage.buckets.get` - `storage.buckets.getIamPolicy` - `storage.objects.create` - `storage.objects.delete` - `storage.objects.get` - `storage.objects.list` This can be done through a Google account, a service account, or any principal identity supported by GCP. - If running `rpk` from a Google account, the user must acquire new user credentials to use for [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login). - If running `rpk` from a service account, the user must create a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating), then [export GOOGLE\_APPLICATION\_CREDENTIALS](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) and [set the account as the default in gcloud](https://cloud.google.com/sdk/gcloud/reference/config/set): ```bash export GOOGLE_APPLICATION_CREDENTIALS= gcloud config set account $SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com ``` 3. To validate your configuration, run: ```bash rpk cloud byoc gcp apply --redpanda-id='' --project-id='' --validate-only ``` 4. Click **Next**. 5. On the **Deploy** page, similar to standard BYOC clusters, log in to Redpanda Cloud and deploy the agent. > 📝 **NOTE** > > Redpanda Cloud does not support customer access or modifications to any of the internal data plane resources. This restriction allows Redpanda Data to manage all configuration changes internally to ensure a 99.99% service level agreement (SLA) for BYOC clusters. ## [](#delete-cluster)Delete cluster You can delete the cluster in the Cloud UI. 1. Log in to [Redpanda Cloud](https://cloud.redpanda.com). 2. Select your cluster. 3. Go to the **Dataplane settings** page and click **Delete**, then confirm your deletion. ## [](#manage-custom-resource-labels-and-network-tags)Manage custom resource labels and network tags Your organization might require custom resource labels and network tags for cost allocation, audit compliance, or governance policies. After cluster creation, you can manage labels and network tags on your cluster’s **Dataplane settings** page in the Redpanda Cloud UI, or with the [Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/). > ⚠️ **CAUTION** > > Do not add labels or network tags directly to the GCP node pools of a BYOC cluster, either in the GCP console or with commands such as `gcloud container node-pools update`. Google Kubernetes Engine (GKE) treats a node pool label or tag update as a node replacement and aggressively replaces all nodes, rather than performing a controlled rolling upgrade. This bypasses the failover process that Redpanda requires for safe node cycling. It can leave persistent volume claims (PVCs) in a pending state and may require recovering the entire cluster. > > The Dataplane settings page and the Control Plane API are the supported methods. With both, Redpanda applies labels directly to the Google Compute Engine (GCE) instances and disks in your cluster, and network tags directly to the instances. Because nothing is applied to the node pools, GKE does not replace any nodes. ### [](#use-the-dataplane-settings-page)Use the Dataplane settings page To manage labels and network tags in the Redpanda Cloud UI: 1. In the Redpanda Cloud UI, open your [cluster](https://cloud.redpanda.com/clusters), and click **Dataplane settings**. 2. Under **Manage resource labels and network tags for your cluster**, click **Add**, and enter a key and value for each label. Keys and values are case-sensitive. To apply a network tag to the GCE instances in the cluster, use a key that starts with `gcp.network-tag.`. For example, the key `gcp.network-tag.web-servers` applies the `web-servers` [network tag](https://cloud.google.com/vpc/docs/add-remove-network-tags). You can add up to 10 entries on this page. To manage up to 16, use the [Control Plane API](#use-the-control-plane-api). 3. Click **Save**. ### [](#use-the-control-plane-api)Use the Control Plane API The Control Plane API allows up to 16 custom resource labels and network tags in GCP. Make sure you have: - The cluster ID. You can find this in the Redpanda Cloud UI, in the **Details** section of the cluster overview. - A valid bearer token for the Control Plane API. For details, see [Authenticate to the API](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). Then complete the following steps: 1. To refresh agent permissions so the Redpanda agent can update labels and network tags, run: ```bash export CLUSTER_ID="" export PROJECT_ID="" rpk cloud byoc gcp apply --redpanda-id="$CLUSTER_ID" --project-id="$PROJECT_ID" ``` This step is required because label/tag management requires additional IAM permissions that may not have been granted during initial cluster creation: - `compute.disks.get` - `compute.disks.list` - `compute.disks.setLabels` - `compute.instances.setLabels` 2. To update labels and network tags, invoke the Control Plane API. First, set your authentication token: ```bash export AUTH_TOKEN="" ``` The `PATCH` call sets the labels and network tags specified under `"cloud_provider_tags"`. It replaces the existing labels and tags with the specified labels and tags. Include all desired labels and tags in the request. To remove a single entry, omit it from the map you send. ```bash cluster_patch_body=$(cat <<'JSON' { "cloud_provider_tags": { "environment": "production", "cost-center": "engineering", "gcp.network-tag.web-servers": "true", "gcp.network-tag.database-access": "true" } } JSON ) curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` To remove all labels and network tags, send an empty `cloud_provider_tags` object: ```bash cluster_patch_body='{"cloud_provider_tags": {}}' curl -X PATCH "https://api.redpanda.com/v1/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $AUTH_TOKEN" \ -d "$cluster_patch_body" ``` > 📝 **NOTE** > > For BYOVPC clusters, custom labels are not applied to the customer-managed resources that are deployed by the customer. ## [](#next-steps)Next steps - [Configure private networking](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/) - [Enable Redpanda SQL on a BYOVPC Cluster on GCP](https://docs.redpanda.com/cloud-data-platform/sql/get-started/enable-sql-byovpc-gcp/) --- # Page 575: Create Remote Read Replicas **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/remote-read-replicas.md --- # Create Remote Read Replicas > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Create Remote Read Replicas latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/byoc/remote-read-replicas page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/byoc/remote-read-replicas.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/byoc/remote-read-replicas.adoc description: Learn how to create a remote read replica topic with BYOC, which is a read-only topic that mirrors a topic on a different cluster. page-git-created-date: "2024-08-01" page-git-modified-date: "2026-07-06" --- A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. You can create a separate remote cluster just for consumers of this topic and populate its topics from object storage. A read-only topic on a remote cluster can serve any consumer, without increasing the load on the source cluster. Because these read-only topics access data directly from object storage, there’s no impact to the performance of the cluster. Remote read replica topics do not store any data. When a cluster running a remote read replica is terminated, the topic data only exists on the origin cluster. Redpanda Cloud supports remote read replica topics in BYOC clusters on AWS or GCP. These clusters can be ephemeral; that is, created temporarily to handle specific or transient workloads, but they don’t have to be. The ability to make them ephemeral provides flexibility and cost efficiency: you can scale resources up or down as needed and pay only for what you use. > ❗ **IMPORTANT** > > Creating the remote read replica topic is only one of several required steps. To set up a remote read replica, you must complete all of the following: > > 1. **BYOVPC only**: Grant the reader cluster’s service account read access to the source cluster’s storage bucket. See [BYOVPC: Grant storage permissions](#byovpc-storage-permissions). > > 2. Link the reader cluster to the source cluster by setting `read_replica_cluster_ids`. See [Link the reader cluster to the source cluster](#link-clusters). Creating the topic alone does _not_ link the clusters. > > 3. Create the remote read replica topic. See [Create remote read replica topic](#create-remote-read-replica-topic). ## [](#prerequisites)Prerequisites To use remote read replicas, you need: - A BYOC reader cluster in Ready state. This separate reader cluster must exist in the same Redpanda organization as the source cluster. - AWS: The reader cluster must be in the same region and the same account as the source cluster. - GCP: The reader cluster can be in the same or a different region as the source cluster. The reader cluster must be in the same project as the source cluster. - Azure: Remote read replicas are not supported. ### [](#byovpc-storage-permissions)BYOVPC: Grant storage permissions > 📝 **NOTE** > > This prerequisite only applies to BYOVPC deployments. Skip this step if you’re enabling remote read replicas on standard BYOC clusters. #### GCP To grant additional permissions to the cloud storage manager of the reader cluster, run: ```bash gcloud storage buckets add-iam-policy-binding \ gs:// \ --member="serviceAccount:" \ --role="roles/storage.objectViewer" ``` #### AWS To grant additional permissions to the cloud storage manager of the reader cluster, set the `source_cluster_bucket_names` and `reader_cluster_id` variables in [cloud-examples](https://github.com/redpanda-data/cloud-examples/blob/main/customer-managed/aws/terraform/variables.tf). This should be done in the Terraform of the reader cluster. If the reader cluster’s service account does not have read access to the source cluster’s storage bucket, creating the remote read replica topic fails with an error like `UNKNOWN_SERVER_ERROR: Unable to perform requested topic operation`. ## [](#link-clusters)Link the reader cluster to the source cluster Linking is a required step. It does more than record the relationship between the clusters: - It orders upgrades safely across the linked clusters. - It prevents the source cluster from being deleted while it is linked to a reader cluster. The way linking interacts with storage permissions depends on your deployment type: - Standard BYOC: Linking the clusters grants the reader cluster read access to the source cluster’s storage bucket, so you cannot create a remote read replica topic until the clusters are linked. - BYOVPC: You grant the reader cluster read access to the bucket manually (see [BYOVPC: Grant storage permissions](#byovpc-storage-permissions)), so creating the topic succeeds even when the clusters are not linked. This makes the linking step easy to miss. > ⚠️ **WARNING** > > Creating a remote read replica topic does _not_ link the reader cluster to the source cluster. You must also set `read_replica_cluster_ids` as described in this section. Without this link, upgrades are not ordered safely across the clusters, and the source cluster is not protected from deletion, even if remote read replica topics exist. Add or remove reader clusters to a source cluster in Redpanda Cloud with the [Cloud Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/controlplane/). For information on accessing the Cloud API, see the [authentication guide](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication). 1. To update your source cluster to add one or more reader cluster IDs, make a [`PATCH /v1/clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request. The full list of clusters is expected on every call. If an ID is removed from the list, it is removed as a reader cluster. ```bash export SOURCE_CLUSTER_ID=....... export READER_CLUSTER_ID=....... curl -X PATCH $API_HOST/v1/clusters/$SOURCE_CLUSTER_ID \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ -d @- << EOF { "read_replica_cluster_ids": ["$READER_CLUSTER_ID"] } EOF ``` 2. Optional: To see the list of reader clusters on a given source cluster, make a [`GET /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster) request: ```bash export SOURCE_CLUSTER_ID=....... curl -X GET $API_HOST/v1/clusters/$SOURCE_CLUSTER_ID \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" ``` > 📝 **NOTE** > > A source cluster cannot be deleted while any reader cluster IDs are set in its `read_replica_cluster_ids` list. This deletion protection is based on the link, not on the existence of remote read replica topics. When you delete a reader cluster, that cluster’s ID is automatically removed from any existing source cluster `read_replica_cluster_ids` lists. ## [](#create-remote-read-replica-topic)Create remote read replica topic To create a remote read replica topic, run: ```bash rpk topic create -c redpanda.remote.readreplica= --tls-enabled ``` - For ``, use the same name as the original topic. - For ``, use the bucket specified in the `cloud_storage_bucket` properties for the origin cluster. For standard BYOC clusters, the source cluster bucket name follows the pattern: `redpanda-cloud-storage-${SOURCE_CLUSTER_ID}` ## [](#optional-tune-for-live-topics)Optional: Tune for live topics For remote read replicas reading from a live topic (that is, a topic that’s being actively written to by a source cluster), it may be advantageous to control how often segments are flushed to object storage. By default, this is set to 60 minutes. To tune `cloud_storage_segment_max_upload_interval_sec` on the source cluster, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). (For cold topics, where segments are closed and older than 60 minutes, this configuration is unnecessary: the data is already uploaded to object storage.) --- # Page 576: Dedicated **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster.md --- # Dedicated > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Dedicated latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/create-dedicated-cloud-cluster page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/create-dedicated-cloud-cluster.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/create-dedicated-cloud-cluster.adoc description: Learn how to create a Dedicated cluster and start streaming. page-git-created-date: "2025-04-01" page-git-modified-date: "2026-06-04" --- After you log in to [Redpanda Cloud](https://cloud.redpanda.com), you land on the **Clusters** page. This page lists all the clusters in your organization. ## [](#create-a-dedicated-cluster)Create a Dedicated cluster 1. On the Clusters page, click **Create cluster**, then click **Create** for Dedicated. Enter a cluster name, then select the resource group, cloud provider (AWS, GCP, or Azure), [region, tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/), availability, and Redpanda version. > ❗ **IMPORTANT** > > Dedicated on Azure is in [limited availability](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#limited-availability). It is production-ready and covered by Redpanda Support for early adopters. > 📝 **NOTE** > > - If you plan to create a private network in your own VPC, select the region where your VPC is located. > > - Three availability zones provide two backups in case one availability zone goes down. 2. Click **Next**. 3. On the Network page, enter the connection type: public or private. For private networks: - Your network name is used to identify this network. - For a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/), choose one that does not overlap with your existing VPCs or your Redpanda network. Private networks require either a VPC peering connection or a private connectivity service, such as [AWS PrivateLink](https://docs.redpanda.com/cloud-data-platform/networking/configure-privatelink-in-cloud-ui/), [GCP Private Service Connect](https://docs.redpanda.com/cloud-data-platform/networking/configure-private-service-connect-in-cloud-ui/), or [Azure Private Link](https://docs.redpanda.com/cloud-data-platform/networking/azure-private-link/). - Clusters with private networking include a setting for API Gateway network access. Public access exposes endpoints for Redpanda Console, the Data Plane API, and the MCP Server API, but they remain protected by your authentication and authorization controls. Private access restricts endpoint access to your VPC/VNet only. On Azure, private access incurs an additional cost, since it involves deploying two network load balancers, instead of one. > 📝 **NOTE** > > After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. 4. Click **Create**. After the cluster is created, you can select the cluster on the **Clusters** page to see the overview for it. ## [](#start-streaming-example)Start streaming: example Use `rpk`, Redpanda’s CLI, to build a basic streaming application that creates a topic, produces messages to it, and consumes messages from it. To learn about `rpk`, see the [Introduction to rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/intro-to-rpk/). 1. Login to Redpanda Cloud, and select your resource group using the interactive prompt. ```bash rpk cloud login ``` 2. On the **Overview** page, copy your bootstrap server address and set it as an environment variable on your local machine: ```bash export REDPANDA_BROKERS="" ``` 3. Go to **Security** > **Users**, click **Create user**, and create a user called **redpanda-chat-account** that uses the SCRAM-SHA-256 mechanism. 4. In the **User created successfully** dialog, copy the password and set the following environment variables on your local machine: ```bash export REDPANDA_SASL_USERNAME="redpanda-chat-account" export REDPANDA_SASL_PASSWORD="" export REDPANDA_SASL_MECHANISM="SCRAM-SHA-256" ``` 5. Click **Go to user details**. 6. Under **ACLs**, click **\+ Add ACL**, and define the following rule to grant the user full access to the `chat-room` topic: - **Resource Type**: Topic - **Pattern Type**: Literal - **Resource Name**: `chat-room` - **Operation**: All - **Permission**: Allow - **Host**: `*` 7. Click **Add ACL**. 8. Use `rpk` on your local machine to authenticate to Redpanda as the **redpanda-chat-account** user and get information about the cluster: ```bash rpk cluster info -X tls.enabled=true ``` 9. Create a topic called `chat-room`. You granted permissions to the **redpanda-chat-account** user to access only this topic. ```bash rpk topic create chat-room -X tls.enabled=true ``` Output: TOPIC STATUS chat-room OK 10. Produce a message to the topic: ```bash rpk topic produce chat-room -X tls.enabled=true ``` 11. Enter a message, then press Enter: ```text Pandas are fabulous! ``` Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 12. Press Ctrl+C to finish producing messages to the topic. 13. Consume one message from the topic: ```bash rpk topic consume chat-room --num 1 -X tls.enabled=true ``` Your message is displayed along with its metadata: ```json { "topic": "chat-room", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ### [](#explore-your-topic)Explore your topic In Redpanda Cloud, go to **Topics** > **chat-room**. The message that you produced to the topic is displayed along with some other details about the topic. ### [](#clean-up)Clean up If you don’t want to continue experimenting with your cluster, you can delete it. Go to **Dataplane settings** and click **Delete cluster**. ## [](#next-steps)Next steps - [Learn more about Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/) - [Learn about private networking](https://docs.redpanda.com/cloud-data-platform/networking/dedicated/) --- # Page 577: Serverless **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless.md --- # Serverless > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Serverless latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-types/serverless page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-types/serverless.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/cluster-types/serverless.adoc description: Learn how to create a Serverless cluster and start streaming. page-topic-type: overview personas: evaluator, app_developer, platform_admin learning-objective-1: Identify the use cases and usage limits for Serverless clusters learning-objective-2: Describe how to create a Serverless cluster and connect a client learning-objective-3: Recognize which features are supported and unsupported on Serverless page-git-created-date: "2024-06-06" page-git-modified-date: "2026-07-10" --- Serverless is the fastest and easiest way to start data streaming. With Serverless clusters, you host your data in Redpanda’s VPC, and Redpanda handles automatic scaling, provisioning, operations, and maintenance. This is a production-ready deployment option with a cluster available instantly, and you only pay for what you consume. You can view detailed billing activity for each cluster and edit payment methods on the **Billing** page. After reading this page, you will be able to: - Identify the use cases and usage limits for Serverless clusters - Describe how to create a Serverless cluster and connect a client - Recognize which features are supported and unsupported on Serverless ## [](#serverless-usage-limits)Serverless usage limits Each Serverless cluster has the following maximum usage limits: - **Ingress**: 100 MB/s - **Egress**: 300 MB/s - **Partitions**: 5,000 - **Message size**: 20 MiB - **Retention**: unlimited - **Storage**: unlimited - **Users**: 30 - **ACLs**: 120 - **Consumer groups**: 200 - **Connections**: 10,000 - **Producer IDs**: 250 - **Schema Registry**: - **Max schemas**: 500 - **Max subjects**: 500 - **Rate limit**: 100 requests/s - **Redpanda Connect pipelines**: 100 > 📝 **NOTE** > > The partition limit is the number of logical partitions before replication occurs. Redpanda Cloud uses a replication factor of 3. ## [](#prerequisites)Prerequisites Make sure you have the latest version of `rpk`, the Redpanda CLI. See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). ## [](#get-started-with-serverless)Get started with Serverless > 📝 **NOTE** > > Serverless on GCP is currently in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release. Choose the option that fits how you want to subscribe: ### Free trial A [free trial on AWS](https://www.redpanda.com/try-redpanda) is the fastest way to get started with Serverless. Each free-trial customer qualifies for $100 (USD) in credits to spend in the first 30 days. This should be enough to run Redpanda with reasonable throughput. No credit card is required. To continue using Serverless after your trial expires, you can enter a credit card and pay as you go. Any remaining credit balance is used before you are charged. When either the credits expire or the days in the trial expire, the clusters move into a suspended state, and you won’t be able to access your data in either the Redpanda Cloud Console or with the Kafka API. There is a seven-day grace period following the end of the trial when you can add your credit card and restore service. After that, the data is permanently deleted. For questions about the trial, use the **#serverless** [Community Slack](https://redpandacommunity.slack.com/) channel. After you start a trial, Redpanda instantly prepares an account for you. The first time you sign in, you can answer a few quick questions about your project so Redpanda can tailor your experience. Your account includes a `welcome` cluster with a `hello-world` demo topic you can explore. It includes sample data so you can see how real-time messaging works before sending your own data. On that first visit, the **Overview** page shows a **Get started** button with guided ways to [interact with your cluster](#interact-with-your-cluster): create a Redpanda Connect [pipeline](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#pipeline), use `rpk` from the command line, or connect with your own Kafka client. To get started with `rpk`: 1. Log in with `rpk cloud login`. 2. Consume from the `hello-world` topic with `rpk topic consume hello-world`. 3. In the [Redpanda Cloud Console](https://cloud.redpanda.com), navigate to the **Topics** page and open the `hello-world` topic to see the included messages. ### Redpanda Sales To request a private offer with possible discounts for annual committed use, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). When you subscribe to Serverless through Redpanda Sales, you gain immediate access to Enterprise support. Redpanda creates a cloud organization for you and sends you a welcome email. ### AWS Marketplace New subscriptions to Redpanda Cloud through [AWS Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/aws-pay-as-you-go/) receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. > 📝 **NOTE** > > When you subscribe to Redpanda through AWS Marketplace, you do not have immediate access to Enterprise support, only the [Community Slack](https://redpandacommunity.slack.com/) channel. For Enterprise support, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). Redpanda creates a cloud organization for you and sends you a welcome email. ### Google Cloud Marketplace New subscriptions to Redpanda Cloud through [Google Cloud Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/gcp-pay-as-you-go/) receive $300 (USD) in free credits to spend in the first 30 days. Google Cloud Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. > 📝 **NOTE** > > When you subscribe to Redpanda through Google Cloud Marketplace, you do not have immediate access to Enterprise support, only the [Community Slack](https://redpandacommunity.slack.com/) channel. For Enterprise support, contact [Redpanda Sales](https://www.redpanda.com/price-estimator). Redpanda creates a cloud organization for you and sends you a welcome email. ## [](#create-a-serverless-cluster)Create a Serverless cluster To create a Serverless cluster: 1. In the [Redpanda Cloud Console](https://cloud.redpanda.com), on the **Clusters** page, click **Create cluster**, then click **Create** for Serverless. 2. Enter a cluster name, then select the resource group. If you don’t have an existing resource group, you can create one. Refresh the page to see newly-created resource groups. 3. Select a cloud provider: AWS or GCP. (GCP is currently in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release.) 4. Select a [region](https://docs.redpanda.com/cloud-data-platform/reference/tiers/serverless-regions/). For best performance, select the region closest to your applications. Redpanda expects your applications to be deployed in the same cloud provider and region as your Serverless cluster. 5. **AWS only**: Clusters on AWS can enable private access between their VPC and Redpanda, so data does not traverse the public internet. Private connectivity is implemented using AWS PrivateLink for secure traffic. - When you enable both public access and private access on the cluster, you can choose between the public address or the private address. When the public address is used the data flows over the public internet. - You can either create a new PrivateLink or use an existing one from the same resource group. - You can enable or disable private access at any time on the cluster’s **Dataplane settings** page. - Enabling private access incurs additional charges. > 📝 **NOTE** > > After private access is disabled, attempts to reach the private endpoints will fail. However, the PrivateLink endpoint in your AWS account and the PrivateLink resource in Redpanda Cloud both remain provisioned and continue to incur charges until you explicitly delete them. 6. Click **Create cluster**. 7. To start working with your cluster, go to the **Topics** page to create a topic and produce messages to it. Add team members on the **Security** > **Users** page, then click into a user to assign roles or add ACLs from their detail page. ## [](#interact-with-your-cluster)Interact with your cluster > 💡 **TIP** > > The cluster’s **Overview** page includes a **Get started** guide to help you start streaming data into and out of Redpanda. See also: [Redpanda Connect Quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/) The **Overview** page lists your bootstrap server URL and security settings in the **How to connect - Kafka API** tab. Here you can add a Kafka client to interact with your cluster. Or, Redpanda can generate a sample application to interact with your cluster. Run [`rpk generate app`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-generate/rpk-generate-app/), and select Go as the language. Follow the commands in the terminal to run the application, create a demo topic, produce to the topic, and consume the data back. The first time you sign in, the **Overview** page shows a **Get started** button whose **Connect from your terminal** option walks you through using `rpk`. You can also use these `rpk` commands at any time: - [`rpk cloud login`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-login/): Use this to log in to Redpanda Cloud or to refresh the session. - [`rpk topic`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-topic/rpk-topic/): Use this to manage topics, produce data, and consume data. - [`rpk profile print`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-profile/rpk-profile-print/): Use this to view your `rpk` configuration and see the URL for your Serverless cluster. - [`rpk security user`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-security/rpk-security-user/): Use this to manage users and permissions. > 📝 **NOTE** > > Redpanda Serverless is opinionated about Kafka configurations. For example, automatic topic creation is disabled. Some systems expect the Kafka service to automatically create topics when a message is produced to a topic that doesn’t exist. Create topics on the **Topics** page or with `rpk topic create`. ## [](#supported-features)Supported features - Redpanda Serverless supports the Kafka API. Serverless clusters work with all Kafka clients. See [Kafka Compatibility](https://docs.redpanda.com/cloud-data-platform/develop/kafka-clients/). - Serverless clusters support all major Apache Kafka messages for managing topics, producing/consuming data (including transactions), managing groups, managing offsets, and managing ACLs. (User management is available in the [Redpanda Cloud Console](https://cloud.redpanda.com) or with `rpk security acl`.) ### [](#unsupported-features)Unsupported features Not all features included in BYOC clusters are available in Serverless. For example, the following features are not supported: - HTTP Proxy API - Multiple availability zones (AZs) - Role-based access control (RBAC) in the data plane and mTLS authentication for Kafka API clients - Group-based access control (GBAC) - Kafka Connect - Configurable maintenance windows ## [](#maintenance-and-upgrades)Maintenance and upgrades Redpanda manages all maintenance for Serverless clusters. Because Serverless runs on shared, multi-tenant infrastructure, you cannot configure a maintenance window or schedule upgrades for an individual cluster. Redpanda may run maintenance operations on Serverless clusters at any time. Continuous operations are integral to keeping Serverless clusters up to date and secure. Operations run in a rolling fashion and are designed to be non-disruptive. Mainstream Kafka client libraries reconnect automatically when broker connections restart. If you need control over when maintenance runs on your cluster, use a Dedicated or BYOC cluster, both of which support configurable maintenance windows. For more information, see [Upgrades and Maintenance](https://docs.redpanda.com/cloud-data-platform/manage/maintenance/). ## [](#next-steps)Next steps - [Set up private access for Serverless clusters](https://docs.redpanda.com/cloud-data-platform/networking/serverless/aws/) - [Manage Redpanda Cloud with Terraform](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) - [Learn more about Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/) - [Manage topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/config-topics/) - [Learn about billing](https://docs.redpanda.com/cloud-data-platform/billing/billing/) --- # Page 578: Introduction to Redpanda **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/intro-to-events.md --- # Introduction to Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Introduction to Redpanda latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: intro-to-events page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: intro-to-events.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/intro-to-events.adoc description: Learn about Redpanda event streaming. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Distributed systems often require data and system updates to happen as quickly as possible. In software architecture, these updates can be handled with either messages or events. - With messages, updates are sent directly from one component to another to trigger an action. - With events, updates indicate that an action occurred at a specific time, and are not directed to a specific recipient. An event is simply a record of something changing state. For example, the event of a credit card transaction includes the product purchased, the payment, the delivery, and the time of the purchase. The event occurred in the purchasing component, but it also impacted the inventory, the payment processing, and the shipping components. In an event-driven architecture, all actions are defined and packaged as events to precisely identify individual actions and how they’re processed throughout the system. Instead of processing updates in consecutive order, event-driven architecture lets components process events at their own pace. This helps developers build fast and scalable systems. ## [](#what-is-redpanda)What is Redpanda? Redpanda is an event streaming platform: it provides the infrastructure for streaming real-time data. Producers are client applications that send data to Redpanda in the form of events. Redpanda safely stores these events in sequence and organizes them into topics, which represent a replayable log of changes in the system. Consumers are client applications that subscribe to Redpanda topics to asynchronously read events. Consumers can store, process, or react to the events. Redpanda decouples producers from consumers to allow for asynchronous event processing, event tracking, event manipulation, and event archiving. Producers and consumers interact with Redpanda using the Apache Kafka® API. ![Producers and consumers in a cluster](https://docs.redpanda.com/cloud-data-platform/shared/_images/cluster.png) | Event-driven architecture (Redpanda) | Message-driven architecture | | --- | --- | | Producers send events to an event processing system (Redpanda) that acknowledges receipt of the write. This guarantees that the write is durable within the system and can be read by multiple consumers. | Producers send messages directly to each consumer. The producer must wait for acknowledgement that the consumer received the message before it can continue with its processes. | Event streaming lets you extract value out of each event by analyzing, mining, or transforming it for insights. You can: - Take one event and consume it in multiple ways. - Replay events from the past and route them to new processes in your application. - Run transformations on the data in real-time or historically. - Integrate with other event processing systems that use the Kafka API. ## [](#redpanda-differentiators)Redpanda differentiators Redpanda is less complex and less costly than any other commercial mission-critical event streaming platform. It’s fast, it’s easy, and it keeps your data safe. - Redpanda is designed for maximum performance on any data streaming workload. It can scale up to use all available resources on a single machine and scale out to distribute performance across multiple nodes. Built on C++, Redpanda delivers greater throughput and up to 10x lower p99 latencies than other platforms. This enables previously unimaginable use cases that require high throughput, low latency, and a minimal hardware footprint. - Redpanda is packaged as a single binary: it doesn’t rely on any external systems. It’s compatible with the Kafka API, so it works with the full ecosystem of tools and integrations built on Kafka. Redpanda can be deployed on bare metal, containers, or virtual machines in a data center or in the cloud. And Redpanda Console makes it easy to set up, manage, and monitor your clusters. Additionally, Tiered Storage lets you offload log segments to object storage in near real-time, providing long-term data retention and topic recovery. - Redpanda uses the [Raft consensus algorithm](https://raft.github.io/) throughout the platform to coordinate writing data to log files and replicating that data across multiple servers. Raft facilitates communication between the nodes in a Redpanda cluster to make sure that they agree on changes and remain in sync, even if a minority of them are in a failure state. This allows Redpanda to tolerate partial environmental failures and deliver predictable performance, even at high loads. - Redpanda provides data sovereignty. With the Bring Your Own Cloud (BYOC) offering, you deploy Redpanda in your own virtual private cloud, and all data is contained in your environment. Redpanda handles provisioning, monitoring, and upgrades, but you manage your streaming data without Redpanda’s control plane ever seeing it. ## [](#redpanda-streaming-versions)Redpanda Streaming versions You can deploy Redpanda in a self-hosted environment (Redpanda Streaming) or as a fully managed cloud service (Redpanda Cloud). Redpanda Streaming version numbers follow the convention AB.C.D, where AB is the two-digit year, C is the feature release, and D is the patch release. For example, version 22.3.1 indicates the first patch release on the third feature release of the year 2022. Patch releases include bug fixes and minor improvements, with no change to user-facing behavior. New and enhanced features are documented with each feature release. Redpanda Cloud releases on a continuous basis and uptakes Redpanda Streaming versions. --- # Page 579: Partner Integrations **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/partner-integration.md --- # Partner Integrations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Partner Integrations latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: partner-integration page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: partner-integration.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/partner-integration.adoc description: Learn about Redpanda integrations built and supported by our partners. page-git-created-date: "2024-07-25" page-git-modified-date: "2026-05-26" --- Learn about Redpanda integrations built and supported by our partners. | Partner | Description | More information | | --- | --- | --- | | Superstream | Superstream optimizes and improves Redpanda (and other Kafka platforms) for cost reduction, increased reliability, and improved visibility. | Superstream for Redpanda | | Aklivity Zilla | Zilla is a multi-protocol proxy that abstracts Redpanda for non-native clients, such as browsers and IoT devices, by exposing Redpanda topics using user-defined REST, Server-Sent Events (SSE), MQTT, or gRPC API entry points. | Modern Eventing with CQRS, Redpanda and Zilla | | Bytewax | Bytewax is an open source framework and distributed stream processing engine in Python. | Enriching streaming data with Bytewax and Redpanda | | ClickHouse | ClickHouse is a high-performance, column-oriented SQL database management system (DBMS) for online analytical processing (OLAP). | Building an OLAP database with ClickHouse and Redpanda | | Conduktor | Conduktor provides simple, flexible, and powerful tooling for Kafka developers and infrastructure. | Conduktor & Redpanda: Best of breed Kafka experience | | Decodable | Decodable is a real-time data processing platform powered by Apache Flink and Debezium. | Decodable + Redpanda | | ElastiFlow | ElastiFlow captures and analyzes flow and SNMP data to provide detailed insights into network performance and security. | Leveraging Redpanda for Enhanced Network Observability: ElastiFlow Integration | | Materialize | Materialize is a data warehouse purpose-built for operational workloads where an analytical data warehouse would be too slow, and a stream processor would be too complicated. | Ingesting data from Redpanda with Materialize | | PeerDB | PeerDB provides a fast, simple, and cost-effective way to replicate data from Postgres to data warehouses, queues and storage. | Quickstart guide | | Pinecone | Pinecone is a vector database for building accurate and performant AI applications at scale. The Pinecone connector for Redpanda Connect provides a production-ready integration from many existing data sources through simple YAML configuration. | Redpanda Connect integration | | RisingWave | RisingWave is a distributed SQL streaming database that enables simple, efficient, and reliable processing of streaming data. | Ingesting data from Redpanda with Risingwave | | Timeplus | Timeplus is a stream processor that provides powerful end-to-end capabilities, leveraging the open source streaming engine Proton. | Realizing low latency streaming analytics with Timeplus and Redpanda | | Tinybird | Tinybird is a data platform for data and engineering teams to solve complex real-time, operational, and user-facing analytics use cases at any scale. | Building a complete IoT backend with Redpanda and Tinybird | | Quix | Quix is a complete platform for building, deploying, and monitoring stream processing pipelines in Python. | Integrating Redpanda with Quix | | Yugabyte | YugabyteDB is an open-source, distributed SQL database that combines the capabilities of relational databases with the scalability of NoSQL systems. | How to Integrate Yugabyte CDC Connector with Redpanda | ## [](#how-to-contribute-to-this-page)How to contribute to this page To request a partner integration with Redpanda Data, reach out to ([partners@redpanda.com](mailto:partners@redpanda.com\)). Provide a link to your product documentation or a blogpost explaining how your product integrates with Redpanda. After meeting these requirements, you can [contribute to this page](https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/partner-integration.adoc). --- # Page 580: What’s New in Redpanda Cloud **URL**: https://docs.redpanda.com/cloud-data-platform/get-started/whats-new-cloud.md --- # What’s New in Redpanda Cloud > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: What’s New in Redpanda Cloud latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: whats-new-cloud page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: whats-new-cloud.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/get-started/pages/whats-new-cloud.adoc description: Summary of new features in Redpanda Cloud. page-git-created-date: "2024-06-06" page-git-modified-date: "2026-08-13" --- This page lists new features added to Redpanda Cloud. ## [](#august-2026)August 2026 ### [](#redpanda-sql-on-gcp)Redpanda SQL on GCP Redpanda SQL is now available on GCP, for both BYOC and BYOVPC clusters. Run real-time SQL queries on your Redpanda topics, including the Iceberg history of Iceberg-enabled topics through a GCP Lakehouse (formerly BigLake) catalog, using standard PostgreSQL syntax. - On a BYOC cluster, enable the SQL engine from the Cloud Console, Cloud API, or Terraform provider. See [Enable Redpanda SQL](https://docs.redpanda.com/cloud-data-platform/sql/get-started/deploy-sql-cluster/). - On a BYOVPC cluster, provision the SQL-specific GCP resources with the Redpanda BYOVPC Terraform module, then supply them as customer-managed resources when enabling the engine. See [Enable Redpanda SQL on a BYOVPC Cluster on GCP](https://docs.redpanda.com/cloud-data-platform/sql/get-started/enable-sql-byovpc-gcp/). ### [](#redpanda-sql-byovpc-support-on-aws)Redpanda SQL: BYOVPC support on AWS Redpanda SQL is now available on BYOVPC clusters on AWS. Provision the required SQL-specific AWS resources using the Redpanda BYOVPC Terraform module, then supply them as customer-managed resources when enabling the SQL engine via the Cloud Console, Cloud API, or Terraform provider. See [Enable Redpanda SQL on a BYOVPC Cluster on AWS](https://docs.redpanda.com/cloud-data-platform/sql/get-started/enable-sql-byovpc-aws/). ### [](#decode-iceberg-keys-values-and-headers-independently)Decode Iceberg keys, values, and headers independently On BYOC and BYOVPC clusters with Iceberg-enabled topics, the `redpanda.iceberg.mode` topic property now accepts a section-based syntax that lets you independently control how Redpanda translates the record key, value, and headers into the Iceberg table. You can decode keys and header values using a schema or store them as UTF-8 strings, instead of the default raw bytes. See [Configure key, value, and header translation](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#configure-key-value-and-header-translation). ### [](#resolve-iceberg-topic-schemas-within-a-schema-registry-context)Resolve Iceberg topic schemas within a Schema Registry context On BYOC and BYOVPC clusters, you can now set the `redpanda.schema.registry.context` topic property on an Iceberg-enabled topic to resolve its schemas within a specific [Schema Registry context](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-contexts/) instead of the default context. See [Resolve schemas within a Schema Registry context](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#resolve-schemas-within-a-context). ### [](#expanded-kafka-client-validation)Expanded Kafka client validation Redpanda now validates additional non-Java Kafka clients at their current versions, aligned with Kafka 4.x: confluent-kafka-go, Sarama, and confluent-kafka-python. See [Kafka Compatibility](https://docs.redpanda.com/cloud-data-platform/develop/kafka-clients/). ## [](#july-2026)July 2026 ### [](#schema-registry-contexts-enabled-by-default)Schema Registry contexts enabled by default On BYOC and Dedicated clusters, [Schema Registry contexts](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-contexts/) are now enabled by default, so you can register and manage schemas in isolated namespaces without an enablement step. See [Upgrade considerations](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-contexts/#upgrade-considerations) if any of your existing subject names start with `:.`. ### [](#serdes-client-support-for-schema-registry-contexts)SerDes client support for Schema Registry contexts Any SerDes client, in any language, can now target a Schema Registry context through the base URL alone: append `/contexts/{context}` to your cluster’s Schema Registry URL in the client’s `schema.registry.url` setting. Schema ID lookups sent through the prefix are scoped to that context automatically. Previously, only the Java Confluent SerDes could target non-default contexts. See [Use contexts with SerDes clients](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-contexts/#serdes-clients). ### [](#egress-allowlist-for-redpanda-connect-pipelines)Egress allowlist for Redpanda Connect pipelines You can now allow Redpanda Connect pipelines on BYOC and Dedicated clusters to open outbound connections to destinations that the data plane firewall blocks by default. Add up to 16 CIDR and port range combinations to a cluster with the `redpanda_connect.allowed_destination_cidr_ports` field in the Cloud API, or with the same attribute on a `redpanda_cluster` resource in the Redpanda Terraform provider (v2.1.1+). See [Configure Egress for Redpanda Connect Pipelines](https://docs.redpanda.com/cloud-data-platform/networking/connect-egress-allowlist/). ### [](#cloud-topics-enabled-by-default)Cloud Topics enabled by default [Cloud Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/cloud-topics/) are now enabled by default: create a Cloud Topic by setting its storage mode during topic creation, with no cluster-level enablement step. You can use Cloud Topics exclusively or together with standard topics on the same cluster. ### [](#new-aws-regions-for-byoc)New AWS regions for BYOC For [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions), Redpanda added support for the following AWS regions: - ap-northeast-2 (Seoul) - ap-south-2 (Hyderabad) ### [](#centralized-egress-for-byoc-on-azure-beta)Centralized egress for BYOC on Azure: beta You can route all Azure BYOC cluster egress through your own Azure Firewall and hub VNet instead of a per-cluster NAT Gateway, so outbound traffic exits through your centralized inspection point. This is useful for regulated environments that require a single, predictable public IP for outbound allowlisting or that prohibit per-cluster NAT Gateways. You can set this up in the Cloud UI or Cloud API when you create the network, or add or change it on an existing network through the Cloud API. Centralized egress is in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release and is enabled per organization. Contact your account team for access. See [Configure Centralized Egress with Azure Firewall](https://docs.redpanda.com/cloud-data-platform/networking/byoc/azure/nat-free-egress/). ### [](#improved-serverless-trial-onboarding)Improved Serverless trial onboarding New Serverless free-trial users now get a guided start in the Redpanda Cloud UI. After you sign up, a few quick questions help Redpanda tailor your experience. The welcome cluster’s **Overview** page summarizes your trial credits, and the **Get started** button offers options to start streaming: create a Redpanda Connect pipeline, use `rpk` from the command line, or connect with your own Kafka client. See [Get started with Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#get-started-with-serverless). ### [](#sql-editor-in-redpanda-cloud-console)SQL editor in Redpanda Cloud Console Redpanda Cloud Console now includes a built-in SQL editor. For SQL-enabled environments, you can write and run `SELECT` queries against your Redpanda topics directly in the Console without installing a separate PostgreSQL client. The editor supports syntax highlighting, autocomplete, query history, and CSV and JSON export. For Iceberg-enabled topics, an indicator shows when results span both live topic data and Iceberg-committed history. To enable Redpanda SQL, see [Enable Redpanda SQL](https://docs.redpanda.com/cloud-data-platform/sql/get-started/deploy-sql-cluster/). To use the editor, see [Use the SQL Editor](https://docs.redpanda.com/cloud-data-platform/sql/query-data/sql-editor/). ### [](#self-service-organization-deletion-for-serverless)Self-service organization deletion for Serverless You can now permanently delete a Serverless organization (free trial and pay-as-you-go plans) directly from the Cloud UI, without contacting Redpanda Support. The new **Manage organization** page, available from your profile icon, shows your plan and organization-wide resource counts, and walks you through the deletion prerequisites: delete all clusters, delete all private links, and settle any pending or outstanding invoices. See [Manage Your Organization](https://docs.redpanda.com/cloud-data-platform/manage/manage-organization/). ### [](#redpanda-connect-updates)Redpanda Connect updates - Inputs: - [jira](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/jira/): Streams Jira issues, comments, or changelog entries via JQL with incremental polling. ## [](#june-2026)June 2026 ### [](#terraform-provider-redpanda-sql-support)Terraform provider: Redpanda SQL support The Redpanda Terraform provider (v2.1.0+) now supports enabling and managing Redpanda SQL on BYOC clusters on AWS. Use the `rpsql` block on a `redpanda_cluster` resource to enable SQL, configure compute replicas, and retrieve the SQL endpoint URL. See [Enable Redpanda SQL on a BYOC cluster](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#enable-redpanda-sql-on-a-byoc-cluster). ### [](#terraform-provider-secrets-management)Terraform provider: Secrets management The Redpanda Terraform provider (v2.0.0+) now supports managing cluster secrets with the new `redpanda_secret` resource. The secret value is a write-only attribute, so Terraform never stores it in your state file, and scopes control which features can use the secret, such as Redpanda Connect pipelines. See [Manage cluster secrets](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#manage-cluster-secrets). ### [](#terraform-provider-shadow-link-management)Terraform provider: Shadow link management The Redpanda Terraform provider (v2.0.0+) now supports managing shadow links for disaster recovery on BYOC and Dedicated clusters with the new `redpanda_shadow_link` resource. Define the link between source and shadow clusters, client connection settings, and topic, consumer group, ACL, and Schema Registry synchronization options in code, with the source cluster password stored as a cluster secret rather than in your state file. See the [`redpanda_shadow_link` reference](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/shadow_link) and [Configure Shadowing](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/setup/). ### [](#centralized-egress-for-byoc-on-gcp-beta)Centralized egress for BYOC on GCP: beta You can route all GCP BYOC cluster egress through your own GCP hub VPC and NAT VM instead of a per-cluster Cloud NAT, so outbound traffic exits through your centralized inspection point. This is useful for regulated environments that require a single, predictable public IP for outbound allowlisting or that prohibit per-cluster Cloud NAT. Centralized egress is in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release and is enabled per organization. Contact your account team for access. See [Configure Centralized Egress with GCP VPC Peering](https://docs.redpanda.com/cloud-data-platform/networking/byoc/gcp/nat-free-egress/). ### [](#gcp-lakehouse-catalog-for-iceberg-topics)GCP Lakehouse catalog for Iceberg topics BYOC clusters on GCP can now use GCP Lakehouse as an Iceberg REST catalog. See [Use Iceberg Topics with GCP Lakehouse](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-gcp-biglake/). ### [](#customer-managed-default-topic-settings)Customer-managed default topic settings You can now set cluster-wide defaults for new topics on BYOC and Dedicated clusters. The [`default_topic_partitions`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#default_topic_partitions), [`log_retention_ms`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#log_retention_ms), and [`retention_bytes`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#retention_bytes) cluster properties are now customer-managed. The default topic retention period (`log_retention_ms`) previously could only be changed by Redpanda support. See [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). ### [](#redpanda-connect-updates-2)Redpanda Connect updates - Processors: - [try\_catch](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/try_catch/): Combines the behavior of the `try` and `catch` processors into a single block with an explicit recovery path. ## [](#may-2026)May 2026 ### [](#redpanda-console-redesigned-security-page)Redpanda Console: redesigned Security page Redpanda Console has a redesigned Security page with three tabs (**Users**, **Roles**, and **Permissions**). Each user and role has a detail page for managing its permissions. - The **Users** tab lists each user with their assigned roles and a count of their ACLs. Filter the list by name using regular expressions; for example, `^prod-` matches every user starting with `prod-`. - Open a user or role to manage permissions on its detail page. The **ACLs** section shows one row per rule, with columns for type, resource, operation, permission, and host, and supports three actions: - **\+ Add ACL** opens a focused form where you specify the resource type, pattern type, resource name, operation, permission, and host. - **Allow all operations** grants full wildcard access across all resource types in a single step. Use this for testing only; it is too broad for production. - Select rows with the checkboxes and click **Delete selected** to remove ACLs in bulk. - The **Permissions** tab is a unified, cluster-wide view of every principal with ACLs. Each row shows direct ACL counts and ACLs inherited from roles, with a red badge highlighting any principal that has Deny rules. Expand a row to see all of that principal’s ACLs in one table: direct rules first, then sections labeled **VIA ROLE: ** for each role they inherit from. Search across principals, resources, and roles, or click **Create ACL** to add a rule from scratch. See [Configure ACLs](https://docs.redpanda.com/cloud-data-platform/security/authorization/acl/) for the full ACL reference and [Configure RBAC in the Data Plane](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac_dp/) for role management. ### [](#redpanda-sql)Redpanda SQL Redpanda SQL is available on BYOC clusters running on AWS. Run real-time SQL queries on Redpanda topic data, including the Iceberg history of Iceberg-enabled topics, using standard PostgreSQL syntax. Connect with `psql` or any PostgreSQL driver. See the [Quickstart](https://docs.redpanda.com/cloud-data-platform/sql/get-started/sql-quickstart/) and [Overview](https://docs.redpanda.com/cloud-data-platform/sql/get-started/overview/). ### [](#centralized-egress-for-byoc-on-aws-beta)Centralized egress for BYOC on AWS: beta You can route all BYOC cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway, so outbound traffic exits through your centralized inspection point. This is useful for regulated environments that prohibit per-VPC NAT Gateways and for consolidating egress behind a single, predictable public IP for outbound allowlisting. Centralized egress is in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release and is enabled per organization. Contact your account team for access. See [Configure Centralized Egress with AWS Transit Gateway](https://docs.redpanda.com/cloud-data-platform/networking/byoc/aws/nat-free-egress/). ### [](#schema-registry-authorization-enabled-by-default)Schema Registry Authorization enabled by default Schema Registry Authorization is now enabled automatically on all new BYOC and Dedicated clusters. The [`schema_registry_enable_authorization`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#schema_registry_enable_authorization) cluster property is set to `true` at provisioning, and the predefined Admin, Writer, and Reader roles include Schema Registry permissions for the `subject` and `registry` ACL resource types. You can use ACLs and RBAC roles to grant fine-grained access to schemas and subjects without any additional setup. See [Schema Registry Authorization](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-authorization/) and [Predefined roles](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/#predefined-roles). ### [](#account-impersonation-schema-registry-support)Account impersonation: Schema Registry support [Account impersonation](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#account-impersonation) now supports Schema Registry in addition to the Kafka API. With Schema Registry impersonation enabled, the schemas and subjects users see in the Redpanda Cloud UI match exactly what they can access with the Cloud API or `rpk`. You can enable impersonation independently for each subsystem from the **Dataplane settings** page. ### [](#redpanda-connect-updates-3)Redpanda Connect updates - Inputs: - [salesforce](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce/): Executes a SOQL query against Salesforce and emits one message per record. - [salesforce\_cdc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc/): Captures change data from Salesforce objects using the Pub/Sub API. - [salesforce\_graphql](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql/): Executes GraphQL queries against Salesforce. - Outputs: - [gcp\_bigquery\_write\_api](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_bigquery_write_api/): Writes records to BigQuery using the Storage Write API for higher throughput and lower latency than the streaming insert API. - Metrics: - [open\_telemetry\_collector](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/metrics/open_telemetry_collector/): Pushes metrics using the OpenTelemetry Protocol (OTLP) over HTTP or gRPC. - Tracers: - [open\_telemetry\_collector](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/open_telemetry_collector/): Pushes tracing data using the OpenTelemetry Protocol (OTLP) over HTTP or gRPC. - Behavior changes: - [`gcp_bigquery_write_api`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/gcp_bigquery_write_api/) output: The default value of `max_in_flight` has been reduced from `64` to `4`. If you rely on the previous behavior for throughput, set `max_in_flight` explicitly in your pipeline configuration. The output also gained new fields for tuning Storage Write API behavior: `max_cached_streams`, `schema_resolve_timeout`, and `schema_evolution_timeout`. - New field support: - Kafka producer tuning fields on the [`kafka_franz`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/kafka_franz/) and [`redpanda`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda/) outputs: `acks`, `max_in_flight_requests`, `max_buffered_records`, `max_buffered_bytes`, `record_retries`, and `record_delivery_timeout`. Use these to align pipeline producers with broker durability and back-pressure requirements. - Removed components: - `salesforce` processor (introduced in April 2026): Replaced with the dedicated [`salesforce`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce/), [`salesforce_cdc`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc/), and [`salesforce_graphql`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql/) inputs. The [`salesforce_sink`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/salesforce_sink/) output remains available. ### [](#extended-serverless-free-trial)Extended Serverless free trial The free trial for Redpanda Serverless now lasts 30 days, up from 14 days. The $100 (USD) credit allowance and 7-day grace period are unchanged. Sign up at [redpanda.com](https://www.redpanda.com/try-data-streaming). See [Serverless clusters](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/). ### [](#service-account-token-rate-limits)Service account token rate limits A daily limit now applies to service account access token requests for each organization. Clients that exceed the limit receive `HTTP 429` responses. Cache tokens until close to expiry to stay within the limit, and contact Redpanda Support if your workload requires a higher daily limit. See [Service account token rate limits](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#service-account-token-rate-limits). ## [](#april-2026)April 2026 ### [](#self-service-sign-up-through-google-cloud-marketplace)Self-service sign-up through Google Cloud Marketplace You can now subscribe to Redpanda Cloud directly through Google Cloud Marketplace with pay-as-you-go billing, with no sales contact required. Self-service sign-up provisions Serverless and Dedicated clusters only. New subscriptions receive $300 (USD) in free credits to spend in the first 30 days. See [Use GCP Pay As You Go](https://docs.redpanda.com/cloud-data-platform/billing/gcp-pay-as-you-go/). ### [](#iceberg-configurable-table-namespace)Iceberg: Configurable table namespace You can now set a custom namespace for Iceberg tables instead of the default `redpanda` namespace, using the `iceberg_default_catalog_namespace` cluster property. A custom namespace is useful when multiple clusters write to the same catalog provider (such as AWS Glue), because each cluster must use a distinct namespace to avoid table name collisions. This property must be set when you first enable Iceberg and cannot be changed afterward. See [Enable Iceberg integration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration). ### [](#group-based-access-control-gbac)Group-based access control (GBAC) - With [GBAC in the control plane](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/gbac/), you can manage access to organization-level resources using OIDC groups from your identity provider. Assign OIDC groups to roles so that users inherit access based on their group membership. - With [GBAC in the data plane](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/gbac_dp/), you can configure cluster-level permissions for provisioned users at scale using OIDC groups. Because group membership is managed by your identity provider, onboarding and offboarding require no changes in Redpanda. GBAC is available for BYOC and Dedicated clusters. In addition to the predefined roles (including Reader, Writer, and Admin) that you cannot modify or delete, you can now create custom roles. ### [](#remote-mcp-deprecated)Remote MCP: Deprecated Remote MCP has been deprecated and removed from Redpanda Cloud. ### [](#increased-serverless-limits-for-redpanda-connect-pipelines)Increased Serverless limits for Redpanda Connect pipelines Serverless clusters now support up to 100 Redpanda Connect pipelines. See [Serverless usage limits](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/#_serverless_usage_limits). ### [](#terraform-provider-write-only-attributes-for-sensitive-fields)Terraform provider: Write-only attributes for sensitive fields The Redpanda Terraform provider (v1.6.0+) now supports [Terraform 1.11+ write-only attributes](https://developer.hashicorp.com/terraform/plugin/framework/resources/write-only-arguments) for sensitive fields such as user passwords and pipeline client secrets. Use the new `password_wo` and `password_wo_version` attributes (and equivalents for other sensitive fields) to keep credentials out of your `.tfstate` file. See [Manage sensitive attributes with write-only fields](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#manage-sensitive-attributes-with-write-only-fields). ### [](#redpanda-connect-updates-4)Redpanda Connect updates - The Redpanda Connect pipeline creation and editing workflow has been simplified. The new UI replaces the previous multi-page wizard with a visual pipeline diagram, an IDE-like configuration editor, slash commands for inserting variables, and inline links to component documentation. See the [Redpanda Connect quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/) to try it out. - Outputs: - [arc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/arc/): Send data to an Arc columnar analytical database using its high-performance MessagePack ingestion endpoint. - [salesforce\_sink](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/salesforce_sink/): Write messages to Salesforce, routing each Kafka topic to its own sObject configuration. Supports both realtime (sObject Collections REST API) and bulk modes (Bulk API 2.0). - Processors: - [string\_split](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/string_split/): Splits strings into multiple parts using a delimiter, creating new messages or fields for each part. - `salesforce` (deprecated May 2026): Replaced with dedicated [salesforce](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce/), [salesforce\_cdc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_cdc/), and [salesforce\_graphql](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/salesforce_graphql/) inputs. ## [](#march-2026)March 2026 Redpanda Console now supports paginating past the previous 500-record cap when you browse topic messages, so you can inspect large topics without being limited to the initial result set. See [Paginate Messages in Redpanda Console](https://docs.redpanda.com/cloud-data-platform/develop/consume-data/paginate-messages-events/). ### [](#redpanda-connect-updates-5)Redpanda Connect updates - Inputs: - [oracledb\_cdc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/oracledb_cdc/): Stream changes from an Oracle database for Change Data Capture (CDC). - [aws\_cloudwatch\_logs](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_cloudwatch_logs/): Consume log events from AWS CloudWatch Logs. Supports filtering by log streams, CloudWatch filter patterns, and configurable start times. - [aws\_dynamodb\_cdc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/aws_dynamodb_cdc/): Consume item-level changes from DynamoDB Streams with automatic checkpointing and shard management. - Outputs: - [iceberg](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/iceberg/): Write data to Apache Iceberg tables using the REST catalog. - Bloblang methods: - [`escape_url_path`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#escape_url_path): Escapes a string for safe use in URL path segments using percent-encoding. - [`parse_logfmt`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#parse_logfmt): Parses a logfmt-encoded string into an object of key-value pairs. - [`unescape_url_path`](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/bloblang/methods/#unescape_url_path): Unescapes a URL path segment, converting percent-encoded sequences back to their original characters. - Removed components: - `legacy_redpanda_migrator` input and output - `legacy_redpanda_migrator_offsets` input and output - `redpanda_migrator_bundle` input and output Use the unified [`redpanda_migrator`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/redpanda_migrator/) input and [`redpanda_migrator`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/redpanda_migrator/) output instead. ### [](#cloud-topics)Cloud Topics [Cloud Topics](https://docs.redpanda.com/cloud-data-platform/develop/topics/cloud-topics/) are now available, making it possible to use durable cloud storage (S3, ADLS, GCS) as the primary backing store instead of local disk, eliminating over 90% of cross-AZ replication costs. This makes them ideal for latency-tolerant, high-throughput workloads such as observability streams, analytics pipelines, and AI/ML training data feeds, where cross-AZ networking charges are the dominant cost driver. You can use Cloud Topics exclusively or in combination with standard topics on a cluster supporting low-latency workloads. ### [](#schema-registry-metadata-properties)Schema Registry metadata properties [Schema Registry metadata properties](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/#metadata-properties) let you store and retrieve arbitrary key-value pairs alongside schemas. Properties such as `owner`, `team`, or `application.version` travel with the schema through its lifecycle, making it easier to track ownership and lineage without modifying the schema itself. You can set metadata when registering a schema using the `POST /subjects/{subject}/versions` endpoint or with the `--metadata-properties` flag in `rpk registry schema create`. Metadata is returned in API responses and viewable with `rpk registry schema get --print-metadata` or in Redpanda Cloud Console. ### [](#schema-registry-contexts)Schema Registry Contexts [Schema Registry contexts](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-contexts/) provide isolated namespaces that separate schemas, subjects, and configuration within a single Schema Registry instance. Each context maintains its own schema ID counter, mode settings, and compatibility settings. On Serverless clusters, Redpanda uses contexts internally for per-tenant isolation. Contexts are not exposed to end users on Serverless. On BYOC and Dedicated clusters, contexts are available and user-configurable. ### [](#user-based-throughput-quotas)User-based throughput quotas Redpanda now supports throughput quotas based on authenticated user principals. Unlike client-based quotas (which rely on self-declared `client-id` values), [user-based quotas](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/#set-user-based-quotas) enforce limits using verified identities from SASL, mTLS, or OIDC authentication. You can set quotas for individual users, default users, or fine-grained user/client combinations. ### [](#iceberg-expanded-json-schema-support)Iceberg: Expanded JSON Schema support Redpanda now supports additional JSON Schema patterns when translating to Iceberg tables: - `$ref` support: Internal references using `$ref` (for example, `"$ref": "#/definitions/myType"`) are resolved from schema resources declared in the same document. External references are not yet supported. - Map type from `additionalProperties`: `additionalProperties` objects that contain subschemas now translate to Iceberg `map`. - `oneOf` nullable pattern: The `oneOf` keyword is now supported for the standard nullable pattern if exactly one branch is `{"type":"null"}` and the other is a non-null schema. See [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#how-iceberg-modes-translate-to-table-format) for JSON types mapping and updated requirements. ### [](#ordered-rack-preference-for-leader-pinning)Ordered rack preference for leader pinning [Leader pinning](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/leader-pinning/) now supports the `ordered_racks` configuration value, which lets you specify preferred racks in priority order. Unlike `racks`, which distributes leaders uniformly across all listed racks, `ordered_racks` places leaders in the highest-priority available rack and fails over to subsequent racks only when higher-priority racks become unavailable. ### [](#byovpc-on-aws-ga)BYOVPC on AWS: GA [BYOVPC on AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/) is now generally available (GA). With Bring Your Own VPC (BYOVPC), you deploy the Redpanda data plane into your own VPC and manage security policies and resources yourself, including subnets, IAM roles, firewall rules, and storage buckets. The Redpanda BYOVPC Terraform Module contains Terraform code that deploys the resources required for a BYOVPC cluster on AWS. Secrets management is enabled by default with the Terraform module. ### [](#iceberg-topics-with-snowflake-open-catalog-ga)Iceberg topics with Snowflake Open Catalog: GA The [Snowflake and Open Catalog integration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog/) for Iceberg topics is now generally available (GA). ### [](#billing-notifications)Billing notifications Redpanda Cloud now sends email notifications to organization admins when credit or commit balances reach spending thresholds (50%, 30%, 10%, and 0% remaining). You can manage your notification preferences or opt out at any time. See [Manage Billing Notifications](https://docs.redpanda.com/cloud-data-platform/billing/billing-notifications/). ## [](#february-2026)February 2026 ### [](#serverless-on-aws-ga)Serverless on AWS: GA [Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) on AWS is now generally available (GA). This release includes private networking with AWS PrivateLink. You can use the Cloud Console, the Cloud API, or the Redpanda Terraform provider to create and manage Serverless private links. Serverless is the easiest and fastest way to begin streaming data with Redpanda. ### [](#enable-schema-id-validation)Enable schema ID validation You can now enable [schema ID validation](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-id-validation/) by [configuring the `enable_schema_id_validation` cluster property](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). This controls whether or not Redpanda validates schema IDs in records and which topic properties are enforced. Use caution when enabling this property, because it could cause decompression across topics and increase CPU load. ### [](#cross-region-aws-privatelink)Cross-region AWS PrivateLink AWS PrivateLink now supports cross-region connectivity, allowing clients in different AWS regions to connect to your Redpanda cluster through PrivateLink. Configure supported regions in the [Cloud Console](https://docs.redpanda.com/cloud-data-platform/networking/configure-privatelink-in-cloud-ui/#cross-region-privatelink), using the [Cloud API](https://docs.redpanda.com/cloud-data-platform/networking/aws-privatelink/#cross-region-privatelink), or with the Redpanda Terraform provider (v1.7.0+) using the `supported_regions` attribute of the `aws_private_link` block. See [Configure cross-region AWS PrivateLink](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#configure-cross-region-aws-privatelink). This feature requires multi-AZ cluster deployments. ## [](#january-2026)January 2026 ### [](#redpanda-connect-updates-6)Redpanda Connect updates - Inputs: - [otlp\_grpc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/otlp_grpc/): Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Exposes an OpenTelemetry Collector gRPC receiver that accepts traces, logs, and metrics, converting them to individual Redpanda OTEL v1 protobuf messages optimized for Kafka partitioning. - [otlp\_http](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/otlp_http/): Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Supports both protobuf and JSON formats at standard OTLP endpoints, converting telemetry data to individual messages with embedded Resource and Scope metadata. - Outputs: - [otlp\_grpc](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/otlp_grpc/): Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Accepts batches of Redpanda OTEL v1 protobuf messages and converts them to OTLP format for transmission to OpenTelemetry collectors. - [otlp\_http](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/otlp_http/): Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Supports both protobuf and JSON content types for flexible integration with OpenTelemetry backends. ### [](#redpanda-connect-and-roles-in-terraform-provider)Redpanda Connect and Roles in Terraform provider The [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) now supports managing roles and Redpanda Connect pipelines. Use the provider to create and manage role-based access control and data pipelines in Redpanda Cloud. ## [](#december-2025)December 2025 ### [](#shadowing)Shadowing Redpanda Cloud now supports [Shadowing](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/), a disaster recovery solution that provides asynchronous, offset-preserving replication between distinct Redpanda clusters. Shadowing enables cross-region data protection by replicating topic data, configurations, consumer group offsets, ACLs, and Schema Registry data with byte-level fidelity. The shadow cluster operates in read-only mode while continuously receiving updates from the source cluster. During a disaster, you can failover individual topics or an entire shadow link to make resources fully writable for production traffic. Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. ### [](#metrics-for-serverless)Metrics for Serverless You can now view and export metrics from Serverless clusters to third-party monitoring systems like Prometheus and Grafana. See [Monitor Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/) for details on configuring monitoring for your Serverless cluster and [Metrics Reference](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/) for a list of metrics available in Serverless. ### [](#account-impersonation)Account impersonation BYOC and Dedicated clusters now support unified authentication and authorization between the Redpanda Cloud UI and Redpanda with [account impersonation](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#account-impersonation). This means you can authenticate to fine-grained access within Redpanda using the same credentials you use to authenticate to Redpanda Cloud. With account impersonation (originally called user impersonation), the topics users see in the UI are identical to what they can access with the Cloud API or `rpk`, ensuring consistent permissions across all interfaces and clear auditing of data plane user actions. ### [](#redpanda-connect-updates-7)Redpanda Connect updates - Tracers: - [Redpanda](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/tracers/redpanda/): The Redpanda tracer exports distributed tracing data to a Redpanda topic, enabling you to monitor and debug your Redpanda Connect pipelines. Traces are exported in OpenTelemetry format as JSON, allowing integration with observability platforms like Jaeger, Grafana Tempo, or custom trace consumers. ## [](#november-2025)November 2025 ### [](#serverless-on-gcp-beta)Serverless on GCP: beta You can now create [Serverless clusters](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) on Google Cloud Platform (GCP). Serverless on GCP is in a [beta](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#beta) release. ### [](#support-for-additional-regions)Support for additional regions [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions) on Azure now support the Sweden Central and Germany West Central regions. ### [](#connected-client-monitoring)Connected client monitoring You can view details about Kafka client connections using `rpk` or the Data Plane API. This allows you to view detailed information about active client connections on a cluster, and identify and troubleshoot problematic clients. For more information, see the [connected client details](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/#view-connected-client-details) example in the Manage Throughput guide. ### [](#increased-message-size-limit)Increased message size limit Redpanda Cloud increased the [message size limit](https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic/) on newly-created topics. BYOC and Dedicated clusters have a default message size limit of 20 MiB with a maximum of 32 MiB. Serverless clusters have a default message size limit of 8 MiB with a maximum of 20 MiB. Configure the message size limit with the `max.message.bytes` topic property. The message size setting on existing topics is not changed, but the message size limit on existing topics can only be updated to the new maximum. ### [](#redpanda-connect-updates-8)Redpanda Connect updates Redpanda Connect provides a simplified [quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/) experience in the UI that helps you to start building data pipelines. The quickstart creates pipelines to stream data into and out of Redpanda using the pipeline editor. ### [](#get-started-with-serverless)Get Started with Serverless A Serverless cluster’s **Overview** page now provides a **Get Started** guide to help you start streaming your own data with a [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/) pipeline. It lets you stream data into and out of Redpanda without writing producer/consumer code. ### [](#remote-read-replicas-ga)Remote read replicas: GA [Remote read replicas](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/remote-read-replicas/) are now generally available (GA) for BYOC clusters on AWS and GCP. This feature allows you to create read-only topics that mirror a topic on a different cluster, providing greater flexibility and scalability for your data streaming needs. ### [](#schema-registry-and-acls-in-terraform-provider)Schema Registry and ACLs in Terraform provider The [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) now supports managing schemas and Schema Registry ACLs. You can use the provider to register schemas in formats such as Avro, Protobuf, or JSON Schema, and control access to Schema Registry subjects and operations through ACLs. ## [](#october-2025)October 2025 ### [](#api-gateway-access)API Gateway access BYOC and Dedicated clusters with private networking now allow control of API Gateway network access, independent of the Redpanda cluster. When you create a cluster, you can choose either public or private access for the API Gateway: - Public access exposes Redpanda Console, Data Plane API, and MCP Server API endpoints over the internet, although they remain protected by your authentication and authorization controls. - Private access restricts endpoint access to your private network (VPC or VNet) only. After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. ### [](#redpanda-connect-updates-9)Redpanda Connect updates - Inputs: - [Microsoft SQL Server CDC](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/microsoft_sql_server_cdc/): Streams change data from a Microsoft SQL Server database into Redpanda Connect using Change Data Capture (CDC). - Outputs: - [CyborgDB](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/cyborgdb/): Write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. - Processors: - [`jira`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/processors/jira/): Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. - Deprecated components: - `redpanda_migrator` input and output (renamed to `legacy_redpanda_migrator`) - `redpanda_migrator_offsets` input and output (renamed to `legacy_redpanda_migrator_offsets`) Migrate from these deprecated components to the new unified `redpanda_migrator` input/output pair. For detailed migration instructions, see [Migrate to the Unified Redpanda Migrator](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/migrate-unified-redpanda-migrator/). - `redpanda_migrator_bundle` input and output (these are part of the legacy migration architecture and internally depend on the deprecated `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets` components) - `kafka`, `kafka_franz`, and `redpanda_common` inputs and outputs. These components have been consolidated into the unified `redpanda` input and output components. Migrate existing configurations to use the new `redpanda` components for continued support and access to the latest features. For detailed information about recent component updates, see [What’s New in Redpanda Connect](https://docs.redpanda.com/connect/get-started/whats-new/). ## [](#september-2025)September 2025 ### [](#multi-factor-authentication)Multi-factor authentication Enable multi-factor authentication (MFA) to add an extra layer of security to your Redpanda Cloud account. After you enable MFA, you’ll enter your credentials, then be prompted for a one-time code from your authenticator app when you log in. Administrators can also [enforce MFA](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#multi-factor-authentication-mfa) for all members of an organization. ### [](#redpanda-cloud-management-mcp-server-beta)Redpanda Cloud Management MCP Server: beta Connect AI assistants like Claude directly to your Redpanda Cloud account with the new Redpanda Cloud Management MCP Server. This server runs on your computer and provides AI tools for managing clusters, topics, and other cloud resources through natural language commands. Ask your AI assistant to "Create a new topic called user-events" or "List all clusters in my account" and it will handle the technical details automatically. Get started with the [rpk cloud mcp install](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cloud/rpk-cloud-mcp-install/) command. The Redpanda Cloud Management MCP Server uses the Model Context Protocol (MCP) to extend AI assistants with Redpanda-specific capabilities, making cloud operations more accessible through conversational interfaces. ### [](#automatic-topic-creation-and-topic-limit)Automatic topic creation and topic limit For BYOC and Dedicated clusters, you can now configure the `auto_create_topics_enabled` cluster property to automatically create a topic if a client produces to a non-existent topic. For all clusters: each cluster now has a limit of 40,000 topics. ## [](#august-2025)August 2025 ### [](#manage-custom-resource-tags-in-byoc)Manage custom resource tags in BYOC After cluster creation, you can manage custom cloud provider tags and labels on BYOC and BYOVPC/BYOVNet clusters for [AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/create-byoc-cluster-aws/#manage-custom-tags), [Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/create-byoc-cluster-azure/#manage-custom-tags), and [GCP](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/create-byoc-cluster-gcp/#manage-custom-resource-labels-and-network-tags) using the Cloud Control Plane API. This involves refreshing Redpanda agent permissions with `rpk cloud byoc` due to new IAM permissions. ### [](#iceberg-topics-with-aws-glue)Iceberg topics with AWS Glue A new [integration with AWS Glue Data Catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-aws-glue/) allows you to add Redpanda topics as Iceberg tables in your data lakehouse. The AWS Glue catalog integration is available in BYOC clusters with Redpanda version 25.2 and later. See [Integrate with REST Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/rest-catalog/) for supported Iceberg REST catalog integrations. ### [](#manage-throughput)Manage throughput Redpanda Cloud now lets you [manage throughput](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/) configuration at the broker and client levels. You can manage client quotas with [`rpk cluster quotas`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas/) or with the Kafka API. When no quotas apply, the client has unlimited throughput. ## [](#july-2025)July 2025 ### [](#iceberg-topics-in-redpanda-cloud-ga)Iceberg topics in Redpanda Cloud: GA [Iceberg topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/) are now generally available (GA) in Redpanda Cloud. ### [](#byoc-on-azure-ga)BYOC on Azure: GA [BYOC for Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/create-byoc-cluster-azure/) is now generally available (GA). ### [](#schema-registry-authorization)Schema Registry Authorization You can now use [Schema Registry Authorization](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-authorization/) to control access to Schema Registry subjects and operations. Schema Registry Authorization offers more granular control over who can do what with your Redpanda Schema Registry resources. ACLs used for Schema Registry access also support RBAC roles. ### [](#kafka-connect-disabled-on-new-clusters)Kafka Connect disabled on new clusters [Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/) is now disabled by default on all new clusters. To unlock this feature for your account, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). If you previously enabled Kafka Connect on a cluster and want to [disable it](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/), you can use the Cloud API. ### [](#allowlist-nat-gateway-ip)Allowlist NAT gateway IP The [Redpanda NAT gateway IP address](https://docs.redpanda.com/cloud-data-platform/networking/cloud-security-network/#nat-gateways) is now provided in the Cloud UI and the Cloud API for BYOC and Dedicated clusters. If necessary, you can use this IP address to allowlist egress traffic from your Redpanda Connect data sources. ### [](#mtls-and-sasl-authentication-for-kafka-api-on-aws)mTLS and SASL authentication for Kafka API on AWS You can now enable mTLS and SASL authentication simultaneously for the Kafka API on AWS clusters. If you enable both mTLS and SASL on AWS clusters, Redpanda creates two distinct listeners: an mTLS listener operating on one port and a SASL listener operating on a different port. See [Authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#service-authentication) for details on available authentication methods in Redpanda Cloud. ### [](#azure-private-link-in-the-ui-ga)Azure Private Link in the UI: GA You can now [configure Azure Private Link](https://docs.redpanda.com/cloud-data-platform/networking/azure-private-link-in-ui/) for a new BYOC or Dedicated cluster using the Cloud UI. The Azure Private Link service is generally available (GA) in both the Cloud UI and the Cloud API. ### [](#redpanda-connect-in-redpanda-cloud-ga)Redpanda Connect in Redpanda Cloud: GA [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) is now generally available (GA) in all Redpanda Cloud clusters: BYOC (including BYOVPC/BYOVNet), Dedicated, and Serverless. ### [](#redpanda-connect-updates-10)Redpanda Connect updates Redpanda Connect includes the following updates: - The [GCP Spanner CDC](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gcp_spanner_cdc/) component lets you capture changes from Google Cloud Spanner and stream them into Redpanda. You can use it to ingest data from GCP Spanner databases, enabling real-time data processing and analytics. - The [Slack Reaction](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/outputs/slack_reaction/) component lets you send messages to a Slack channel in response to events in Redpanda. You can use it to create alerts, notifications, or other automated responses based on data changes in Redpanda. - The [Redpanda Cache](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/caches/redpanda/) component lets you cache data in Redpanda, improving performance and reducing latency for data access. You can use it to store frequently accessed data, such as configuration settings or user profiles, in Redpanda. For more detailed information about recent component updates, see [What’s New in Redpanda Connect](https://docs.redpanda.com/connect/get-started/whats-new/). ### [](#serverless-client-connections)Serverless client connections [Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) clusters have a new usage limit of 10,000 connections. ## [](#june-2025)June 2025 ### [](#schema-registry-ui-for-serverless)Schema Registry UI for Serverless The [Schema Registry UI](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-ui/) is now available for Serverless clusters. ### [](#amazon-vpc-transit-gateway)Amazon VPC Transit Gateway For BYOC and BYOVPC clusters on AWS, you can set up an [Amazon VPC Transit Gateway](https://docs.redpanda.com/cloud-data-platform/networking/byoc/aws/transit-gateway/) to connect VPCs to Redpanda services while maintaining control over network traffic. ### [](#support-for-additional-regions-2)Support for additional regions Serverless clusters now support the following new [regions on AWS](https://docs.redpanda.com/cloud-data-platform/reference/tiers/serverless-regions/): ap-northeast-1 (Tokyo), ap-southeast-1 (Singapore), and eu-west-2 (London). ### [](#http-gateway)HTTP gateway The [`gateway`](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/inputs/gateway/) component is now available in Redpanda Connect for Redpanda Cloud. This component allows you to create an HTTP endpoint that can receive data from any HTTP client and stream it into Redpanda. You can use the gateway to ingest data from IoT devices, web applications, or any other HTTP-based source. See the [Ingest Real-Time Sensor Telemetry with the HTTP Gateway](https://docs.redpanda.com/cloud-data-platform/develop/connect/guides/cloud/gateway/) guide for more information. ## [](#may-2025)May 2025 ### [](#redpanda-connect-for-byovnet-on-azure-beta)Redpanda Connect for BYOVNet on Azure: beta [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) is now enabled when you create a BYOVNet cluster on [Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/vnet-azure/). ### [](#secrets-management-for-byovpc-clusters-on-aws-and-gcp)Secrets management for BYOVPC clusters on AWS and GCP You can now create new BYOVPC clusters with secrets management enabled by default on [AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/) and [GCP](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp/). You can also enable secrets management for existing BYOVPC clusters on AWS and GCP. For GCP, see [Enable Secrets Management for BYOVPC Clusters on GCP](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/enable-secrets-byovpc-gcp/). For AWS, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). ### [](#serverless-standard-deprecated)Serverless Standard: deprecated Serverless Standard is deprecated. All existing clusters will be migrated to the new [Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) platform (with higher usage limits, 99.9% SLA, and additional regions) on August 31, 2025. - Retirement date: August 30, 2025 ### [](#cloud-api-beta-versions-deprecated)Cloud API beta versions: deprecated The Cloud Control Plane API versions v1beta1 and v1beta2, and Data Plane API versions v1alpha1 and v1alpha2 are deprecated. These Cloud API versions will be removed in a future release and are not recommended for use. The deprecation timeline is: - Announcement date: May 27, 2025 - End-of-support date: November 28, 2025 - Retirement date: May 28, 2026 See the [Cloud API Deprecation Policy](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-deprecation-policy) for more information. ### [](#read-only-cluster-configuration-properties)Read-only cluster configuration properties You can now [view the value of read-only cluster configuration properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#view-cluster-property-values) with `rpk cluster config` or with the Cloud API. Available properties are listed in [Cluster Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/) and [Object Storage Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/object-storage-properties/). ### [](#iceberg-topics-in-azure-beta)Iceberg topics in Azure: beta [Iceberg topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/) are now supported for BYOC clusters in Azure. ### [](#support-for-additional-region)Support for additional region [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions) on GCP now support the us-west2 (Los Angeles) region. ### [](#redpanda-terraform-provider-ga)Redpanda Terraform provider: GA The [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) is now generally available (GA). The provider lets you create and manage resources in Redpanda Cloud, such as clusters, topics, users, ACLs, networks, and resource groups. ## [](#april-2025)April 2025 ### [](#mtls-and-sasl-authentication-for-kafka-api-on-gcp)mTLS and SASL authentication for Kafka API on GCP You can now enable mTLS and SASL authentication simultaneously for the Kafka API on GCP clusters. If you enable both mTLS and SASL on GCP clusters, Redpanda creates two distinct listeners: an mTLS listener operating on one port and a SASL listener operating on a different port. See [Authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#service-authentication) for details on available authentication methods in Redpanda Cloud. ### [](#increased-number-of-supported-partitions)Increased number of supported partitions The number of partitions (pre-replication) Redpanda Cloud supports for each [usage tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/) has been doubled. For example, the number of supported partitions in tier 1 went from 1,000 to 2,000, and tier 5 went from 22,800 to 45,600. ### [](#iceberg-topics-beta)Iceberg topics: beta The [Iceberg integration for Redpanda](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/) allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines. Iceberg topics are supported for BYOC clusters in AWS and GCP. ### [](#cluster-configuration)Cluster configuration You can now [configure certain cluster properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/) with `rpk cluster config` or with the Cloud API. For example, you can enable and manage [Iceberg topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/), [data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/), and [audit logging](https://docs.redpanda.com/cloud-data-platform/manage/audit-logging/). Available properties are listed in [Cluster Configuration Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/). Iceberg topics properties are available for clusters running Redpanda version 25.1 or later. ### [](#manage-secrets-for-cluster-configuration)Manage secrets for cluster configuration Redpanda Cloud now supports managing secrets that you can reference in cluster properties, for example, to configure Iceberg topics. You can create, update, and delete secrets and reference a secret in cluster properties using `rpk` or the Cloud API. See also: - Manage secrets using [`rpk security secret`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-security/rpk-security-secret/) - Manage secrets using the [Data Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/#manage-secrets) - Reference a secret in a cluster property using [`rpk cluster config set`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-config-set/) - Reference a secret in a cluster property using the [Control Plane API](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/) ### [](#data-transforms-ga)Data transforms: GA WebAssembly [data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/) are now generally available in Redpanda Cloud. Data transforms let you run common data streaming tasks within Redpanda, like filtering, scrubbing, and transcoding. Data transforms are supported for BYOC and Dedicated clusters running Redpanda version 24.3 and later. ### [](#redpanda-connect-for-byovpc-on-aws-and-gcp-beta)Redpanda Connect for BYOVPC on AWS and GCP: beta Redpanda Connect is now enabled when you create a BYOVPC cluster on [AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/) or [GCP](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/vpc-byo-gcp/). You can also add Redpanda Connect to an [existing BYOVPC GCP cluster](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp/). ## [](#march-2025)March 2025 ### [](#serverless)Serverless For a better customer experience, the Serverless Standard and Serverless Pro products have merged into a single offering. [Serverless clusters](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) now include the higher usage limits, 99.9% SLA, additional AWS regions, and the free trial. ### [](#cloud-api-ga)Cloud API: GA The Cloud API is now generally available. It includes endpoints for [managing Serverless clusters](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-serverless-controlplane-api/), configuring RBAC in [BYOC](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/#manage-rbac), [Serverless](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-serverless-controlplane-api/#manage-rbac), and [Dedicated](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dedicated-controlplane-api/#manage-rbac) clusters, and [using Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/#use-redpanda-connect). To get started, see the [Redpanda Cloud API overview](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview) or try the [Cloud API Quickstart](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-quickstart). For full reference documentation, see [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) and [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/). ### [](#support-for-additional-regions-3)Support for additional regions [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions) on GCP now support the europe-southwest1 (Madrid) region. ### [](#byovpc-support-in-the-redpanda-terraform-provider-0-14-0-beta)BYOVPC support in the Redpanda Terraform provider 0.14.0: Beta The [Redpanda Terraform provider](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/cluster#byovpc) now supports BYOVPC clusters on AWS and GCP. You can use the provider to create and manage BYOVPC clusters in Redpanda Cloud. ## [](#february-2025)February 2025 ### [](#role-based-access-control-rbac)Role-based access control (RBAC) With [RBAC in the control plane](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/), you can manage access to organization-level resources like clusters, resource groups, and networks. For example, you could grant everyone access to clusters in a development resource group while limiting access to clusters in a production resource group. Or, you could limit access to geographically-dispersed clusters in accordance with data residency laws. With [RBAC in the data plane](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac_dp/), you can configure cluster-level permissions for provisioned users at scale. ### [](#improved-private-service-connect-support-with-az-affinity)Improved Private Service Connect support with AZ affinity The latest version of the Redpanda [GCP Private Service Connect](https://docs.redpanda.com/cloud-data-platform/networking/gcp-private-service-connect/) service provides the ability to allow requests from Private Service Connect endpoints to stay within the same availability zone, avoiding additional networking costs. The service is now fully supported (GA). To upgrade, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). > ❗ **IMPORTANT** > > Deprecated: The original GCP Private Service Connect service is deprecated and will be removed in a future release. ### [](#serverless-pro-usage-limits-increased)Serverless Pro usage limits increased Usage limits for Serverless Pro clusters increased to: ingress = 100 MBps, egress = 300 MBps, partitions = 5000. ### [](#cloud-api-reference)Cloud API reference The Cloud API reference is now provided as separate references for the [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) and [Data Plane APIs](https://docs.redpanda.com/api/doc/cloud-dataplane/). The Control Plane API and Data Plane APIs follow separate OpenAPI specifications, so the reference is updated to better reflect the structure of the Cloud APIs and to improve usability of the documentation. See also: [Cloud API Overview](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview). ## [](#january-2025)January 2025 ### [](#new-tiers-and-regions-on-azure)New tiers and regions on Azure [Tiers 1-5](https://docs.redpanda.com/cloud-data-platform/reference/tiers/) are now supported for BYOC and Dedicated clusters running on Azure. Also, the following [regions](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/#dedicated-supported-regions) were added for Dedicated clusters: Central US, East US 2, Norway East. ### [](#serverless-pro-la)Serverless Pro: LA Serverless Pro is a new enterprise-level cluster option. It is similar to Serverless Standard, but with higher usage limits and Enterprise support. This is a limited availability (LA) release. To start using Serverless Pro, contact [Redpanda Sales](https://redpanda.com/try-redpanda?section=enterprise-trial). ### [](#aws-privatelink-ga)AWS PrivateLink: GA AWS PrivateLink is now generally available for private networking in the [Cloud UI](https://docs.redpanda.com/cloud-data-platform/networking/configure-privatelink-in-cloud-ui/) and the [Cloud API](https://docs.redpanda.com/cloud-data-platform/networking/aws-privatelink/). ## [](#december-2024)December 2024 ### [](#support-for-additional-regions-4)Support for additional regions For [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions), Redpanda added support for the following regions: - GCP: europe-west9 (Paris), southamerica-west1 (Santiago) - AWS: ap-southeast-3 (Jakarta), eu-north-1 (Stockholm), eu-south-1 (Milan), eu-west-3 (Paris) ### [](#redpanda-connect-updates-11)Redpanda Connect updates Redpanda Connect is now available on Dedicated clusters. This is a limited availability (LA) release. [Secret management](https://docs.redpanda.com/cloud-data-platform/develop/connect/configuration/secret-management/) is also available on BYOC, Dedicated, and Serverless clusters so that you can add secrets to your pipelines without exposing them. ### [](#leader-pinning)Leader pinning For a Redpanda cluster deployed across multiple availability zones (AZs), [leader pinning](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/leader-pinning/) ensures that a topic’s partition leaders are geographically closer to clients. Leader pinning can lower networking costs and help guarantee lower latency by routing produce and consume requests to brokers located in certain AZs. ## [](#november-2024)November 2024 ### [](#byovpc-on-aws-beta)BYOVPC on AWS: beta With standard BYOC clusters, Redpanda manages security policies and resources for your VPC, including subnetworks, service accounts, IAM roles, firewall rules, and storage buckets. For the highest level of security, you can manage these resources yourself with a [BYOVPC on AWS](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/), previously known as _customer-managed VPC_. ### [](#customer-managed-vnet-on-azure-la)Customer-managed VNet on Azure: LA With standard BYOC clusters, Redpanda manages security policies and resources for your virtual network (VNet), including subnetworks, managed identities, IAM roles, security groups, and storage accounts. For the highest level of security, you can manage these resources yourself with a [customer-managed VNet on Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/vnet-azure/). Because Azure functionality is provided in limited availability, to unlock this feature, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). ## [](#october-2024)October 2024 ### [](#byoc-support-in-the-terraform-provider-0-10)BYOC support in the Terraform provider 0.10 The [Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) now supports BYOC clusters. You can use the provider to create and manage BYOC clusters in Redpanda Cloud. ### [](#azure-marketplace-for-dedicated-clusters)Azure Marketplace for Dedicated clusters You can contact [Redpanda sales](https://redpanda.com/try-redpanda?section=enterprise-trial) to request a private offer for monthly or annual [committed use through the Azure Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/azure-commit/). You can then quickly provision Dedicated clusters in Redpanda Cloud, and you can view your bills and manage your subscription directly in Azure Marketplace. ### [](#support-for-aws-graviton3)Support for AWS Graviton3 Redpanda now supports compute-optimized tiers with AWS Graviton3 processors. This saves over 50% in instance costs in all [BYOC tiers](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/). ### [](#redpanda-terraform-provider-for-redpanda-cloud-beta)Redpanda Terraform Provider for Redpanda Cloud: beta The [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) lets you create and manage resources in Redpanda Cloud, such as clusters, topics, users, ACLs, networks, and resource groups. ## [](#september-2024)September 2024 ### [](#schedule-maintenance-windows)Schedule maintenance windows Redpanda Cloud now offers greater flexibility to schedule upgrades to your cluster. By default, Redpanda Cloud may run maintenance operations on any day at any time. You can override this default and \* [schedule a maintenance window](https://docs.redpanda.com/cloud-data-platform/manage/maintenance/#maintenance-windows), which requires Redpanda Cloud to run operations on your specified day and time. ### [](#redpanda-connect-la-for-byoc-beta-for-serverless)Redpanda Connect: LA for BYOC, beta for Serverless [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) is now integrated into Redpanda Cloud and available as a fully-managed service. This is a limited availability (LA) release for BYOC and a beta release for Serverless. [Choose from a range of connectors, processors, and other components](https://docs.redpanda.com/cloud-data-platform/develop/connect/components/about/) to quickly build and deploy streaming data pipelines or AI applications from the [Cloud UI](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/) or using the [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/group/endpoint-redpanda-connect-pipeline). Comprehensive metrics, monitoring, and per pipeline scaling are also available. To start using Redpanda Connect, [try this quickstart](https://docs.redpanda.com/cloud-data-platform/develop/connect/connect-quickstart/). For more detailed information about recent component updates, see [What’s New in Redpanda Connect](https://docs.redpanda.com/connect/get-started/whats-new/). ### [](#dedicated-on-azure-la)Dedicated on Azure: LA Redpanda now supports [Dedicated clusters on Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/create-dedicated-cloud-cluster/). This is a limited availability (LA) release for Dedicated clusters. ### [](#remote-read-replicas-on-customer-managed-vpc)Remote read replicas on customer-managed VPC The beta release of [remote read replicas](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/remote-read-replicas/) has been extended to support customer-managed VPC deployments. ## [](#july-2024)July 2024 ### [](#redpanda-cloud-docs)Redpanda Cloud docs The [Redpanda Docs site](https://docs.redpanda.com/home/) has been redesigned for an easier experience navigating Redpanda Cloud docs. We hope that our docs help and inspire our users. Please share your feedback with the links at the bottom of any doc page. ### [](#byoc-on-azure-la)BYOC on Azure: LA Redpanda now supports [BYOC clusters on Azure](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/azure/create-byoc-cluster-azure/). This is a limited availability (LA) release for BYOC clusters. ### [](#enhancements-to-serverless-la)Enhancements to Serverless: LA - The [Redpanda Cloud API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-serverless-controlplane-api/) now includes support for [Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/). - The Redpanda Schema Registry API is now exposed for Serverless. - Serverless subscriptions can now see detailed billing activity on the **Billing** page. - Serverless added a 99.5% uptime [SLA](https://www.redpanda.com/legal/redpanda-cloud-service-level-agreement) (service level agreement). ### [](#self-service-sign-up-for-dedicated-on-aws-marketplace)Self service sign up for Dedicated on AWS Marketplace To start using Dedicated, sign up on the [AWS Marketplace](https://docs.redpanda.com/cloud-data-platform/billing/aws-pay-as-you-go/). New subscriptions receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your credits have been used, you can continue using your cluster without any commitment, only paying for what you consume. ### [](#support-for-additional-regions-5)Support for additional regions For [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions) and [Dedicated clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/#dedicated-supported-regions), Redpanda added support for the following regions: - GCP: asia-east1 (Taiwan), asia-northeast1 (Tokyo), southamerica-east1 (São Paulo) - AWS: ap-east-1 (Hong Kong), ap-northeast-1 (Tokyo), me-central-1 (UAE) ## [](#june-2024)June 2024 ### [](#remote-read-replica-topics-on-byoc-beta)Remote read replica topics on BYOC: beta You can now create [remote read replica topics](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/remote-read-replicas/) on a BYOC cluster with the Cloud API. A remote read replica topic is a read-only topic that mirrors a topic on a different cluster. It can serve any consumer, without increasing the load on the source cluster. ### [](#higher-connection-limits-in-usage-tiers)Higher connection limits in usage tiers Redpanda has increased the number of client connections in all [tiers](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/). For example, tier 1 now supports up to 9,000 maximum connections, and tier 9 supports up to 450,000 maximum connections. Connections are regulated per broker for best performance. ## [](#may-2024)May 2024 ### [](#cloud-api-beta)Cloud API: beta The Cloud API allows you to programmatically manage clusters and resources in your Redpanda Cloud organization. For more information, see the [Cloud API Quickstart](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-quickstart), the [Cloud API Overview](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview), and the full [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) and [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/) reference documentation. ### [](#mtls-authentication-for-kafka-api-clients)mTLS authentication for Kafka API clients mTLS authentication is now available for Kafka API clients. You can [enable mTLS](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#mtls) for your cluster using the Cloud API. ### [](#manage-private-connectivity-in-the-ui)Manage private connectivity in the UI You can now manage GCP Private Service Connect and AWS PrivateLink connections to your BYOC or Dedicated cluster on the **Dataplane settings** page in Redpanda Cloud. See the steps for [PrivateLink](https://docs.redpanda.com/cloud-data-platform/networking/configure-privatelink-in-cloud-ui/) and [Private Service Connect](https://docs.redpanda.com/cloud-data-platform/networking/configure-private-service-connect-in-cloud-ui/). ### [](#single-message-transforms)Single message transforms Redpanda now provides [single message transforms (SMTs)](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/transforms/) to help you modify data as it passes through a connector, without needing additional stream processors. ### [](#support-for-additional-regions-6)Support for additional regions - For [BYOC clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/#byoc-supported-regions), Redpanda added support for the GPC us-west1 region (Oregon) and the AWS ap-south-1 region (Mumbai). - For [Dedicated clusters](https://docs.redpanda.com/cloud-data-platform/reference/tiers/dedicated-tiers/#dedicated-supported-regions), Redpanda added support for the AWS ap-south-1 region. ### [](#simplified-navigation-and-namespaces-renamed-resource-groups)Simplified navigation and namespaces renamed resource groups Redpanda Cloud has a simplified navigation, with clusters and networks available at the top level. It now has a global view of all resources in your organization. Namespaces are now called [resource groups](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#resource-group), although the functionality remains the same. ## [](#april-2024)April 2024 ### [](#additional-cloud-tiers-for-byoc)Additional cloud tiers for BYOC When you create a BYOC or Dedicated cluster, you select a [cloud tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/) with the expected usage for your cluster, including the maximum ingress, egress, partitions (pre-replication), and connections. Redpanda has added tiers 8 and 9 for BYOC clusters, which provide higher supported configurations. ## [](#march-2024)March 2024 ### [](#serverless-limited-availability)Serverless: limited availability [Redpanda Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) moved out of beta and into limited availability (LA). This means that it has usage limits. During LA, existing clusters can scale to the usage limits, but new clusters may need to wait for availability. Serverless is the fastest and easiest way to start data streaming. It is a production-ready deployment option with automatically-scaling clusters available instantly. To start using Serverless, [sign up for a free trial](https://redpanda.com/try-redpanda/cloud-trial#serverless). This is no base cost, and with pay-as-you-go billing after the trial, you only pay for what you consume. ### [](#authentication-with-sso)Authentication with SSO Redpanda Cloud now supports OpenID Connect (OIDC) integration, so administrators can leverage existing identity providers for user authentication to your Redpanda organization with [single sign-on](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/#single-sign-on) (SSO). Redpanda uses OIDC to delegate the authentication process to an external IdP, such as Okta. To enable this for your account, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). ## [](#february-2024)February 2024 ### [](#aws-privatelink)AWS PrivateLink [AWS PrivateLink](https://docs.redpanda.com/cloud-data-platform/networking/aws-privatelink/) is now available as an easy and highly secure way to connect to Redpanda Cloud from your VPC. You can set up the PrivateLink endpoint service for a new cluster or an existing cluster. To enable AWS PrivateLink for your account, contact [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). ### [](#additional-cloud-tiers)Additional cloud tiers When you create a cluster, you select a [cloud tier](https://docs.redpanda.com/cloud-data-platform/reference/tiers/byoc-tiers/) with the expected throughput for your cluster, including the maximum ingress, egress, partitions, and connections. On February 5, Redpanda added tiers 6 and 7 for BYOC clusters, which provide higher throughput limits. ## [](#january-2024)January 2024 ### [](#usage-based-billing-in-marketplace)Usage-based billing in marketplace Redpanda Cloud now supports [usage-based billing](https://docs.redpanda.com/cloud-data-platform/billing/billing/) for Dedicated clusters. Contact [Redpanda sales](https://redpanda.com/try-redpanda?section=enterprise-trial) to request a private offer for monthly or annual committed use. You can then use existing Google Cloud Marketplace or AWS Marketplace credits to quickly provision Dedicated Cloud clusters, and you can view your bills and manage your subscription directly in the marketplace. ## [](#december-2023)December 2023 ### [](#serverless-clusters-beta)Serverless clusters: beta [Redpanda Serverless](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/serverless/) is a managed streaming service (Kafka API) that completely abstracts users from scaling and operational concerns, and you only pay for what you consume. It’s the fastest and easiest way to start event streaming in the cloud. You can try the beta release of Redpanda Serverless with a free trial. ## [](#november-2023)November 2023 ### [](#aws-byoc-support-for-arm-based-graviton2)AWS BYOC support for ARM-based Graviton2 BYOC clusters on AWS now support ARM-based Graviton2 instances. This lowers VM costs and supports increased partition count. ### [](#iceberg-sink-connector)Iceberg Sink connector With the [managed connector for Apache Iceberg](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/create-iceberg-sink-connector/), you can write data into Iceberg tables. This enables integration with the data lake ecosystem and efficient data management for complex analytics. ### [](#schema-registry-management)Schema Registry management In the Redpanda Console UI, you can [perform Schema Registry operations](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-ui/), such as registering a schema, creating a new version of it, and configuring compatibility. The **Schema Registry** page lists verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to. ### [](#maintenance-windows)Maintenance windows With maintenance windows, you have greater flexibility to plan upgrades to your cluster. By default, Redpanda Cloud upgrades take place on Tuesdays. Optionally, on the **Dataplane settings** page, you can select a window of specific off-hours for your business for Redpanda to apply updates. All times are in Coordinated Universal Time (UTC). Updates may start at any time during that window. --- # Page 581: Manage **URL**: https://docs.redpanda.com/cloud-data-platform/manage.md --- # Manage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/index.adoc description: Manage Redpanda. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-05-07" --- - [Manage Your Organization](manage-organization/) View details about your Redpanda Cloud organization and delete your Serverless organization from the Manage organization page. - [Redpanda CLI](rpk/) The `rpk` tool is a single binary application that provides a way to interact with your Redpanda clusters from the command line. - [Cluster Maintenance](cluster-maintenance/) Learn about cluster maintenance and configuration properties. - [Mountable Topics](mountable-topics/) Safely attach and detach Tiered Storage topics to and from a cluster. - [Integrate Redpanda with Iceberg](iceberg/) Generate Iceberg tables for your Redpanda topics for data lakehouse access. - [Schema Registry](schema-reg/) Redpanda's Schema Registry provides the interface to store and manage event schemas. - [Disaster Recovery](disaster-recovery/) Learn about disaster recovery options for Redpanda Cloud. - [Redpanda Cloud API](api/) Use REST APIs to manage Redpanda Cloud resources. - [Redpanda Terraform Provider](terraform-provider/) Use the Redpanda Terraform provider to create and manage Redpanda Cloud resources. - [Monitor Redpanda Cloud](monitor-cloud/) Learn how to configure monitoring on your BYOC or Dedicated cluster to maintain system health and optimize performance. --- # Page 582: Redpanda Cloud API **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api.md --- # Redpanda Cloud API > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Redpanda Cloud API latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/index.adoc description: Use REST APIs to manage Redpanda Cloud resources. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-03-20" --- - [Use the Control Plane API](controlplane/) Use the Control Plane API to manage resources in your Redpanda Cloud organization. - [Use the Data Plane APIs](cloud-dataplane-api/) Use the Data Plane APIs to manage your Redpanda Cloud clusters. --- # Page 583: Use the Control Plane API with BYOC **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api.md --- # Use the Control Plane API with BYOC > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use the Control Plane API with BYOC latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/cloud-byoc-controlplane-api page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/cloud-byoc-controlplane-api.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/cloud-byoc-controlplane-api.adoc description: Use the Control Plane API to manage resources in your Redpanda Cloud BYOC environment. page-git-created-date: "2024-08-01" page-git-modified-date: "2025-03-20" --- The Redpanda Cloud API is a collection of REST APIs that allow you to interact with different parts of Redpanda Cloud. The Control Plane API enables you to programmatically manage your organization’s Redpanda infrastructure outside of the Cloud UI. You can call the API endpoints directly, or use tools like Terraform or Python scripts to automate cluster management. See [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) for the full API reference documentation. ## [](#control-plane-api)Control Plane API The Control Plane API is one central API that allows you to provision clusters, networks, and resource groups. The Control Plane API consists of the following endpoint groups: - [Clusters](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-clusters) - [Networks](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-networks) - [Operations](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-operations) - [Resource Groups](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-resource-groups) - [Control Plane Role Bindings](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-role-bindings) - [Control Plane Users](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-users) - [Control Plane Service Accounts](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-service-accounts) ## [](#lro)Long-running operations Some endpoints do not directly return the resource itself, but instead return an operation. The following is an example response of [`POST /clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster): ```bash { "operation": { "id": "cqfc6vdmvio001r4vu4", "metadata": { "@type": "type.googleapis.com/redpanda.api.controlplane.v1.CreateClusterMetadata", "cluster_id": "cqg168balf4e4pm8ptu" }, "state": "STATE_IN_PROGRESS", "started_at": "2024-07-23T20:31:29.948Z", "type": "TYPE_CREATE_CLUSTER", "resource_id": "cqg168balf4e4pm8ptu" } } ``` The response object represents the long-running operation of creating a cluster. Cluster creation is an example of an operation that can take a longer period of time to complete. ### [](#check-operation-state)Check operation state To check the progress of an operation, make a request to the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint using the operation ID as a parameter: ```bash curl -H "Authorization: Bearer " https://api.redpanda.com/v1/operations/ ``` > 💡 **TIP** > > When using a shell substitution variable for the token, use double quotes to wrap the header value. The response contains the current state of the operation: `IN_PROGRESS`, `COMPLETED`, or `FAILED`. ## [](#cluster-tiers)Cluster tiers When you create a BYOC or Dedicated cluster, you select a usage tier. Each tier provides tested and guaranteed workload configurations for throughput, partitions (pre-replication), and connections. Availability depends on the region and the cluster type. See the full list of regions, zones, and tiers available with each provider in the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers). ## [](#create-a-cluster)Create a cluster To create a new cluster, first create a resource group and network, if you have not already done so. ### [](#create-a-resource-group)Create a resource group Create a resource group by making a POST request to the [`/v1/resource-groups`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_createresourcegroup) endpoint. Pass a name for your resource group in the request body. ```bash curl -H 'Content-Type: application/json' \ -H "Authorization: Bearer " \ -d '{ "resource_group": { "name": "" } }' -X POST https://api.redpanda.com/v1/resource-groups ``` A resource group ID is returned. Pass this ID later when you call the Create Cluster endpoint. ### [](#create-a-network)Create a network Create a network by making a request to [`POST /v1/networks`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-networkservice_createnetwork). Choose a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/) that does not overlap with your existing VPCs or your Redpanda network. ```bash curl -d \ '{ "network": { "cidr_block": "10.0.0.0/20", "cloud_provider": "CLOUD_PROVIDER_GCP", "cluster_type": "TYPE_BYOC", "name": "", "resource_group_id": "", "region": "us-west1" } }' -H "Content-Type: application/json" \ -H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/networks ``` This endpoint returns a [long-running operation](#lro). ### [](#create-a-new-cluster)Create a new cluster After the network is created, make a request to the [`POST /v1/clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster) with the resource group ID and network ID in the request body. ```bash curl -d \ '{ "cluster": { "cloud_provider": "CLOUD_PROVIDER_GCP", "connection_type": "CONNECTION_TYPE_PUBLIC", "name": "my-new-cluster", "resource_group_id": "", "network_id": "", "region": "us-west1", "throughput_tier": "", "type": "TYPE_BYOC", "zones": [ "us-west1-a", "us-west1-b", "us-west1-c" ], "cluster_configuration": { "custom_properties": { "audit_enabled":true } } } }' -H "Content-Type: application/json" \ -H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/clusters ``` Replace `` with a usage tier that is valid for your region and cluster type. For example, `tier-1-gcp-v2-x86`. See the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers) for the full list of regions, zones, and tiers. The Create Cluster endpoint returns a [long-running operation](#lro). When the operation completes, you can retrieve cluster details by calling [`GET /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster), and passing the cluster ID as a parameter. #### [](#additional-steps-to-create-a-byoc-cluster)Additional steps to create a BYOC cluster 1. Ensure that you have installed `rpk`. 2. After making a Create Cluster request, run `rpk cloud byoc`. Pass `metadata.cluster_id` from the Create Cluster response: ##### AWS ```bash rpk cloud byoc aws apply --redpanda-id= ``` ##### Azure ```bash rpk cloud byoc azure apply --redpanda-id= --subscription-id= ``` ##### GCP ```bash rpk cloud byoc gcp apply --redpanda-id= --project-id= ``` ## [](#update-cluster-configuration)Update cluster configuration To update your cluster configuration properties, make a request to the [`PATCH /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body. ```bash curl -H "Authorization: Bearer " \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{ "cluster_configuration": { "custom_properties": { "iceberg_enabled":true, "iceberg_catalog_type":"rest" } } }' -X PATCH "https://api.cloud.redpanda.com/v1/clusters/" ``` The Update Cluster endpoint returns a [long-running operation](#lro). [Check the operation state](#check-operation-state) to verify that the update is complete. ## [](#delete-a-cluster)Delete a cluster To delete a cluster, make a request to the [`DELETE /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_deletecluster) endpoint, passing the cluster ID as a parameter. This is a [long-running operation](#lro). ```bash curl -H "Authorization: Bearer " -X DELETE https://api.redpanda.com/v1/clusters/ ``` ### [](#additional-steps-to-delete-a-byoc-cluster)Additional steps to delete a BYOC cluster 1. Make a request to [`GET /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster) to check the state of the cluster. Wait until the state is `STATE_DELETING_AGENT`. 2. After the state changes to `STATE_DELETING_AGENT`, run `rpk cloud byoc` to destroy the agent. #### AWS ```bash rpk cloud byoc aws destroy --redpanda-id= ``` #### Azure ```bash rpk cloud byoc azure destroy --redpanda-id= ``` #### GCP ```bash rpk cloud byoc gcp destroy --redpanda-id= --project-id= ``` 3. When the cluster is deleted, the delete operation’s state changes to `STATE_COMPLETED`. At this point, you may make a DELETE request to the [`/v1/networks/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-networkservice_deletenetwork) endpoint to delete the network. This is a long running operation. 4. Optional: After the network is deleted, make a request to [`DELETE /v1/resource-groups/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_deleteresourcegroup) to delete the resource group. ## [](#manage-rbac)Manage RBAC You can also use the Control Plane API to manage [RBAC configurations](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/). ### [](#list-role-bindings)List role bindings To see role assignments for IAM user and service accounts, make a GET request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_listrolebindings) endpoint. ```bash curl https://api.redpanda.com/v1/role-bindings?filter.role_name=&filter.scope.resource_type=SCOPE_RESOURCE_TYPE_CLUSTER \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-role-binding)Get role binding To see roles assignments for a specific IAM account, make a GET request to the [`/v1/role-bindings/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_getrolebinding) endpoint, passing the role binding ID as a parameter. ```bash curl "https://api.redpanda.com/v1/role-bindings/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-user)Get user To see details of an IAM user account, make a GET request to the [`/v1/users/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-userservice_getuser) endpoint, passing the user account ID as a parameter. ```bash curl "https://api.redpanda.com/v1/users/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#create-role-binding)Create role binding To assign a role to an IAM user or service account, make a POST request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_createrolebinding) endpoint. Specify the role and scope, which includes the specific resource ID and an optional resource type, in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/role-bindings" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "role_name": "", "account_id": "", "scope": { "resource_type": "SCOPE_RESOURCE_TYPE_CLUSTER", "resource_id": "" } }' ``` For ``, use one of roles listed in [Predefined roles](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/#predefined-roles) (`Reader`, `Writer`, `Admin`). ### [](#create-service-account)Create service account > 📝 **NOTE** > > Service accounts are assigned the Admin role for all resources in the organization. To create a new service account, make a POST request to the [`/v1/service-accounts`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serviceaccountservice_createserviceaccount) endpoint, with a service account name and optional description in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/service-accounts" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "service_account": { "name": "", "description": "" } }' ``` ## [](#next-steps)Next steps - [Use the Data Plane APIs](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/) --- # Page 584: Use the Data Plane APIs **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api.md --- # Use the Data Plane APIs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use the Data Plane APIs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/cloud-dataplane-api page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/cloud-dataplane-api.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/cloud-dataplane-api.adoc description: Use the Data Plane APIs to manage your Redpanda Cloud clusters. page-git-created-date: "2024-06-06" page-git-modified-date: "2025-08-20" --- The Redpanda Cloud API is a collection of REST APIs that allow you to interact with different parts of Redpanda Cloud. The Data Plane APIs enable you to programmatically manage the resources within your clusters, including topics, users, access control lists (ACLs), and connectors. You can call the API endpoints directly, or use tools like Terraform or Python scripts to automate resource management. See [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/) for the full Data Plane API reference documentation. The [data plane](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-cloud-api-overview#topic-cloud-api-architecture) contains the actual Redpanda clusters. Every cluster is its own data plane, and so it has its own distinct [Data Plane API URL](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-cloud-api-overview#topic-data-plane-apis-url). ## [](#get-data-plane-api-url)Get Data Plane API URL ### BYOC or Dedicated To retrieve the Data Plane API URL of a cluster, make a request to the [`GET /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster) endpoint of the Control Plane API. ### Serverless To retrieve the Data Plane API URL of a cluster, make a request to the [`GET /v1/serverless/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_getserverlesscluster) endpoint of the Control Plane API. The response includes a `dataplane_api.url` value: ```bash "id": "....", "name": "my-cluster", .... "dataplane_api": { "url": "https://api-xyz.abc.fmc.ppd.cloud.redpanda.com" }, ... ``` ## [](#data-plane-apis)Data Plane APIs ### [](#create-a-user)Create a user To create a new user in your Redpanda cluster, make a POST request to the [`/v1/users`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-userservice_createuser) endpoint, including the SASL mechanism, username, and password in the request body: ```bash curl -X POST "https:///v1/users" \ -H "Authorization: Bearer " \ -H "accept: application/json" \ -H "content-type: application/json" \ -d '{"mechanism":"SASL_MECHANISM_SCRAM_SHA_256","name":"payment-service","password":"secure-password"}' ``` > 💡 **TIP** > > When using a shell substitution variable for the token, use double quotes to wrap the header value. The success response returns the newly-created username and SASL mechanism: { "user": { "name": "payment-service", "mechanism": "SASL\_MECHANISM\_SCRAM\_SHA\_256" } } ### [](#create-an-acl)Create an ACL To create a new ACL in your Redpanda cluster, make a [`POST /v1/acls`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-aclservice_createacl) request. The following example ACL allows all operations on any Redpanda topic for a user with the name `payment-service`. ```bash curl -X POST "https:///v1/acls" \ -H "Authorization: Bearer " \ -H "accept: application/json" \ -H "content-type: application/json" \ -d '{"host":"*","operation":"OPERATION_ALL","permission_type":"PERMISSION_TYPE_ALLOW","principal":"User:payment-service","resource_name":"*","resource_pattern_type":"RESOURCE_PATTERN_TYPE_LITERAL","resource_type":"RESOURCE_TYPE_TOPIC"}' ``` The success response is empty, with a 201 status code. {} ### [](#create-a-topic)Create a topic To create a new Redpanda topic without specifying any further parameters, such as the desired topic-level configuration or partition count, make a POST request to [`/v1/topics`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-topicservice_createtopic) endpoint: ```bash curl -X POST "/v1/topics" \ -H "Authorization: Bearer " \ -H "accept: application/json" \ -H "content-type: application/json" \ -d '{"name":""}' ``` ### [](#manage-secrets)Manage secrets Secrets are stored externally in your cloud provider’s secret management service. Redpanda fetches the secrets when you reference them in cluster properties. #### [](#create-a-secret)Create a secret Make a request to [`POST /v1/secrets`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_createsecret). You must use a Base64-encoded secret. ```bash curl -X POST "https:///v1/secrets" \ -H "accept: application/json" \ -H "authorization: Bearer " \ -H "content-type: application/json" \ -d '{"id":"","scopes":["SCOPE_REDPANDA_CLUSTER"],"secret_data":""}' ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The API key you generated during authentication. - ``: The name of the secret you want to add. Use only the following characters: `^[A-Z][A-Z0-9_]*$`. - ``: The Base64-encoded secret. - This scope: `"SCOPE_REDPANDA_CLUSTER"`. The response returns the name and scope of the secret. You can then use the Control Plane API or `rpk` to [set a cluster property value](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/) to reference a secret, using the secret name. For the Control Plane API, you must use the following notation with the secret name in the request body to correctly reference the secret: ```bash "iceberg_rest_catalog_client_secret": "${secrets.}" ``` #### [](#update-a-secret)Update a secret Make a request to [`PUT /v1/secrets/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_updatesecret). You can only update the secret value, not its name. You must use a Base64-encoded secret. ```bash curl -X PUT "https:///v1/secrets/" \ -H "accept: application/json" \ -H "authorization: Bearer " \ -H "content-type: application/json" \ -d '{"scopes":["SCOPE_REDPANDA_CLUSTER"],"secret_data":""}' ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The name of the secret you want to update. The secret’s name is also its ID. - ``: The API key you generated during authentication. - This scope: `"SCOPE_REDPANDA_CLUSTER"`. - ``: Your new Base64-encoded secret. The response returns the name and scope of the secret. It might take several minutes for the new secret value to propagate to any cluster properties that reference it. #### [](#delete-a-secret)Delete a secret Before you delete a secret, make sure that you remove references to it from your cluster configuration. Make a request to [`DELETE /v1/secrets/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-secretservice_deletesecret). ```bash curl -X DELETE "https:///v1/secrets/" \ -H "accept: application/json" \ -H "authorization: Bearer " \ ``` You must include the following values: - ``: The base URL for the Data Plane API. - ``: The name of the secret you want to delete. - ``: The API key you generated during authentication. ### [](#use-redpanda-connect)Use Redpanda Connect Use the API to manage [Redpanda Connect pipelines](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) in Redpanda Cloud. > 📝 **NOTE** > > The Pipeline APIs for Redpanda Connect are supported in BYOC and Serverless clusters only. #### [](#get-redpanda-connect-pipeline)Get Redpanda Connect pipeline To get details of a specific pipeline, make a [`GET /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_getpipeline) request. ```bash curl "https:///v1/redpanda-connect/pipelines/" ``` #### [](#stop-a-redpanda-connect-pipeline)Stop a Redpanda Connect pipeline To stop a running pipeline, make a [`PUT /v1/redpanda-connect/pipelines/{id}/stop`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_stoppipeline) request. ```bash curl -X PUT "https:///v1/redpanda-connect/pipelines//stop" ``` #### [](#start-a-redpanda-connect-pipeline)Start a Redpanda Connect pipeline To start a previously stopped pipeline, make a [`PUT /v1/redpanda-connect/pipelines/{id}/start`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_startpipeline) request. ```bash curl -X PUT "https:///v1/redpanda-connect/pipelines//start" ``` #### [](#update-a-redpanda-connect-pipeline)Update a Redpanda Connect pipeline To update a pipeline, make a [`PUT /v1/redpanda-connect/pipelines/{id}`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-redpandaconnectservice_updatepipeline) request. You update a pipeline configuration to scale resources, for example the number of CPU cores and amount of memory allocated. ```bash curl -X PUT "https://api.redpanda.com/v1/redpanda-connect/pipelines/" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"resources":{"cpu_shares":"8","memory_shares":"8G"}}' ``` ### [](#manage-kafka-connect)Manage Kafka Connect Use the API to configure your [Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/) clusters. > ❗ **IMPORTANT** > > - To enable this feature, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). To disable this feature, see [Disable Kafka Connect](https://docs.redpanda.com/cloud-data-platform/develop/managed-connectors/disable-kc/). > > - Redpanda Support does not manage or monitor Kafka Connect. For fully-supported connectors, consider [Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/). > > - When Kafka Connect is enabled, there is a dedicated node running even when no connectors are deployed. > 📝 **NOTE** > > Kafka Connect is supported in BYOC and Dedicated clusters only. #### [](#create-a-kafka-connect-cluster-secret)Create a Kafka Connect cluster secret Kafka Connect cluster secret data must first be in JSON format, and then Base64-encoded. 1. Prepare the secret data in JSON format: ```none {"secret.access.key": ""} ``` 2. Encode the secret data in Base64: ```none echo '{"secret.access.key": ""}' | base64 ``` 3. Use the [Secrets API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-kafkaconnectservice_createsecret) to create a secret that stores the Base64-encoded secret data: ```bash curl -X POST "https:///v1/kafka-connect/clusters/redpanda/secrets" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"name":"","secret_data":""}' ``` The response returns an `id` that you can use to [create the Kafka Connect connector](#create-a-kafka-connect-connector). #### [](#create-a-kafka-connect-connector)Create a Kafka Connect connector To create a connector, make a POST request to [`/v1/kafka-connect/clusters/{cluster_name}/connectors`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-kafkaconnectservice_createconnector). The following example shows how to create an S3 sink connector with the name `my-connector`: ```bash curl -X POST "/v1/kafka-connect/clusters/redpanda/connectors" \ -H "Authorization: Bearer " \ -H "accept: application/json" \ -H "content-type: application/json" \ -d '{"config":{"connector.class":"com.redpanda.kafka.connect.s3.S3SinkConnector","topics":"test-topic","aws.secret.access.key":"${secretsManager::secret.access.key}","aws.s3.bucket.name":"bucket-name","aws.access.key.id":"access-key","aws.s3.bucket.check":"false","region":"us-east-1"},"name":"my-connector"}' ``` > ⚠️ **CAUTION** > > The field `aws.secret.access.key` in this example contains sensitive information that usually shouldn’t be added to a configuration directly. Redpanda recommends that you first create a secret and then use the secret ID to inject the secret in your Create Connector request. > > If you had created a secret following the example from the previous section [Create a Kafka Connect cluster secret](#create-a-kafka-connect-cluster-secret), use the `id` returned in the Create Secret response to replace the placeholder `` in this Create Connector example. The syntax `${secretsManager::secret.access.key}` tells the Kafka Connect cluster to load ``, specifying the key `secret.access.key` from the secret JSON. Example success response: { "name": "my-connector", "config": { "aws.access.key.id": "access-key", "aws.s3.bucket.check": "false", "aws.s3.bucket.name": "bucket-name", "aws.secret.access.key": "secret-key", "connector.class": "com.redpanda.kafka.connect.s3.S3SinkConnector", "name": "my-connector", "region": "us-east-1", "topics": "test-topic" }, "tasks": \[\], "type": "sink" } #### [](#restart-a-kafka-connect-connector)Restart a Kafka Connect connector To restart a connector, make a POST request to the [`/v1/kafka-connect/clusters/{cluster_name}/connectors/{name}/restart`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-kafkaconnectservice_restartconnector) endpoint: ```bash curl -X POST "/v1/kafka-connect/clusters/redpanda/connectors/my-connector/restart" \ -H "Authorization: Bearer " \ -H "accept: application/json"\ -H "content-type: application/json" \ -d '{"include_tasks":false,"only_failed":false}' ``` ## [](#limitations)Limitations - Client SDKs are not available. --- # Page 585: Use the Control Plane API with Dedicated Cloud **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dedicated-controlplane-api.md --- # Use the Control Plane API with Dedicated Cloud > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use the Control Plane API with Dedicated Cloud latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/cloud-dedicated-controlplane-api page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/cloud-dedicated-controlplane-api.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/cloud-dedicated-controlplane-api.adoc description: Use the Control Plane API to manage resources in your Redpanda Cloud Dedicated environment. page-git-created-date: "2024-08-01" page-git-modified-date: "2025-03-20" --- The Redpanda Cloud API is a collection of REST APIs that allow you to interact with different parts of Redpanda Cloud. The Control Plane API enables you to programmatically manage your organization’s Redpanda infrastructure outside of the Cloud UI. You can call the API endpoints directly, or use tools like Terraform or Python scripts to automate cluster management. See [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) for the full API reference documentation. ## [](#control-plane-api)Control Plane API The Control Plane API is one central API that allows you to provision clusters, networks, and resource groups. The Control Plane API consists of the following endpoint groups: - [Clusters](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-clusters) - [Networks](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-networks) - [Operations](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-operations) - [Resource Groups](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-resource-groups) - [Control Plane Role Bindings](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-role-bindings) - [Control Plane Users](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-users) - [Control Plane Service Accounts](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-service-accounts) ## [](#lro)Long-running operations Some endpoints do not directly return the resource itself, but instead return an operation. The following is an example response of [`POST /clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster): ```bash { "operation": { "id": "cqfc6vdmvio001r4vu4", "metadata": { "@type": "type.googleapis.com/redpanda.api.controlplane.v1.CreateClusterMetadata", "cluster_id": "cqg168balf4e4pm8ptu" }, "state": "STATE_IN_PROGRESS", "started_at": "2024-07-23T20:31:29.948Z", "type": "TYPE_CREATE_CLUSTER", "resource_id": "cqg168balf4e4pm8ptu" } } ``` The response object represents the long-running operation of creating a cluster. Cluster creation is an example of an operation that can take a longer period of time to complete. ### [](#check-operation-state)Check operation state To check the progress of an operation, make a request to the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint using the operation ID as a parameter: ```bash curl -H "Authorization: Bearer " https://api.redpanda.com/v1/operations/ ``` > 💡 **TIP** > > When using a shell substitution variable for the token, use double quotes to wrap the header value. The response contains the current state of the operation: `IN_PROGRESS`, `COMPLETED`, or `FAILED`. ## [](#cluster-tiers)Cluster tiers When you create a BYOC or Dedicated cluster, you select a usage tier. Each tier provides tested and guaranteed workload configurations for throughput, partitions (pre-replication), and connections. Availability depends on the region and the cluster type. See the full list of regions, zones, and tiers available with each provider in the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers). ## [](#create-a-cluster)Create a cluster To create a new cluster, first create a resource group and network, if you have not already done so. ### [](#create-a-resource-group)Create a resource group Create a resource group by making a POST request to the [`/v1/resource-groups`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_createresourcegroup) endpoint. Pass a name for your resource group in the request body. ```bash curl -H 'Content-Type: application/json' \ -H "Authorization: Bearer " \ -d '{ "resource_group": { "name": "" } }' -X POST https://api.redpanda.com/v1/resource-groups ``` A resource group ID is returned. Pass this ID later when you call the Create Cluster endpoint. ### [](#create-a-network)Create a network Create a network by making a request to [`POST /v1/networks`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-networkservice_createnetwork). Choose a [CIDR range](https://docs.redpanda.com/cloud-data-platform/networking/cidr-ranges/) that does not overlap with your existing VPCs or your Redpanda network. ```bash curl -d \ '{ "network": { "cidr_block": "10.0.0.0/20", "cloud_provider": "CLOUD_PROVIDER_GCP", "cluster_type": "TYPE_DEDICATED", "name": "", "resource_group_id": "", "region": "us-central1" } }' -H "Content-Type: application/json" \ -H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/networks ``` This endpoint returns a [long-running operation](#lro). ### [](#create-a-new-cluster)Create a new cluster After the network is created, make a request to the [`POST /v1/clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster) with the resource group ID and network ID in the request body. ```bash curl -d \ '{ "cluster": { "cloud_provider": "CLOUD_PROVIDER_GCP", "connection_type": "CONNECTION_TYPE_PUBLIC", "name": "my-new-cluster", "resource_group_id": "", "network_id": "", "region": "us-central1", "throughput_tier": "", "type": "TYPE_DEDICATED", "zones": [ "us-central1-a", "us-central1-b", "us-central1-c" ], "cluster_configuration": { "custom_properties": { "audit_enabled":true } } } }' -H "Content-Type: application/json" \ -H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/clusters ``` Replace `` with a usage tier that is valid for your region and cluster type. For example, `tier-1-gcp-v2-x86`. See the [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-regions-and-usage-tiers) for the full list of regions, zones, and tiers. The Create Cluster endpoint returns a [long-running operation](#lro). When the operation completes, you can retrieve cluster details by calling [`GET /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster), and passing the cluster ID as a parameter. ## [](#update-cluster-configuration)Update cluster configuration To update your cluster configuration properties, make a request to the [`PATCH /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body. ```bash curl -H "Authorization: Bearer " \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{ "cluster_configuration": { "custom_properties": { "audit_enabled":true } } }' -X PATCH "https://api.cloud.redpanda.com/v1/clusters/" ``` The Update Cluster endpoint returns a [long-running operation](#lro). [Check the operation state](#check-operation-state) to verify that the update is complete. ## [](#delete-a-cluster)Delete a cluster To delete a cluster, make a request to the [`DELETE /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_deletecluster) endpoint, passing the cluster ID as a parameter. This is a [long-running operation](#lro). ```bash curl -H "Authorization: Bearer " -X DELETE https://api.redpanda.com/v1/clusters/ ``` ## [](#manage-rbac)Manage RBAC You can also use the Control Plane API to manage [RBAC configurations](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/). ### [](#list-role-bindings)List role bindings To see role assignments for IAM user and service accounts, make a GET request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_listrolebindings) endpoint. ```bash curl https://api.redpanda.com/v1/role-bindings?filter.role_name=&filter.scope.resource_type=SCOPE_RESOURCE_TYPE_CLUSTER \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-role-binding)Get role binding To see roles assignments for a specific IAM account, make a GET request to the [`/v1/role-bindings/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_getrolebinding) endpoint, passing the role binding ID as a parameter. ```bash curl "https://api.redpanda.com/v1/role-bindings/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-user)Get user To see details of an IAM user account, make a GET request to the [`/v1/users/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-userservice_getuser) endpoint, passing the user account ID as a parameter. ```bash curl "https://api.redpanda.com/v1/users/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#create-role-binding)Create role binding To assign a role to an IAM user or service account, make a POST request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_createrolebinding) endpoint. Specify the role and scope, which includes the specific resource ID and an optional resource type, in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/role-bindings" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "role_name": "", "account_id": "", "scope": { "resource_type": "SCOPE_RESOURCE_TYPE_CLUSTER", "resource_id": "" } }' ``` For ``, use one of roles listed in [Predefined roles](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/#predefined-roles) (`Reader`, `Writer`, `Admin`). ### [](#create-service-account)Create service account > 📝 **NOTE** > > Service accounts are assigned the Admin role for all resources in the organization. To create a new service account, make a POST request to the [`/v1/service-accounts`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serviceaccountservice_createserviceaccount) endpoint, with a service account name and optional description in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/service-accounts" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "service_account": { "name": "", "description": "" } }' ``` ## [](#next-steps)Next steps - [Use the Data Plane APIs](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/) --- # Page 586: Use the Control Plane API with Serverless **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-serverless-controlplane-api.md --- # Use the Control Plane API with Serverless > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use the Control Plane API with Serverless latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/cloud-serverless-controlplane-api page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/cloud-serverless-controlplane-api.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/cloud-serverless-controlplane-api.adoc description: Use the Control Plane API to manage resources in your Redpanda Serverless environment. page-git-created-date: "2024-08-01" page-git-modified-date: "2025-03-20" --- The Redpanda Cloud API is a collection of REST APIs that allow you to interact with different parts of Redpanda Cloud. The Control Plane API enables you to programmatically manage your organization’s Redpanda infrastructure outside of the Cloud UI. You can call the API endpoints directly, or use tools like Terraform or Python scripts to automate cluster management. See [Control Plane API](https://docs.redpanda.com/api/doc/cloud-controlplane/) for the full API reference documentation. ## [](#control-plane-api)Control Plane API The Control Plane API is one central API that allows you to provision clusters, networks, and resource groups. The Control Plane API consists of the following endpoint groups: - [Operations](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-operations) - [Resource Groups](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-resource-groups) - [Serverless Clusters](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-serverless-clusters) - [Serverless Regions](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-serverless-regions) - [Control Plane Role Bindings](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-role-bindings) - [Control Plane Users](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-users) - [Control Plane Service Accounts](https://docs.redpanda.com/api/doc/cloud-controlplane/group/endpoint-control-plane-service-accounts) ## [](#create-a-cluster)Create a cluster To create a new serverless cluster, you can use the default resource group, or create a new resource group if you like. You need to choose a region where your cluster is hosted. ### [](#create-a-resource-group)Create a resource group > 📝 **NOTE** > > This step is optional. Serverless includes a default resource group. To retrieve the default resource group ID, make a GET request to the [`/v1/resource-groups`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_listresourcegroups) endpoint: > > ```bash > curl -H "Authorization: Bearer " https://api.redpanda.com/v1/resource-groups > ``` Create a resource group by making a POST request to the [`/v1/resource-groups`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_createresourcegroup) endpoint. Pass a name for your resource group in the request body. ```bash curl -H 'Content-Type: application/json' \ -H "Authorization: Bearer " \ -d '{ "name": "" }' -X POST https://api.redpanda.com/v1/resource-groups ``` A resource group ID is returned. Pass this ID later when you call the Create Serverless Cluster endpoint. ### [](#choose-a-region)Choose a region To see the available regions for Redpanda Serverless, make a GET request to the [`/v1/serverless/regions`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessregionservice_listserverlessregions) endpoint. You can specify a cloud provider in your request. Serverless currently only supports AWS. ```bash curl -H "Authorization: Bearer " 'https://api.redpanda.com/v1/serverless/regions?cloud_provider=CLOUD_PROVIDER_AWS' ``` > 💡 **TIP** > > When using a shell substitution variable for the token, use double quotes to wrap the header value. ```json { "serverless_regions": [ { "name": "eu-central-1", "display_name": "eu-central-1", "default_timezone": { "id": "Europe/Berlin", "version": "" }, "cloud_provider": "CLOUD_PROVIDER_AWS", "available": true }, ... ], "next_page_token": "" } ``` You can also see a list of supported regions in [Serverless regions](https://docs.redpanda.com/cloud-data-platform/reference/tiers/serverless-regions/). ### [](#create-a-new-serverless-cluster)Create a new serverless cluster Create a Serverless cluster by making a request to [`POST /v1/serverless/clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_createserverlesscluster) with the resource group ID and serverless region name in the request body. ```bash curl -H 'Content-Type: application/json' \ -H "Authorization: Bearer " \ -d '{ "serverless_cluster": { "name": "", "resource_group_id": "", "serverless_region": "us-east-1" } }' -X POST https://api.redpanda.com/v1/serverless/clusters ``` The Create Serverless Cluster endpoint returns a [long-running operation](#lro-serverless). When the operation completes, you can retrieve cluster details by calling [`GET /v1/serverless/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_getserverlesscluster), and passing the cluster ID as a parameter. ## [](#update-cluster-configuration)Update cluster configuration To update your cluster configuration properties, make a request to the [`PATCH /v1/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, passing the cluster ID as a parameter. Include the properties to update in the request body. ```bash curl -H "Authorization: Bearer " \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{ "cluster_configuration": { "custom_properties": { "audit_enabled":true } } }' -X PATCH "https://api.cloud.redpanda.com/v1/clusters/" ``` The Update Cluster endpoint returns a [long-running operation](#lro). [Check the operation state](#check-operation-state) to verify that the update is complete. ## [](#delete-a-cluster)Delete a cluster To delete a cluster, make a request to the [`DELETE /v1/serverless/clusters/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_getserverlesscluster) endpoint, passing the cluster ID as a parameter. This is a [long-running operation](#lro-serverless). ```bash curl -H "Authorization: Bearer " -X DELETE https://api.redpanda.com/v1/serverless/clusters/ ``` Optional: When the cluster is deleted, the delete operation’s state changes to `STATE_COMPLETED`. At this point, you may make a DELETE request to the [`/v1/resource-groups/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-resourcegroupservice_deleteresourcegroup) endpoint to delete the resource group. ## [](#lro-serverless)Long-running operations Some endpoints do not directly return the resource itself, but instead return an operation. The following is an example response of [`POST /serverless/clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serverlessclusterservice_createserverlesscluster): ```bash { "operation": { "id": "cqaramrndjr40k3qei50", "metadata": null, "state": "STATE_IN_PROGRESS", "started_at": { "seconds": "1721087323", "nanos": 888601218 }, "finished_at": null, "type": "TYPE_CREATE_SERVERLESS_CLUSTER" } } ``` The response object represents the long-running operation of creating a cluster. Cluster creation is an example of an operation that can take a longer period of time to complete. ### [](#check-operation-state)Check operation state To check the progress of an operation, make a request to the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint using the operation ID as a parameter: ```bash curl -H "Authorization: Bearer " https://api.redpanda.com/v1/operations/ ``` The response contains the current state of the operation: `IN_PROGRESS`, `COMPLETED`, or `FAILED`. ## [](#manage-rbac)Manage RBAC You can also use the Control Plane API to manage [RBAC configurations](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/). ### [](#list-role-bindings)List role bindings To see role assignments for IAM user and service accounts, make a GET request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_listrolebindings) endpoint. ```bash curl https://api.redpanda.com/v1/role-bindings?filter.role_name=&filter.scope.resource_type=SCOPE_RESOURCE_TYPE_CLUSTER \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-role-binding)Get role binding To see roles assignments for a specific IAM account, make a GET request to the [`/v1/role-bindings/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_getrolebinding) endpoint, passing the role binding ID as a parameter. ```bash curl "https://api.redpanda.com/v1/role-bindings/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#get-user)Get user To see details of an IAM user account, make a GET request to the [`/v1/users/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-userservice_getuser) endpoint, passing the user account ID as a parameter. ```bash curl "https://api.redpanda.com/v1/users/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" ``` ### [](#create-role-binding)Create role binding To assign a role to an IAM user or service account, make a POST request to the [`/v1/role-bindings`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-rolebindingservice_createrolebinding) endpoint. Specify the role and scope, which includes the specific resource ID and an optional resource type, in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/role-bindings" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "role_name": "", "account_id": "", "scope": { "resource_type": "SCOPE_RESOURCE_TYPE_CLUSTER", "resource_id": "" } }' ``` For ``, use one of roles listed in [Predefined roles](https://docs.redpanda.com/cloud-data-platform/security/authorization/rbac/rbac/#predefined-roles) (`Reader`, `Writer`, `Admin`). ### [](#create-service-account)Create service account > 📝 **NOTE** > > Service accounts are assigned the Admin role for all resources in the organization. To create a new service account, make a POST request to the [`/v1/service-accounts`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-serviceaccountservice_createserviceaccount) endpoint, with a service account name and optional description in the request body. ```bash curl -X POST "https://api.redpanda.com/v1/service-accounts" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "service_account": { "name": "", "description": "" } }' ``` ## [](#next-steps)Next steps - [Use the Data Plane APIs](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/) --- # Page 587: Use the Control Plane API **URL**: https://docs.redpanda.com/cloud-data-platform/manage/api/controlplane.md --- # Use the Control Plane API > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use the Control Plane API latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: api/controlplane/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: api/controlplane/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/api/controlplane/index.adoc description: Use the Control Plane API to manage resources in your Redpanda Cloud organization. page-git-created-date: "2024-08-01" page-git-modified-date: "2025-03-20" --- - [Use the Control Plane API with BYOC](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/) Use the Control Plane API to manage resources in your Redpanda Cloud BYOC environment. - [Use the Control Plane API with Dedicated Cloud](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dedicated-controlplane-api/) Use the Control Plane API to manage resources in your Redpanda Cloud Dedicated environment. - [Use the Control Plane API with Serverless](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-serverless-controlplane-api/) Use the Control Plane API to manage resources in your Redpanda Serverless environment. --- # Page 588: Audit Logging **URL**: https://docs.redpanda.com/cloud-data-platform/manage/audit-logging.md --- # Audit Logging > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Audit Logging latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: audit-logging page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: audit-logging.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/audit-logging.adoc description: Learn how to use Redpanda's audit logging capabilities. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Audit logging is supported on BYOC and Dedicated clusters running Redpanda version 24.3 and later. To configure audit logging, see [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). Many scenarios for streaming data include the need for fine-grained auditing of user activity related to the system. This is especially true for regulated industries such as finance, healthcare, and the public sector. Complying with [PCI DSS v4](https://www.pcisecuritystandards.org/document_library/?document=pci_dss) standards, for example, requires verbose and detailed activity auditing, alerting, and analysis capabilities. Redpanda’s auditing capabilities support recording both administrative and operational interactions with topics and with users. Redpanda complies with the Open Cybersecurity Schema Framework (OCSF), providing a predictable and extensible solution that works seamlessly with industry standard tools. With audit logging enabled, there should be no noticeable changes in performance other than slightly elevated CPU usage. ## [](#audit-log-flow)Audit log flow The Redpanda audit log mechanism functions similar to the Kafka flow. When a user interacts with another user or with a topic, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it. ![Audit log flow](https://docs.redpanda.com/cloud-data-platform/shared/_images/audit-logging-flow.png) By default, any management and authentication actions performed on the cluster yield messages written to the audit log topic that are retained for seven days. Interactions with all topics by all principals are audited. Actions performed using the Kafka API and Admin API are all audited, as are actions performed directly through `rpk`. Messages recorded to the audit log topic comply with the [open cybersecurity schema framework](https://schema.ocsf.io/). Any number of analytics frameworks, such as Splunk or Sumo Logic, can receive and process these messages. Using an open standard ensures Redpanda’s audit logs coexist with those produced by other IT assets, powering holistic monitoring and analysis of your assets. ## [](#audit-log-configuration-options)Audit log configuration options Redpanda’s audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level. To configure audit logging, see [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). - [`audit_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled): Boolean value to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `true`. - [`audit_enabled_event_types`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled_event_types): List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. Default: `'["management","authenticate","admin"]'`. - [`audit_excluded_principals`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_excluded_principals): List of strings in JSON style identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`. ## [](#enable-audit-logging)Enable audit logging Audit logging is enabled by default. Cluster administrators can configure the audited topics and principals. However, only the Redpanda team can configure the type of audited events. For more information or support, contact your Redpanda account team. ## [](#configure-retention-for-audit-logs)Configure retention for audit logs You can export audit events to your SIEM for long-term retention to support audit and compliance needs. Redpanda Data recommends that you retain audit logs for at least one year in a separate system like your SIEM, so if there is an issue with the Redpanda cluster you have access to the audit logs. If you need to change the default seven-day retention period, update the retention settings using the `retention.ms` property for the `_redpanda.audit_log` topic: ```bash # Set 1-year retention (in milliseconds) on the audit log topic rpk topic alter-config _redpanda.audit_log --set retention.ms=31536000000 ``` > 📝 **NOTE** > > In Redpanda Cloud, both `retention.ms` (time-based) and `retention.bytes` (size-based) retention policies are applied simultaneously. Data becomes eligible for deletion when either limit is reached, depending on whichever occurs first. This means neither setting strictly takes precedence; the earliest limit (by time or size) triggers data cleanup. When updating audit log retention, check to make sure you do not already have a size-based retention policy that might remove logs before the period you specify. ## [](#next-steps)Next steps [See samples of audit log messages](audit-log-samples/) --- # Page 589: Sample Audit Log Messages **URL**: https://docs.redpanda.com/cloud-data-platform/manage/audit-logging/audit-log-samples.md --- # Sample Audit Log Messages > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Sample Audit Log Messages latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: audit-logging/audit-log-samples page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: audit-logging/audit-log-samples.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/audit-logging/audit-log-samples.adoc description: Sample Redpanda audit log messages. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-05-26" --- Redpanda’s audit logs comply with version 1.0.0 of the [Open Cybersecurity Schema Framework (OCSF)](https://github.com/ocsf). This provides a predictable and extensible solution that works seamlessly with industry standard tools. This page aggregates several sample log files covering a range of scenarios. ## [](#standard-ocsf-messages)Standard OCSF messages Redpanda produces the following standard OCSF class messages: - Authentication (3002) for all authentication events - Application Lifecycle (6002) for when the audit system is enabled or disabled or when Redpanda starts or stops (if auditing is enabled when Redpanda starts or stops) - API Activity (6003) for any access to the Kafka API, Admin API, or Schema Registry Refer to the [OCSF Schema Definition](https://schema.ocsf.io/) for the field definitions for each event class. ## [](#authentication-events)Authentication events These messages illustrate various scenarios around successful and unsuccessful authentication events. Authentication successful This scenario shows the message resulting from an admin using rpk with successful authentication. This is an authentication type event. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", // This is the Node ID of the broker that produced this audit event "uid": "2", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700533469078, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-SCRAM", "auth_protocol_id": 99, // This is the IP address of the Kafka broker that received the authorization request "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, // Name of the Redpanda kafka server "svc_name": "kafka rpc protocol" }, // Indicates that credentials were not encrypted using TLS "is_cleartext": true, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, // This is the IP address of the client that generated the authorization request "src_endpoint": { "ip": "127.0.0.1", // This is the client ID of the kafka client "name": "rpk", "port": 42906 }, "status_id": 1, "user": { "name": "user", "type_id": 1 } } ``` Authentication successful (OIDC with group claims) This scenario shows a successful OIDC authentication event that includes the user’s IdP group memberships in the `user.groups` field. Group memberships are extracted from the OIDC token and included in all authentication events for OIDC users. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v26.1.1" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700533469078, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-OAUTHBEARER", "auth_protocol_id": 99, "dst_endpoint": { "ip": "127.0.0.1", "port": 9092, "svc_name": "kafka rpc protocol" }, "is_cleartext": false, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, "src_endpoint": { "ip": "10.0.1.50", "name": "kafka-client", "port": 48210 }, "status_id": 1, // IdP group memberships extracted from the OIDC token "user": { "name": "alice@example.com", "type_id": 1, "groups": [ {"type": "idp_group", "name": "engineering"}, {"type": "idp_group", "name": "analytics"} ] } } ``` Authentication failed This scenario illustrates a common failure where a user entered the wrong credentials. This is an authentication type event. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", "uid": "1", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700534756350, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-SCRAM", "auth_protocol_id": 99, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, "is_cleartext": true, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 45236 }, "status_id": 2, "status_detail": "SASL authentication failed: security: Invalid credentials", "user": { "name": "admin", "type_id": 1 } } ``` ## [](#kafka-api-events)Kafka API events The Redpanda Kafka API offers a wide array of options for interacting with your Redpanda clusters. Following are examples of messages from common interactions with the API. Create ACL entry This example illustrates an ACL update that also requires a superuser authentication. It lists the edited ACL and the updated permissions. This is a management type event. ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "time": 1700533393776, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", // This shows a superuser level authorization "policy": { "desc": "superuser", "name": "aclAuthorization" } } ], "user": { "name": "admin", "type_id": 2 } }, "api": { // The API operation performed "operation": "create_acls", "service": { "name": "kafka rpc protocol" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, // List of resources accessed "resources": [ // The created ACL { "name": "create acl", "type": "acl_binding", "data": { "resource_type": "topic", "resource_name": "*", "pattern_type": "literal", "acl_principal": "{type user name user}", "acl_host": "{{any_host}}", "acl_operation": "all", "acl_permission": "allow" } }, // Below indicates that the user had cluster level authorization { "name": "kafka-cluster", "type": "cluster" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 50276 }, "status_id": 1, "unmapped": { // Provides a more parsable output of how the // authorization decision was made "authorization_metadata": { "acl_authorization": { "host": "", "op": "", "permission_type": "AUTHORIZED", "principal": "" }, "resource": { "name": "", "pattern": "", "type": "" } } } } ``` Authorization matched on a group ACL This example shows an API Activity (6003) where the authorization decision matched an ALLOW ACL on a `Group:` principal. The `actor.user.groups` field includes the matched group with type `idp_group`, and the `authorization_metadata` shows the group ACL that granted access. See [Group-Based Access Control](https://docs.redpanda.com/cloud-data-platform/security/authorization/gbac/). ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v26.1.0" }, "version": "1.0.0" }, "severity_id": 1, "time": 1774544504327, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", "policy": { "desc": "acl: {principal type {group} name {/sales} host {{any_host}} op all perm allow}, resource: type {topic} name {sales-topic} pattern {literal}", "name": "aclAuthorization" } } ], // The matched group appears in the user's groups field "user": { "name": "alice", "type_id": 1, "groups": [ { "type": "idp_group", "name": "/sales" } ] } }, "api": { "operation": "produce", "service": { "name": "kafka rpc protocol" } }, "dst_endpoint": { "ip": "127.0.1.1", "port": 9092, "svc_name": "kafka rpc protocol" }, "resources": [ { "name": "sales-topic", "type": "topic" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rdkafka", "port": 42728 }, "status_id": 1, "unmapped": { "authorization_metadata": { "acl_authorization": { "host": "{{any_host}}", "op": "all", "permission_type": "allow", "principal": "type {group} name {/sales}" }, "resource": { "name": "sales-topic", "pattern": "literal", "type": "topic" } } } } ``` Metadata request (with counts) This shows a message for a scenario where a user requests a set of metadata using rpk. It provides detailed information on the type of request and the information sent to the user. This is a describe type event. ```json { "category_uid": 6, "class_uid": 6003, // If present, indicates that >1 of the same authz check was performed // within the period of the audit log collecting entries // This provides start and end time (the time period these events were // observed) "count": 2, "end_time": 1700533480725, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "start_time": 1700533480724, "time": 1700533480724, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", // Represents a policy for a non-super user "policy": { "desc": "acl: {principal {type user name user} host {{any_host}} op all perm allow}, resource: type {topic} name {*} pattern {literal}", "name": "aclAuthorization" } } ], "user": { "name": "user", "type_id": 1 } }, "api": { "operation": "metadata", "service": { "name": "kafka rpc protocol" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, "resources": [ // The topics accessed { "name": "test", "type": "topic" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 53602 }, "status_id": 1, "unmapped": { "authorization_metadata": { "acl_authorization": { "host": "{{any_host}}", "op": "all", "permission_type": "allow", "principal": "{type user name user}" }, "resource": { "name": "*", "pattern": "literal", "type": "topic" } } } } ``` --- # Page 590: Cluster Maintenance **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance.md --- # Cluster Maintenance > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Cluster Maintenance latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/index.adoc description: Learn about cluster maintenance and configuration properties. page-git-created-date: "2025-04-08" page-git-modified-date: "2025-05-07" --- - [Cluster State](cluster-state/) Learn about the current status of a cluster. - [Upgrades and Maintenance](https://docs.redpanda.com/cloud-data-platform/manage/maintenance/) Learn how Redpanda Cloud manages maintenance operations. - [Configure Cluster Properties](config-cluster/) Learn how to configure cluster properties to enable and manage features. - [Audit Logging](https://docs.redpanda.com/cloud-data-platform/manage/audit-logging/) Learn how to use Redpanda's audit logging capabilities. - [About Client Throughput Quotas](about-throughput-quotas/) Understand how Redpanda's user-based and client ID-based throughput quotas work, including entity hierarchy, precedence rules, and quota tracking behavior. - [Manage Throughput](manage-throughput/) Configure broker-wide and client-specific throughput quotas to prevent resource exhaustion and noisy-neighbor issues. - [Configure Client Connections](configure-client-connections/) Learn about guidelines for configuring client connections in Redpanda clusters for optimal availability. --- # Page 591: About Client Throughput Quotas **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/about-throughput-quotas.md --- # About Client Throughput Quotas > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: About Client Throughput Quotas latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/about-throughput-quotas page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/about-throughput-quotas.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/about-throughput-quotas.adoc description: Understand how Redpanda's user-based and client ID-based throughput quotas work, including entity hierarchy, precedence rules, and quota tracking behavior. learning-objective-1: Describe the difference between user-based and client ID-based quotas learning-objective-2: Determine which quota type to use for your use case learning-objective-3: Explain quota precedence rules and how Redpanda tracks quota usage page-git-created-date: "2026-03-31" page-git-modified-date: "2026-05-26" --- Redpanda uses throughput quotas to limit the rate of produce and consume requests from clients. Understanding how quotas work helps you prevent individual clients from disproportionately consuming resources and causing performance degradation for other clients (also known as the "noisy-neighbor" problem), and ensure fair resource sharing across users and applications. After reading this page, you will be able to: - Describe the difference between user-based and client ID-based quotas - Determine which quota type to use for your use case - Explain quota precedence rules and how Redpanda tracks quota usage To configure and manage throughput quotas, see [Manage Throughput](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/). ## [](#throughput-control-overview)Throughput control overview Redpanda provides two ways to control throughput: - Broker-wide limits: Configured using cluster properties. For details, see [Broker-wide throughput limits](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/#broker-wide-throughput-limits). - Client throughput quotas: Configured using the Kafka API. Client quotas enable per-user and per-client rate limiting with fine-grained control through entity hierarchy and precedence rules. This page focuses on client quotas. ## [](#supported-quota-types)Supported quota types Redpanda supports three Kafka API-based quota types: | Quota type | Description | | --- | --- | | producer_byte_rate | Limit throughput of produce requests (bytes per second) | | consumer_byte_rate | Limit throughput of fetch requests (bytes per second) | | controller_mutation_rate | Limit rate of topic mutation requests (partitions created or deleted per second) | All quota types can be applied to groups of client connections based on user principals, client IDs, or combinations of both. ## [](#quota-entities)Quota entities Redpanda uses two pieces of identifying information from each client connection to determine which quota applies: - Client ID: An ID that clients self-declare. Quotas can target an exact client ID (`client-id`) or a prefix (`client-id-prefix`). Multiple client connections that share a client ID or ID prefix are grouped into a single quota entity. - User [principal](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#principal): An authenticated identity verified through SASL, mTLS, or OIDC. Connections that share the same user are considered one entity. You can configure quotas that target either entity type, or combine both for fine-grained control. ### [](#client-id-based-quotas)Client ID-based quotas Client ID-based quotas apply to clients identified by their `client-id` field, which is set by the client application. The client ID is typically a configurable property when you create a client with Kafka libraries. When using client ID-based quotas, multiple clients using the same client ID share the same quota tracking. Client ID-based quotas rely on clients honestly reporting their identity and correctly setting the `client-id` property. This makes client ID-based quotas unsuitable for guaranteeing isolation between tenants. Use client ID-based quotas when: - Authentication is not enabled. - Grouping by application or service name is sufficient. - You operate a single-tenant environment where all clients are trusted. - You need simple rate limiting without user-level isolation. ### [](#user-based-quotas)User-based quotas > ❗ **IMPORTANT** > > User-based quotas require [authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/) to be enabled on your cluster. User-based quotas apply to authenticated user principals. Each user has a separate quota, providing a way to limit the impact of individual users on the cluster. User-based quotas rely on Redpanda’s authentication system to verify user identity. The user principal is extracted from SASL credentials, mTLS certificates, or OIDC tokens and cannot be forged by clients. Use user-based quotas when: - You operate a multi-tenant environment, such as SaaS platforms or enterprises with departments. - You require isolation between users or tenants, to avoid noisy-neighbor issues. - You need per-user billing or metering. ### [](#combined-user-and-client-quotas)Combined user and client quotas You can combine user and client identities for fine-grained control over specific (user, client) combinations. Use combined quotas when: - You need fine-grained control, for example: user `alice` using a specific application. - Different rate limits apply to different apps used by the same user. For example, `alice`'s `payment-processor` gets 10 MB/s, but `alice`'s `analytics-consumer` gets 50 MB/s. See [Quota precedence and tracking](#quota-precedence-and-tracking) for examples. ## [](#quota-precedence-and-tracking)Quota precedence and tracking When a request arrives, Redpanda resolves which quota to apply by matching the request’s authenticated user principal and client ID against configured quotas. Redpanda applies the most specific match, using the precedence order in the following table (highest priority first). The precedence level that matches also determines how quota usage is tracked. Redpanda tracks quota usage using a tracker key that determines which connections share the same quota bucket. How connections are grouped into buckets depends on the type of entity the quota targets. To get independent quota tracking per user and client ID combination, configure quotas that include both dimensions, such as `/config/users//clients/` or `/config/users//clients/`. | Level | Match type | Config path | Tracker key | Isolation behavior | | --- | --- | --- | --- | --- | | 1 | Exact user + exact client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 2 | Exact user + client prefix | /config/users//client-id-prefix/ | (user, client-id-prefix) | Clients matching the prefix share tracking within that user | | 3 | Exact user + default client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 4 | Exact user only | /config/users/ | user | All clients for that user share a single tracking bucket | | 5 | Default user + exact client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 6 | Default user + client prefix | /config/users//client-id-prefix/ | (user, client-id-prefix) | Clients matching the prefix share tracking within each user | | 7 | Default user + default client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 8 | Default user only | /config/users/ | user | All clients for each user share a single tracking bucket (per user) | | 9 | Exact client only | /config/clients/ | client-id | All users with that client ID share a single tracking bucket | | 10 | Client prefix only | /config/client-id-prefix/ | client-id-prefix | All clients matching the prefix share a single bucket across all users | | 11 | Default client only | /config/clients/ | client-id | Each unique client ID tracked independently | | 12 | No quota configured | N/A | N/A | No tracking / unlimited throughput | > ❗ **IMPORTANT** > > The `` entity matches any user or client that doesn’t have a more specific quota configured. This is different from an empty/unauthenticated user (`user=""`), or undeclared client ID (`client-id=""`), which are treated as specific entities. ### [](#unauthenticated-connections)Unauthenticated connections Unauthenticated connections have an empty user principal (`user=""`) and are not treated as `user=`. Unauthenticated connections: - Fall back to client-only quotas. - Have unlimited throughput only if no client-only quota matches. ### [](#example-precedence-resolution)Example: Precedence resolution Given these configured quotas: ```bash rpk cluster quotas alter --add consumer_byte_rate=5000000 --name user=alice --name client-id=app-1 rpk cluster quotas alter --add consumer_byte_rate=10000000 --name user=alice rpk cluster quotas alter --add consumer_byte_rate=20000000 --name client-id=app-1 ``` | User + Client ID | Precedence match | | --- | --- | | user=alice, client-id=app-1 | Level 1: Exact user + exact client | | user=alice, client-id=app-2 | Level 4: Exact user only | | user=bob, client-id=app-1 | Level 9: Exact client only | | user=bob, client-id=app-2 | Level 12: No quota configured | When no quota matches (level 12), the connection is not throttled. ### [](#example-user-only-quota)Example: User-only quota If you configure a 10 MB/s produce quota for user `alice`: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice ``` Then `alice` connecting with client ID `app-1` and `alice` connecting with client ID `app-2` share the same 10 MB/s produce limit. To give each of `alice`'s clients an independent 10 MB/s limit, configure: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice --default client-id ``` ### [](#example-user-default-quota)Example: User default quota If you configure a default 10 MB/s produce quota for all users: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --default user ``` This quota applies to all users who don’t have a more specific quota configured. Each user is tracked independently: `alice` gets her own 10 MB/s bucket, `bob` gets his own 10 MB/s bucket, and so on. Within each user, all client ID values share that user’s bucket. `alice` connecting with client ID `app-1` and `alice` connecting with client ID `app-2` share the same 10 MB/s produce limit, while `bob`'s connections have a separate 10 MB/s limit. ## [](#throttling-enforcement)Throughput throttling enforcement > 📝 **NOTE** > > As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput. Redpanda enforces throughput limits by applying backpressure to clients. When a connection exceeds its throughput limit, Redpanda throttles the connection to bring the rate back within the allowed level: 1. Redpanda adds a `throttle_time_ms` field to responses, indicating how long the client should wait. 2. If the client doesn’t honor the throttle time, Redpanda inserts delays on the connection’s next read operation. In Redpanda Cloud, the throttling delay is set to 30 seconds. ## [](#default-behavior)Default behavior Quotas are opt-in restrictions and not enforced by default. When no quotas are configured, clients have unlimited throughput. ## [](#next-steps)Next steps - [Configure throughput quotas](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/) - [Enable authentication for user-based quotas](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/) --- # Page 592: Cluster State **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/cluster-state.md --- # Cluster State > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Cluster State latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/cluster-state page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/cluster-state.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/cluster-state.adoc description: Learn about the current status of a cluster. page-git-created-date: "2025-07-23" page-git-modified-date: "2025-07-24" --- The cluster state shows the current status of a cluster. Redpanda Cloud updates the state automatically, allowing you to monitor a cluster’s health and availability. ## Serverless | State | Description | | --- | --- | | Creating | Cluster is in the process of having its control plane state created. | | Placing | Cluster is in the process of being placed on a cell with sufficient resources in the data plane. | | Ready | Cluster is running and accepting external requests. | | Deleting | Cluster is in the process of having its control plane state removed. Resources dedicated to the cluster in the data plane are released. | | Failed | Cluster is unable to enter the Ready state from either the Creating or Placing states.Try re-creating the cluster. | | Suspended | Cluster is running but blocks all external requests.This can happen when credits run out. Enter a credit card to return to the Ready state. | ## BYOC/Dedicated | State | Description | | --- | --- | | Creating agent | Cluster is in the process of having its control plane state created, and the Redpanda Cloud agent is being deployed. | | Creating | Cluster is in the process of having its control plane state created. | | Ready | Cluster is running and accepting external requests. | | Deleting | Cluster is in the process of having its control plane state removed. Resources dedicated to the cluster in the data plane are released. | | Deleting agent | Cluster is in the process of having its control plane state and Redpanda Cloud agent removed. | | Upgrading | Cluster is undergoing a rolling upgrade or a scaling operation. | | Failed | Cluster is unable to enter the Ready state from either the Creating or the Creating agent states.Try re-creating the cluster. | | Suspended | Cluster is running but blocks all external requests.This can happen when credits run out. Enter a credit card to return to the Ready state. | --- # Page 593: Configure Cluster Properties **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster.md --- # Configure Cluster Properties > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Cluster Properties latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/config-cluster page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/config-cluster.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/config-cluster.adoc description: Learn how to configure cluster properties to enable and manage features. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-07-02" --- Cluster configuration properties are set to their default values and are automatically replicated across all brokers. You can use cluster properties to enable and manage features such as [Iceberg topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/), [data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/), and [audit logging](https://docs.redpanda.com/cloud-data-platform/manage/audit-logging/). For a complete list of the cluster properties available in Redpanda Cloud, see [Cluster Configuration Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/) and [Object Storage Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/object-storage-properties/). > 📝 **NOTE** > > Some properties are read-only and cannot be changed. For example, `cluster_id` is a read-only property that is automatically set when the cluster is created. ## [](#prerequisites)Prerequisites - **`rpk` version 25.1.2+**: To check your current version, see [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - **Redpanda version 25.1.2+**: You can find the version on your cluster’s Overview page in the Redpanda Cloud UI. To verify that you’re logged into the Redpanda control plane and have the correct `rpk` profile configured for your target cluster, run `rpk cloud login` and select your cluster. ## [](#limitations)Limitations Cluster properties are supported on BYOC and Dedicated clusters running on AWS and GCP. - They are not available on BYOC and Dedicated clusters running on Azure. - They are not available on Serverless clusters. Only the cluster properties listed in [Cluster Configuration Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/) and [Object Storage Properties](https://docs.redpanda.com/cloud-data-platform/reference/properties/object-storage-properties/) are configurable in Redpanda Cloud. Setting an unsupported property through the Cloud API or the [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/) returns a `REASON_INVALID_INPUT` error indicating that the property is not allowed. To control the maximum message size, see [Configure the maximum message size](#max-message-size). ## [](#set-cluster-configuration-properties)Set cluster configuration properties You can set cluster configuration properties using the `rpk` command-line tool or the Cloud API. ### rpk Use `rpk cluster config` to set cluster properties. For example, to enable audit logging, set [`audit_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled) to `true`: ```bash rpk cluster config set audit_enabled true ``` To set a cluster property with a secret, you must use the following notation: ```bash rpk cluster config set iceberg_rest_catalog_client_secret '${secrets.}' ``` > 📝 **NOTE** > > Some properties require a rolling restart, and it can take several minutes for the update to complete. The `rpk cluster config set` command returns the operation ID. ### Cloud API Use the Cloud API to set cluster properties: - Create a cluster by making a [`POST /v1/clusters`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_createcluster) request. Edit `cluster_configuration` in the request body with a key-value pair for `custom_properties`. - Update a cluster by making a [`PATCH /v1/clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request, passing the cluster ID as a parameter. Include the properties to update in the request body. For example, to set [`audit_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled) to `true`: ```bash # Store your cluster ID in a variable. export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token. export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret="` # Update your cluster configuration to enable audit logging. curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties": {"audit_enabled":true}}}' ``` The [`PATCH /clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request returns the ID of a long-running operation. You can check the status of the operation by polling the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint. To set a cluster property with a secret, you must use the following notation with the secret name: ```bash curl -H "Authorization: Bearer " -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"cluster_configuration": { "custom_properties": { "iceberg_rest_catalog_client_secret": "${secrets.}" } } }' ``` > 📝 **NOTE** > > Some properties require a rolling restart for the update to take effect. This triggers a [long-running operation](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/#lro) that can take several minutes to complete. ## [](#set-default-topic-properties)Set default topic properties You can set cluster-wide defaults that apply to new topics: | Property | Description | | --- | --- | | default_topic_partitions | Default number of partitions for new topics. | | log_retention_ms | Default length of time to retain topic data before it becomes eligible for deletion. This sets the default topic retention period, which applies when a topic doesn’t set its own retention.ms. | | retention_bytes | Default maximum size per partition before the oldest data becomes eligible for deletion. Applies when a topic doesn’t set its own retention.bytes. | Set these properties using `rpk` or the Cloud API. For example, to set the default topic retention period to 24 hours: ### rpk ```bash rpk cluster config set log_retention_ms 86400000 ``` ### Cloud API ```bash # Store your cluster ID in a variable. export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token. export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret="` # Set the default topic retention period to 24 hours. curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties": {"log_retention_ms":86400000}}}' ``` ## [](#max-message-size)Configure the maximum message size To control the maximum message size in Redpanda Cloud, set the `max.message.bytes` property on each topic. The default is 20 MiB for BYOC and Dedicated clusters and 8 MiB for Serverless clusters. You can increase it up to 32 MiB for BYOC and Dedicated clusters and 20 MiB for Serverless clusters. To set this property, see [Topics Overview](https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic/). The cluster property `kafka_batch_max_bytes` cannot be set through the Cloud API or the [Redpanda Terraform provider](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/). Attempting to set it returns `REASON_INVALID_INPUT` with the message `The properties [kafka_batch_max_bytes] are not allowed`. ## [](#view-cluster-property-values)View cluster property values You can see the value of a cluster configuration property using `rpk` or the Cloud API. ### rpk Use `rpk cluster config get` to view the current cluster property value. For example, to view the current value of [`audit_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled), run: ```bash rpk cluster config get audit_enabled ``` ### Cloud API Use the Cloud API to get the current configuration property values for a cluster. Make a [`GET /clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_getcluster) request, passing the cluster ID as a parameter. The response body contains the current `computed_properties` values. For example, to get the current value of [`audit_enabled`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#audit_enabled): ```bash # Store your cluster ID in a variable. export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token. export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret="` # Get your cluster configuration property values. curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X GET \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ ``` ## [](#suggested-reading)Suggested reading - [Introduction to rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/intro-to-rpk/) - [Redpanda Cloud API Overview](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview) - [Redpanda Cloud API Quickstart](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-quickstart) --- # Page 594: Configure Client Connections **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/configure-client-connections.md --- # Configure Client Connections > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Client Connections latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/configure-client-connections page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/configure-client-connections.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/configure-client-connections.adoc description: Learn about guidelines for configuring client connections in Redpanda clusters for optimal availability. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-05-26" --- Optimize the availability of your clusters by configuring and tuning properties. > 💡 **TIP** > > Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections. > > - Use the [`redpanda_rpc_active_connections` metric](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_rpc_active_connections) to view current Kafka client connections. > > - For clusters on v25.3 and later, use [`rpk cluster connections list`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-connections-list/) or the `GET /v1/monitoring/kafka/connections` endpoint in the Data Plane API to identify: > > - Which clients and applications are connected > > - Long-lived connections and long-running requests > > - Connections with no activity > > - Whether any clients are causing excessive load > > > By reviewing connection details, you can make informed decisions about tuning connection limits and troubleshooting issues. > > > See also: [Data Plane API reference](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections), [Monitor Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/#throughput) ## [](#limit-client-connections)Limit client connections To mitigate the risk of a client creating too many connections and using too many system resources, you can configure a Redpanda cluster to impose limits on the number of client connections that can be created. The following Redpanda cluster properties limit the number of connections: - [`kafka_connections_max_per_ip`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#kafka_connections_max_per_ip): Similar to Kafka’s `max.connections.per.ip`, this sets the maximum number of connections accepted per IP address by a broker. - [`kafka_connections_max_overrides`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#kafka_connections_max_overrides): A list of IP addresses for which `kafka_connections_max_per_ip` is overridden and doesn’t apply. > 📝 **NOTE** > > - These connection limit properties are disabled by default. You must manually enable them. > > - The total number of connections is not equal to the number of clients, because a client can open multiple connections. As a conservative estimate, for a cluster with N brokers, plan for N + 2 connections per client. ### [](#configure-connection-count-limit-by-client-ip)Configure connection count limit by client IP Configure the `kafka_connections_max_per_ip` property to limit the number of connections from each client IP address. > ❗ **IMPORTANT** > > Per-IP connection controls require Redpanda to see individual client IPs. If clients connect through private link endpoints, NAT gateways, or other shared-IP egress, the per-IP limit applies to the shared IP, affecting all clients behind it and preventing isolation of a single offending client. Similarly, multiple clients running on the same host will share the same IP address, and the limit applies collectively to all those clients. See also: [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/) #### [](#configure-the-limit)Configure the limit To configure `kafka_connections_max_per_ip` safely without disrupting legitimate clients, follow these steps: 1. Set up your monitoring stack for your cluster. See [Monitor Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/). 2. Monitor current connection patterns using the `redpanda_rpc_active_connections` metric with the `redpanda_server="kafka"` filter: ```none redpanda_rpc_active_connections{redpanda_id="CLOUD_CLUSTER_ID", redpanda_server="kafka"} ``` 3. Analyze the connection data to identify the normal range of connections for each broker during typical traffic cycles. For example, in the following Grafana screenshot, the normal range is around 200-300 connections: ![Range of active connections over time](https://docs.redpanda.com/cloud-data-platform/shared/_images/monitor_connections.png) 4. Set the `kafka_connections_max_per_ip` value based on your analysis. Use the upper bound of normal connections observed, or use a lower value if you know how many connections per client IP are being opened. 5. Continue monitoring the connection metrics after applying the limit to ensure that legitimate clients are not affected and that the problematic client is properly controlled. > 📝 **NOTE** > > If you find a high load of unexpected connections from multiple IP addresses, `kafka_connections_max_per_ip` alone may be insufficient. If offending IPs outnumber legitimate client IPs, you may need to set `kafka_connections_max_per_ip` so low that it affects legitimate clients. If this is the case, use `kafka_connections_max_overrides` to exempt known legitimate client IPs from the connection limit. #### [](#limitations)Limitations - Decreasing the limit does not terminate any currently open Kafka API connections. - This limit does not apply to Kafka HTTP Proxy connections. - Clients behind NAT gateways or private links share the same IP address as seen by Redpanda brokers. - The limit may negatively affect tail latencies across all client connections. - All clients behind the shared IP are collectively subject to the single `kafka_connections_max_per_ip` limit. - Connection rejections occur randomly among clients when the limit is reached. For example, suppose `kafka_connections_max_per_ip` is set to 100, but clients behind a NAT gateway collectively need 150 connections. When the limit is reached, clients can make only some of the connections while others get rejected, leaving the client in a not-working state. - Redpanda may modify this property during internal operations. - Availability incidents caused by misconfiguring this feature are excluded from the Redpanda Cloud SLA. ## [](#configure-client-reconnections)Configure client reconnections You can configure the Kafka client backoff and retry properties to change the default behavior of the clients to suit your failure requirements. Set the following Kafka client properties on your application’s producer or consumer to manage client reconnections: - `reconnect.backoff.ms`: Amount of time to wait before attempting to reconnect to the broker. The default is 50 milliseconds. - `reconnect.backoff.max.ms`: Maximum amount of time in milliseconds to wait when reconnecting to a broker. The backoff increases exponentially for each consecutive connection failure, up to this maximum. The default is 1000 milliseconds (1 second). Additionally, you can use Kafka properties to control message retry behavior. Delivery fails when either the delivery timeout or the number of retries is met. - `delivery.timeout.ms`: Amount of time for message delivery, so messages are not retried forever. The default is 120000 milliseconds (2 minutes). - `retries`: Number of times a producer can retry sending a message before marking it as failed. The default value is 2147483647 for Kafka >= 2.1, or 0 for Kafka <= 2.0. - `retry.backoff.ms`: Amount of time to wait before attempting to retry a failed request to a given topic partition. The default is 100 milliseconds. ## [](#see-also)See also - [Configure Producers](https://docs.redpanda.com/cloud-data-platform/develop/produce-data/configure-producers/) - [Manage Throughput](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput/) --- # Page 595: Manage Throughput **URL**: https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/manage-throughput.md --- # Manage Throughput > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Manage Throughput latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: cluster-maintenance/manage-throughput page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: cluster-maintenance/manage-throughput.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/cluster-maintenance/manage-throughput.adoc description: Configure broker-wide and client-specific throughput quotas to prevent resource exhaustion and noisy-neighbor issues. learning-objective-1: Set user-based throughput quotas learning-objective-2: Set client ID-based quotas learning-objective-3: Monitor quota usage and throttling behavior page-git-created-date: "2025-08-19" page-git-modified-date: "2026-05-26" --- Redpanda throttles throughput on ingress and egress independently, and you can configure limits at the broker and client levels. This prevents clients from causing unbounded network and disk usage on brokers. You can configure limits at two levels: - Broker limits: These apply to all clients connected to the broker and restrict total traffic on the broker. See [Broker-wide throughput limits](#broker-wide-throughput-limits). - Client limits: These apply to authenticated users or clients defined by their client ID. You can manage client quotas with [`rpk cluster quotas`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas/), with the Redpanda Cloud UI, with the [Redpanda Cloud Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-quotaservice_listquotas), or with the Kafka API. When no quotas apply, the client has unlimited throughput. > 📝 **NOTE** > > Throughput throttling is supported for BYOC and Dedicated clusters only. After reading this page, you will be able to: - Set user-based throughput quotas - Set client ID-based quotas - Monitor quota usage and throttling behavior ## [](#view-connected-client-details)View connected client details Before configuring throughput quotas, check the [current produce and consume throughput](https://docs.redpanda.com/cloud-data-platform/manage/monitor-cloud/#throughput) of a client. Use the [`rpk cluster connections list`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-connections-list/) command or the [`GET /v1/monitoring/kafka/connections`](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections) Data Plane API endpoint to view detailed information about active Kafka client connections. For example, to view a cluster’s connected clients in order of highest current produce throughput, run: ### rpk ```bash rpk cluster connections list --order-by="recent_request_statistics.produce_bytes desc" ``` ```bash UID STATE USER CLIENT-ID IP:PORT NODE SHARD OPEN-TIME IDLE PROD-TPUT/SEC FETCH-TPUT/SEC REQS/MIN b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55012 0 0 9s 0s 78.9MB 0B 292 36338ca5-86b7-4478-ad23-32d49cfaef61 OPEN UNAUTHENTICATED rpk 127.0.0.1:49722 0 0 13s 13.694243104s 0B 0B 1 7e277ef6-0176-4007-b100-6581bfde570f OPEN UNAUTHENTICATED rpk 127.0.0.1:49736 0 0 13s 10.093957335s 0B 0B 2 567d9918-d3dc-4c74-ab5d-85f70cd3ee35 OPEN UNAUTHENTICATED rpk 127.0.0.1:49748 0 0 13s 0.591413542s 0B 0B 5 08616f21-08f9-46e7-8f06-964bd8240d9b OPEN UNAUTHENTICATED rpk 127.0.0.1:49764 0 0 13s 10.094602845s 0B 0B 2 e4d5b57e-5c76-4975-ada8-17a88d68a62d OPEN UNAUTHENTICATED rpk 127.0.0.1:54992 0 0 10s 0.302090085s 0B 14.5MB 27 b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55002 0 0 8s 7.743592270s 0B 0B 1 62fde947-411d-4ea8-9461-3becc2631b46 CLOSED UNAUTHENTICATED rpk 127.0.0.1:48578 0 0 26s 0.000737836s 0B 0B 1 95387e2e-2ec4-4040-aa5e-4257a3efa1a2 CLOSED UNAUTHENTICATED rpk 127.0.0.1:48564 0 0 26s 0.208180826s 0B 0B 1 ``` ### Data Plane API ```bash curl \ --request GET 'https:///v1/monitoring/kafka/connections' \ --header "Authorization: Bearer $ACCESS_TOKEN" \ --data '{ "filter": "", "order_by": "recent_request_statistics.produce_bytes desc" }' ``` Show example API response ```json { "connections": [ { "node_id": 0, "shard_id": 0, "uid": "b20601a3-624c-4a8c-ab88-717643f01d56", "state": "KAFKA_CONNECTION_STATE_OPEN", "open_time": "2025-10-15T14:15:15.755065000Z", "close_time": "1970-01-01T00:00:00.000000000Z", "authentication_info": { "state": "AUTHENTICATION_STATE_UNAUTHENTICATED", "mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED", "user_principal": "" }, "listener_name": "", "tls_info": { "enabled": false }, "source": { "ip_address": "127.0.0.1", "port": 55012 }, "client_id": "perf-producer-client", "client_software_name": "apache-kafka-java", "client_software_version": "3.9.0", "transactional_id": "my-tx-id", "group_id": "", "group_instance_id": "", "group_member_id": "", "api_versions": { "18": 4, "22": 3, "3": 12, "24": 3, "0": 7 }, "idle_duration": "0s", "in_flight_requests": { "sampled_in_flight_requests": [ { "api_key": 0, "in_flight_duration": "0.000406892s" } ], "has_more_requests": false }, "total_request_statistics": { "produce_bytes": "78927173", "fetch_bytes": "0", "request_count": "4853", "produce_batch_count": "4849" }, "recent_request_statistics": { "produce_bytes": "78927173", "fetch_bytes": "0", "request_count": "4853", "produce_batch_count": "4849" } }, ... ], "total_size": "9" } ``` To view connections for a specific client, you can use a filter expression: ### rpk ```bash rpk cluster connections list --client-id="perf-producer-client" ``` ```bash UID STATE USER CLIENT-ID IP:PORT NODE SHARD OPEN-TIME IDLE PROD-TPUT/SEC FETCH-TPUT/SEC REQS/MIN b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55002 0 0 8s 7.743592270s 0B 0B 1 b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55012 0 0 9s 0s 78.9MB 0B 292 ``` The `USER` field in the connection list shows the authenticated principal. Unauthenticated connections show `UNAUTHENTICATED`, which corresponds to an empty user principal (`user=""`) in quota configurations, not `user=`. ### Data Plane API ```bash curl \ --request GET 'https:///v1/monitoring/kafka/connections' \ --header "Authorization: Bearer $ACCESS_TOKEN" \ --data '{ "filter": "client_id = \"perf-producer-client\"" }' ``` Show example API response ```json { "connections": [ { "node_id": 0, "shard_id": 0, "uid": "b41584f3-2662-4185-a4b8-0d8510f5c780", "state": "KAFKA_CONNECTION_STATE_OPEN", "open_time": "2025-10-15T14:15:15.219538000Z", "close_time": "1970-01-01T00:00:00.000000000Z", "authentication_info": { "state": "AUTHENTICATION_STATE_UNAUTHENTICATED", "mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED", "user_principal": "" }, "listener_name": "", "tls_info": { "enabled": false }, "source": { "ip_address": "127.0.0.1", "port": 55002 }, "client_id": "perf-producer-client", "client_software_name": "apache-kafka-java", "client_software_version": "3.9.0", "transactional_id": "", "group_id": "", "group_instance_id": "", "group_member_id": "", "api_versions": { "18": 4, "3": 12, "10": 4 }, "idle_duration": "7.743592270s", "in_flight_requests": { "sampled_in_flight_requests": [], "has_more_requests": false }, "total_request_statistics": { "produce_bytes": "0", "fetch_bytes": "0", "request_count": "3", "produce_batch_count": "0" }, "recent_request_statistics": { "produce_bytes": "0", "fetch_bytes": "0", "request_count": "3", "produce_batch_count": "0" } }, ... ], "total_size": "2" } ``` The user principal field in the connection list shows the authenticated principal. Unauthenticated connections show `AUTHENTICATION_STATE_UNAUTHENTICATED`, which corresponds to an empty user principal (`user=""`) in quota configurations, not `user=`. To view connections for a specific authenticated user: ```bash rpk cluster connections list --user alice ``` This shows all connections from user `alice`, useful for monitoring clients that are subject to user-based quotas. ## [](#broker-wide-throughput-limits)Broker-wide throughput limits Broker-wide throughput limits account for all Kafka API traffic going into or out of the broker, as data is produced to or consumed from a topic. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits. ## [](#client-throughput-limits)Client throughput limits Redpanda provides configurable throughput quotas for individual clients or authenticated users. Quotas are managed through the Kafka-compatible AlterClientQuotas and DescribeClientQuotas APIs, accessible with `rpk`, Redpanda Console, or Kafka client libraries. Redpanda supports two types of client throughput quotas: - Client ID-based quotas: Limit throughput based on the self-declared `client-id` field. - User-based quotas: Limit throughput based on authenticated user [principal](https://docs.redpanda.com/cloud-data-platform/reference/glossary/#principal). Requires [authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/). You can also combine both types for fine-grained control (for example, limiting a specific user when using a specific client application). For conceptual information about quota types, entity hierarchy, precedence rules, and how Redpanda tracks and enforces quotas through throttling, see [About Client Throughput Quotas](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/about-throughput-quotas/). ### [](#set-user-based-quotas)Set user-based quotas > ❗ **IMPORTANT** > > User-based quotas require authentication to be enabled. To set up authentication, see [Authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/). #### [](#quota-for-a-specific-user)Quota for a specific user To limit throughput for a specific authenticated user across all clients: ```bash rpk cluster quotas alter --add producer_byte_rate=2000000 --name user=alice ``` This limits user `alice` to 2 MB/s for produce requests regardless of the client ID used. To view quotas for a user: ```bash rpk cluster quotas describe --name user=alice ``` Expected output: ```bash user=alice producer_byte_rate=2000000 ``` #### [](#default-quota-for-all-users)Default quota for all users To set a fallback quota for any user without a more specific quota: ```bash rpk cluster quotas alter --add consumer_byte_rate=5000000 --default user ``` This applies a 5 MB/s fetch quota to all authenticated users who don’t have a more specific quota configured. ### [](#remove-a-user-quota)Remove a user quota To remove a quota for a specific user: ```bash rpk cluster quotas alter --delete consumer_byte_rate --name user=alice ``` To remove all quotas for a user: ```bash rpk cluster quotas delete --name user=alice ``` ### [](#set-client-id-based-quotas)Set client ID-based quotas Client ID-based quotas apply to all users using a specific client ID. These quotas do not require authentication. Because the client ID is self-declared, client ID-based quotas are not suitable for guaranteeing isolation between tenants. For multi-tenant environments, Redpanda recommends user-based quotas for per-tenant isolation. #### [](#individual-client-id-throughput-limit)Individual client ID throughput limit > 📝 **NOTE** > > The following sections show how to manage throughput with `rpk`. You can also manage throughput with the [Redpanda Cloud Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-quotaservice_listquotas). To view current throughput quotas set through the Kafka API, run [`rpk cluster quotas describe`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas-describe/). For example, to see the quotas for client ID `consumer-1`: ```bash rpk cluster quotas describe --name client-id=consumer-1 ``` ```bash client-id=consumer-1 producer_byte_rate=140000 ``` To set a throughput quota for a single client, use the [`rpk cluster quotas alter`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas-alter/) command. ```bash rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id=consumer-1 ``` ```bash ENTITY STATUS client-id=consumer-1 OK ``` #### [](#group-of-clients-throughput-limit)Group of clients throughput limit Alternatively, you can view or configure throughput quotas for a group of clients based on a match on client ID prefix. The following example sets the `consumer_byte_rate` quota to client IDs prefixed with `consumer-`: ```bash rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id-prefix=consumer- ``` > 📝 **NOTE** > > A `client-id-prefix` quota group is not related to Kafka consumer groups. The client ID is an application-defined identifier sent with every request. Client libraries typically default to their own name (such as `kgo`, `rdkafka`, `sarama`, or `perf-producer-client`), but applications can set it using the [`client.id`](https://kafka.apache.org/documentation/#consumerconfigs_client.id) configuration property. This makes prefix-based quotas useful for grouping related applications (for example, `inventory-service-` to match `inventory-service-1`, `inventory-service-2`, etc.). #### [](#default-client-throughput-limit)Default client throughput limit You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific client ID or prefix. To specify a produce quota of 1 GB/s through the Kafka API (applies across all produce requests to a single broker), run: ```bash rpk cluster quotas alter --default client-id --add producer_byte_rate=1000000000 ``` ### [](#set-combined-user-and-client-quotas)Set combined user and client quotas You can set quotas for specific (user, client ID) combinations for fine-grained control. #### [](#user-with-specific-client)User with specific client To limit a specific user when using a specific client: ```bash rpk cluster quotas alter --add consumer_byte_rate=1000000 --name user=alice --name client-id=consumer-1 ``` User `alice` using `client-id=consumer-1` is limited to a 1 MB/s fetch rate. The same user with a different client ID would use a different quota (or fall back to less specific matches). To view combined quotas: ```bash rpk cluster quotas describe --name user=alice --name client-id=consumer-1 ``` #### [](#user-with-client-prefix)User with client prefix To set a shared quota for a user across multiple clients matching a prefix: ```bash rpk cluster quotas alter --add producer_byte_rate=3000000 --name user=bob --name client-id-prefix=app- ``` All clients used by user `bob` with a client ID starting with `app-` share a combined 3 MB/s produce quota. #### [](#default-user-with-specific-client)Default user with specific client To set a quota for a specific client across all users: ```bash rpk cluster quotas alter --add producer_byte_rate=500000 --default user --name client-id=payment-processor ``` Any user using `client-id=payment-processor` is limited to a 500 KB/s produce rate, unless they have a more specific quota configured. ### [](#bulk-manage-client-throughput-limits)Bulk manage client throughput limits To more easily manage multiple quotas, you can use the `cluster quotas describe` and [`cluster quotas import`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas-import/) commands to do a bulk export and update. For example, to export all client quotas in JSON format: ```bash rpk cluster quotas describe --format json ``` `rpk cluster quotas import` accepts the output string from `rpk cluster quotas describe --format `: ```bash rpk cluster quotas import --from '{"quotas":[{"entity":[{"name":"analytics-consumer","type":"client-id"}],"values":[{"key":"consumer_byte_rate","values":"10000000"}]},{"entity":[{"name":"analytics-","type":"client-id-prefix"}],"values":[{"key":"producer_byte_rate","values":"10000000"},{"key":"consumer_byte_rate","values":"5000000"}]}]}' ``` You can also save the JSON or YAML output to a file and pass the file path in the `--from` flag. ### [](#view-throughput-limits-in-redpanda-cloud)View throughput limits in Redpanda Cloud You can also use Redpanda Cloud to view enforced limits. In the side menu, go to **Quotas**. ### [](#monitor-client-throughput)Monitor client throughput The following metrics provide insights into client throughput quota usage: - Client quota throughput per rule and quota type: - `/public_metrics` - [`redpanda_kafka_quotas_client_quota_throughput`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throughput) - Client quota throttling delay per rule and quota type, in seconds: - `/public_metrics` - [`redpanda_kafka_quotas_client_quota_throttle_time`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throttle_time) To identify which clients are actively connected and generating traffic, see [View connected client details](#view-connected-client-details). Quota metrics use the `redpanda_quota_rule` label to identify which quota was applied to a request. The label distinguishes between different entity types (user, client, or combinations). See the label values in [`redpanda_kafka_quotas_client_quota_throughput`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throughput). #### [](#track-quota-use-per-entity)Track quota use per entity When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening at a broker level, but they cannot reveal which user, client, or group of clients is utilizing a quota or being throttled. Per-entity quota metrics answer these questions. Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each entity actually uses. Use these metrics to identify throttled users and clients by name and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them, set the `kafka_per_entity_quota_metrics` cluster property. See [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). The change takes effect without a broker restart. When enabled, Redpanda exposes two additional counters: - Total per-entity quota throttling delay, in milliseconds: - `/public_metrics` - `redpanda_kafka_quotas_client_quota_throttle_time_ms_by_entity` - Per-entity quota throughput (bytes for produce and fetch quotas, partition mutations for partition mutation quotas): - `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` Each series is labeled with the entity identity and the quota type: - `redpanda_quota_type`: Always present. One of `produce_quota`, `fetch_quota`, or `partition_mutation_quota`. - `redpanda_quota_user`: The user principal, for user-based quotas. - `redpanda_quota_client_id`: The client ID, for client ID-based quotas. - `redpanda_quota_group_name`: The client ID prefix, for quotas that apply to a [group of clients](#group-of-clients-throughput-limit). Combined quotas, such as a user with a specific client ID, include each matching label on the same series. To keep metric cardinality bounded, Redpanda registers a per-entity series only while an entity is actively being throttled, and removes the series after the entity has been idle. An entity appears in these metrics only after it has been throttled at least once. From that point, Redpanda records the entity’s throughput on every request, not only on throttled requests, so a Prometheus `rate()` query reflects the entity’s actual throughput. To measure how much of its quota an entity is using, divide the entity’s throughput rate by its configured limit. Quota limits apply per broker, so aggregate by broker as well as by entity to keep the results comparable with the configured limits. For example, the following query returns the produce throughput rate for each throttled client ID on each broker: ```promql sum by (redpanda_quota_client_id, pod) ( rate(redpanda_kafka_quotas_client_quota_throughput_by_entity{redpanda_quota_type="produce_quota"}[5m]) ) ``` Compare the result with the entity’s configured limit from [`rpk cluster quotas describe`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-cluster/rpk-cluster-quotas-describe/). A ratio close to 1 means the client is saturating its quota and its requests are being delayed. ## [](#see-also)See also - [About Client Throughput Quotas](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/about-throughput-quotas/) - [Configure Client Connections](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/configure-client-connections/) - [Authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/) --- # Page 596: Disaster Recovery **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery.md --- # Disaster Recovery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Disaster Recovery latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/index.adoc description: Learn about disaster recovery options for Redpanda Cloud. page-git-created-date: "2025-12-12" page-git-modified-date: "2025-12-12" --- Shadowing complements Redpanda’s existing availability and recovery capabilities. High availability actively protects your day-to-day operations, handling reads and writes seamlessly during node or availability zone failures within a region. Shadowing is your safety net for catastrophic regional disasters. Shadowing delivers near real-time, cross-region replication for mission-critical applications that require rapid failover with minimal data loss. > 📝 **NOTE** > > Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. - [Shadowing](shadowing/) Learn about shadowing for disaster recovery in Redpanda Cloud. --- # Page 597: Shadowing **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing.md --- # Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Shadowing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/index.adoc description: Learn about shadowing for disaster recovery in Redpanda Cloud. page-git-created-date: "2025-12-12" page-git-modified-date: "2025-12-12" --- > 📝 **NOTE** > > Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. - [Shadowing Overview](overview/) Overview of shadowing for disaster recovery in Redpanda Cloud. - [Configure Shadowing](setup/) Learn how to configure shadowing for disaster recovery. - [Monitor Shadowing](monitor/) Learn how to monitor shadowing for disaster recovery. - [Configure Failover](failover/) Learn how to configure failover for disaster recovery. - [Failover Runbook](failover-runbook/) Step-by-step runbook for failover procedures in disaster recovery. --- # Page 598: Failover Runbook **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook.md --- # Failover Runbook > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Failover Runbook latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/failover-runbook page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/failover-runbook.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/failover-runbook.adoc description: Step-by-step runbook for failover procedures in disaster recovery. page-git-created-date: "2025-12-12" page-git-modified-date: "2026-05-26" --- This guide provides step-by-step procedures for emergency failover when your primary Redpanda cluster becomes unavailable. Follow these procedures only during active disasters when immediate failover is required. > ❗ **IMPORTANT** > > This is an emergency procedure. For planned failover testing or day-to-day shadow link management, see [Configure Failover](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover/). Ensure you have completed the [disaster readiness checklist](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#disaster-readiness-checklist) before an emergency occurs. > 📝 **NOTE** > > Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. ## [](#emergency-failover-procedure)Emergency failover procedure Follow these steps during an active disaster: 1. [Assess the situation](#assess-situation) 2. [Verify shadow cluster status](#verify-shadow-status) 3. [Document current state](#document-state) 4. [Initiate failover](#initiate-failover) 5. [Monitor failover progress](#monitor-progress) 6. [Update application configuration](#update-applications) 7. [Verify application functionality](#verify-functionality) 8. [Clean up and stabilize](#cleanup-stabilize) ### [](#assess-situation)Assess the situation Confirm that failover is necessary: ```bash # Check if the primary cluster is responding rpk cluster info --brokers prod-cluster-1.example.com:9092,prod-cluster-2.example.com:9092 # If primary cluster is down, check shadow cluster health rpk cluster info --brokers shadow-cluster-1.example.com:9092,shadow-cluster-2.example.com:9092 ``` **Decision point**: If the primary cluster is responsive, consider whether failover is actually needed. Partial outages may not require full disaster recovery. **Examples that require full failover:** - Primary cluster is completely unreachable (network partition, regional outage) - Multiple broker failures preventing writes to critical topics - Data center failure affecting majority of brokers - Persistent authentication or authorization failures across the cluster **Examples that may NOT require failover:** - Single broker failure with sufficient replicas remaining - Temporary network connectivity issues affecting some clients - High latency or performance degradation (but cluster still functional) - Non-critical topic or partition unavailability ### [](#verify-shadow-status)Verify shadow cluster status Check the health of your shadow links: #### Cloud UI 1. From the **Shadow Link** page, select the shadow link you want to view. 2. The **Overview** tab shows the state of the shadow link and its topics. #### rpk ```bash # List all shadow links rpk shadow list # Check the configuration of your shadow link rpk shadow describe # Check the status of your disaster recovery link rpk shadow status ``` For detailed command options, see [`rpk shadow list`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-list/), [`rpk shadow describe`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-describe/), and [`rpk shadow status`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-status/). #### Cloud API ```bash # List all shadow links curl "https://api.redpanda.com/v1/shadow-links" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" # Check the configuration of your shadow link curl "https://api.redpanda.com/v1/shadow-links/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" # Get Data Plane API URL of shadow cluster export DATAPLANE_API_URL=`curl https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" | jq .cluster.dataplane_api` # Check the status of your disaster recovery link curl "https://$DATAPLANE_API_URL/v1/shadowlinks/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` Verify that the following conditions exist before proceeding with failover: - Shadow link state should be `ACTIVE`. - Topics should be in `ACTIVE` state (not `FAULTED`). - Replication lag should be reasonable for your RPO requirements. #### [](#understanding-replication-lag)Understanding replication lag Use [`rpk shadow status`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-status/) or the [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-shadowlinkservice_listshadowlinktopics) to check lag, which shows the message count difference between source and shadow partitions: - **Acceptable lag examples**: 0-1000 messages for low-throughput topics, 0-10000 messages for high-throughput topics - **Concerning lag examples**: Growing lag over 50,000 messages, or lag that continuously increases without recovering - **Critical lag examples**: Lag exceeding your data loss tolerance (for example, if you can only afford to lose 1 minute of data, lag should represent less than 1 minute of typical message volume) ### [](#document-state)Document current state Record the current lag and status before proceeding: #### Cloud UI Capture the status from the **Shadow Link** page. #### rpk ```bash # Capture current status for post-mortem analysis rpk shadow status > failover-status-$(date +%Y%m%d-%H%M%S).log ``` Example output showing healthy replication before failover: shadow link: Overview: NAME UID STATE ACTIVE Tasks: Name Broker\_ID State Reason 1 ACTIVE 2 ACTIVE Topics: Name: , State: ACTIVE Partition SRC\_LSO SRC\_HWM DST\_HWM Lag 0 1234 1468 1456 12 1 2345 2579 2568 11 #### Cloud API ```bash # Capture current status for post-mortem analysis curl "https://$DATAPLANE_API_URL/v1/shadowlinks//topic" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" > failover-status-$(date +%Y%m%d-%H%M%S).log ``` The partition information shows the following: | Field | Description | | --- | --- | | source_last_stable_offset | Source partition last stable offset | | source_high_watermark | Source partition high watermark | | high_watermark | Shadow (destination) partition high watermark | | Lag | Message count difference between source and shadow partitions | > ❗ **IMPORTANT** > > Note the replication lag to estimate potential data loss during failover. The `Tasks` section shows the health of shadow link replication tasks. For details about what each task does, see [Shadow link tasks](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). ### [](#initiate-failover)Initiate failover A complete cluster failover is appropriate If you observe that the source cluster is no longer reachable: #### Cloud UI 1. On your **Shadow Link** page, click **Failover All Topics**. 2. Click to confirm the failover action. The failover process promotes all topics to writable status. #### rpk ```bash # Fail over all topics in the shadow link rpk shadow failover --all ``` For detailed command options, see [`rpk shadow failover`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-failover/). #### Cloud API ```bash # Fail over all topics in the shadow link curl -X POST "$DATAPLANE_API_URL/v1/shadowlink//failover" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` For selective topic failover (when only specific services are affected): #### Cloud UI 1. On your **Shadow Link** page, click the **Failover** button for the topics you want to failover. 2. Click to confirm the failover action. The failover process promotes the selected topics to writable status. #### rpk ```bash # Fail over individual topics rpk shadow failover --topic rpk shadow failover --topic ``` #### Cloud API ```bash # Fail over individual topics curl -X POST "$DATAPLANE_API_URL/v1/shadowlinks//failover" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d '{ "shadowTopicName": "" }' curl -X POST "$DATAPLANE_API_URL/v1/shadowlinks//failover" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d '{ "shadowTopicName": "" }' ``` ### [](#monitor-progress)Monitor failover progress Track the failover process: #### Cloud UI 1. From the **Shadow Link** page, select the shadow link you want to view. 2. Click the **Tasks** tab to view all tasks and their status. #### rpk ```bash # Monitor status until all topics show FAILED_OVER watch -n 5 "rpk shadow status " # Check detailed topic status and lag during emergency rpk shadow status --print-topic ``` Example output during successful failover: shadow link: Overview: NAME UID STATE ACTIVE Tasks: Name Broker\_ID State Reason 1 ACTIVE 2 ACTIVE Topics: Name: , State: FAILED\_OVER Name: , State: FAILED\_OVER Name: , State: FAILING\_OVER #### Cloud API ```bash # Monitor status watch -n 5 'curl "https://$DATAPLANE_API_URL/v1/shadowlinks/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" | jq .' # Check detailed topic status and lag during emergency curl "https://$DATAPLANE_API_URL/v1/shadowlinks//topic" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` **Wait for**: All critical topics to reach `FAILED_OVER` state before proceeding. ### [](#update-applications)Update application configuration Redirect your applications to the shadow cluster by updating connection strings in your applications to point to shadow cluster brokers. If using DNS-based service discovery, update DNS records accordingly. Restart applications to pick up new connection settings and verify connectivity from application hosts to shadow cluster. ### [](#verify-functionality)Verify application functionality Test critical application workflows: ```bash # Verify applications can produce messages rpk topic produce --brokers :9092 # Verify applications can consume messages rpk topic consume --brokers :9092 --num 1 ``` Test message production and consumption, consumer group functionality, and critical business workflows to ensure everything is working properly. ### [](#cleanup-stabilize)Clean up and stabilize After all applications are running normally: #### Cloud UI 1. On your **Shadow Link** page, click **Delete**. 2. Type "delete" to confirm the action. #### rpk ```bash # Optional: Delete the shadow link (no longer needed) rpk shadow delete ``` For detailed command options, see [`rpk shadow delete`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-delete/). #### Cloud API ```bash # Optional: Delete the shadow link (no longer needed) curl -X DELETE https://api.redpanda.com/v1/shadow-links/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` For the full API reference, see [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-shadowlinkservice_deleteshadowlink). > 📝 **NOTE** > > This operation [force deletes](#force-delete-warning) the shadow link. Document the time of failover initiation and completion, applications affected and recovery times, data loss estimates based on replication lag, and issues encountered during failover. ## [](#troubleshoot-common-issues)Troubleshoot common issues > 📝 **NOTE** > > Direct access to shadow cluster logs isn’t available in Redpanda Cloud. To troubleshoot, use: > > - Run [`rpk shadow status`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-status/) to check task states, topic status, and lag. > > - Use the [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-shadowlinkservice_listshadowlinktopics) for programmatic monitoring. > > - Monitor [Prometheus metrics](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/monitor/#shadow-link-metrics) such as `redpanda_shadow_link_shadow_lag` and `redpanda_shadow_link_client_errors`. > > > If these don’t reveal the root cause, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new) with your cluster ID, shadow link name, and the timestamp of the issue. ### [](#topics-stuck-in-failing_over-state)Topics stuck in FAILING_OVER state **Problem**: Topics remain in `FAILING_OVER` state for extended periods **Solution**: Use `rpk shadow status`, the Data Plane API, or Prometheus metrics to check task and topic states, replication lag, and client-error counts, and ensure sufficient cluster resources (CPU, memory, disk space) are available on the shadow cluster. Verify network connectivity between shadow cluster nodes and confirm that all shadow topic partitions have elected leaders and the controller partition is properly replicated with an active leader. If topics remain stuck after addressing these cluster health issues and you need immediate failover, you can force delete the shadow link to failover all topics: #### Cloud UI All failover actions in the Cloud UI include force delete functionality by default. When you failover a shadow link, all topics are immediately promoted to writable status. #### rpk ```bash # Force delete the shadow link to failover all topics rpk shadow delete ``` `rpk shadow delete` force deletes the shadow link by default in Redpanda Cloud. #### Cloud API ```bash curl -X DELETE https://api.redpanda.com/v1/shadow-links/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` The `DELETE /shadow-links/` endpoint of the Control Plane API force deletes the shadow link by default in Redpanda Cloud. > ⚠️ **WARNING** > > Force deleting a shadow link immediately fails over all topics in the link. This action is irreversible and should only be used when topics are stuck and you need immediate access to all replicated data. ### [](#topics-in-faulted-state)Topics in FAULTED state **Problem**: Topics show `FAULTED` state and are not replicating **Solution**: Check for authentication issues, network connectivity problems, or source cluster unavailability. Verify that the shadow link service account still has the required permissions on the source cluster. Use `rpk shadow status`, the Data Plane API, or Prometheus metrics to check task and topic states, replication lag, and client-error counts for the faulted topics. ### [](#application-connection-failures)Application connection failures **Problem**: Applications cannot connect to shadow cluster after failover **Solution**: Verify shadow cluster broker endpoints are correct and check security group and firewall rules. Confirm authentication credentials are valid for the shadow cluster and test network connectivity from application hosts. ### [](#consumer-group-offset-issues)Consumer group offset issues **Problem**: Consumers start from beginning or wrong positions **Solution**: Verify consumer group offsets were replicated (check your filters) and use `rpk group describe ` to check offset positions. If necessary, manually reset offsets to appropriate positions. See [How to manage consumer group offsets in Redpanda](https://support.redpanda.com/hc/en-us/articles/23499121317399-How-to-manage-consumer-group-offsets-in-Redpanda) for detailed reset procedures. ## [](#next-steps)Next steps After successful failover, focus on recovery planning and process improvement. Begin by assessing the source cluster failure and determining whether to restore the original cluster or permanently promote the shadow cluster as your new primary. **Immediate recovery planning:** 1. **Assess source cluster**: Determine root cause of the outage 2. **Plan recovery**: Decide whether to restore source cluster or promote shadow cluster permanently 3. **Data synchronization**: Plan how to synchronize any data produced during failover 4. **Fail forward**: Create a new shadow link with the failed over shadow cluster as source to maintain a DR cluster **Process improvement:** 1. **Document the incident**: Record timeline, impact, and lessons learned 2. **Update runbooks**: Improve procedures based on what you learned 3. **Test regularly**: Schedule regular disaster recovery drills 4. **Review monitoring**: Ensure monitoring caught the issue appropriately --- # Page 599: Configure Failover **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover.md --- # Configure Failover > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Failover latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/failover page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/failover.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/failover.adoc description: Learn how to configure failover for disaster recovery. page-git-created-date: "2025-12-12" page-git-modified-date: "2026-05-26" --- Failover is the process of modifying shadow topics or an entire shadow cluster from read-only replicas to fully writable resources, and ceasing replication from the source cluster. You can fail over individual topics for selective workload migration or fail over the entire cluster for comprehensive disaster recovery. This critical operation transforms your shadow resources into operational production assets, allowing you to redirect application traffic when the source cluster becomes unavailable. You can failover a shadow link using the Redpanda Cloud UI, `rpk`, or the Data Plane API. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. > 📝 **NOTE** > > Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. ## [](#failover-behavior)Failover behavior When you initiate failover, Redpanda performs the following operations: 1. **Stops replication**: Halts all data fetching from the source cluster for the specified topics or entire shadow link 2. **Failover topics**: Converts read-only shadow topics into regular, writable topics 3. **Updates topic state**: Changes topic status from `ACTIVE` to `FAILING_OVER`, then `FAILED_OVER` Topic failover is irreversible. Once failed over, topics cannot return to shadow mode, and automatic fallback to the original source cluster is not supported. > 📝 **NOTE** > > To avoid a split-brain scenario after failover, ensure that all clients are reconfigured to point to the shadow cluster before resuming write activity. ## [](#failover-commands)Failover commands ### [](#get-data-plane-api-url)Get Data Plane API URL If using the Data Plane API, run the following to get the Data Plane API URL of the shadow cluster: ```bash export DATAPLANE_API_URL=`curl https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" | jq .cluster.dataplane_api` ``` You can perform failover at different levels of granularity to match your disaster recovery needs: ### [](#individual-topic-failover)Individual topic failover To fail over a specific shadow topic while leaving other topics in the shadow link still replicating, run: #### Cloud UI 1. On the **Shadow Link** page, select your shadow link. 2. For any of the topics you want to failover, click the corresponding **Failover** button. 3. Click to confirm the failover action. The failover process promotes the selected topics to writable status. #### rpk ```bash rpk shadow failover --topic ``` For detailed command options, see [`rpk shadow failover`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-failover/). #### Data Plane API Send a `POST /shadowlink/{shadow_link_name}/failover` request to the Data Plane API. Specify the name of the shadow topic in the request body: ```bash curl -X POST "$DATAPLANE_API_URL/v1/shadowlink//failover" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d '{ "shadowTopicName": "" }' ``` Use this approach when you need to selectively failover specific workloads or when testing failover procedures. ### [](#complete-shadow-link-failover-cluster-failover)Complete shadow link failover (cluster failover) To fail over all shadow topics associated with the shadow link simultaneously, run: #### Cloud UI 1. On the **Shadow Link** page, select your shadow link. 2. Click **Failover All Topics**. 3. Click to confirm the failover action. The failover process promotes all topics to writable status. #### rpk ```bash rpk shadow failover --all ``` #### Data Plane API Send a `POST /shadowlink/{shadow_link_name}/failover` request to the Data Plane API. If you do not specify a shadow topic in the request body, this command requests a failover of all shadow topics associated with the shadow link: ```bash curl -X POST "$DATAPLANE_API_URL/v1/shadowlink//failover" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` Use this approach during a complete regional disaster when you need to activate the entire shadow cluster as your new production environment. ### [](#force-delete-shadow-link-emergency-failover)Force delete shadow link (emergency failover) #### Cloud UI All failover actions in the Cloud UI include force delete functionality by default. When you failover a shadow link, all topics are immediately promoted to writable status. #### rpk `rpk shadow delete` force deletes the shadow link by default in Redpanda Cloud: ```bash rpk shadow delete ``` #### Control Plane API Use the Control Plane API to force delete a shadow link: ```bash curl -X DELETE 'https://api.redpanda.com/v1/shadow-links/' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` > ⚠️ **WARNING** > > Force deleting a shadow link is irreversible and immediately fails over all topics in the link, bypassing the normal failover state transitions. This action should only be used as a last resort when topics are stuck in transitional states and you need immediate access to all replicated data. ## [](#failover-states)Failover states ### [](#shadow-link-states)Shadow link states The shadow link itself has a simple state model: - **`ACTIVE`**: Shadow link is operating normally, replicating data - **`PAUSED`**: Shadow link replication is temporarily halted by user action Shadow links do not have dedicated failover states. Instead, the link’s operational status is determined by the collective state of its shadow topics. ### [](#shadow-topic-states)Shadow topic states Individual shadow topics progress through specific states during failover: - **`ACTIVE`**: Normal replication state before failover - **`FAULTED`**: Shadow topic has encountered an error and is not replicating - **`FAILING_OVER`**: Failover initiated, replication stopping - **`FAILED_OVER`**: Failover completed successfully, topic fully writable - **`PAUSED`**: Replication temporarily halted by user action ## [](#monitor-failover-progress)Monitor failover progress To monitor failover progress using the status command, run: ### Cloud UI Track the progress of failover operations from the **Shadow Link** page in the Cloud UI. ### rpk ```bash rpk shadow status ``` The output shows individual topic states and any issues encountered during the failover process. For detailed command options, see [`rpk shadow status`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-status/). ### Data Plane API ```bash curl "https://$DATAPLANE_API_URL/v1/shadowlinks/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` Task states during monitoring: - **`ACTIVE`**: Task is operating normally and replicating data - **`FAULTED`**: Task encountered an error and requires attention - **`NOT_RUNNING`**: Task is not currently executing - **`LINK_UNAVAILABLE`**: Task cannot communicate with the source cluster For detailed information about shadow link tasks and their roles, see [Shadow link tasks](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). ## [](#post-failover-cluster-behavior)Post-failover cluster behavior After successful failover, your shadow cluster exhibits the following characteristics: **Topic accessibility:** - Failed over topics become fully writable and readable. - Applications can produce and consume messages normally. - All Kafka APIs are available for failedover topics. - Original offsets and timestamps are preserved. **Shadow link status:** - The shadow link remains but stops replicating data. - Link status shows topics in `FAILED_OVER` state. - You can safely delete the shadow link after successful failover. **Operational limitations:** - No automatic fallback mechanism to the original source cluster. - Data transforms remain disabled until you manually re-enable them. - Audit log history from the source cluster is not available (new audit logs begin immediately). ## [](#failover-considerations-and-limitations)Failover considerations and limitations Before implementing failover procedures, understand these key considerations that affect your disaster recovery strategy and operational planning. **Data consistency:** - Some data loss may occur due to replication lag at the time of failover. - Consumer group offsets are preserved, allowing applications to resume from their last committed position. - In-flight transactions at the source cluster are not replicated and will be lost. **Recovery-point-objective (RPO):** The amount of potential data loss depends on replication lag when disaster occurs. Monitor lag metrics to understand your effective RPO. **Network partitions:** If the source cluster becomes accessible again after failover, do not attempt to write to both clusters simultaneously. This creates a scenario with potential data inconsistencies, since metadata starts to diverge. **Testing requirements:** Regularly test failover procedures in non-production environments to validate your disaster recovery processes and measure RTO. ## [](#next-steps)Next steps After completing failover: - Update your application connection strings to point to the shadow cluster - Verify that applications can produce and consume messages normally - Consider deleting the shadow link if failover was successful and permanent For emergency situations, see [Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/). --- # Page 600: Monitor Shadowing **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/monitor.md --- # Monitor Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Monitor Shadowing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/monitor page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/monitor.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/monitor.adoc description: Learn how to monitor shadowing for disaster recovery. page-git-created-date: "2025-12-12" page-git-modified-date: "2026-05-26" --- Monitor your [shadow links](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/setup/) to ensure proper replication performance and understand your disaster recovery readiness. Use `rpk` commands, metrics, and status information to track shadow link health and troubleshoot issues. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. ## [](#status-commands)Status commands To list existing shadow links: ### Cloud UI At the organization level of the Cloud UI, navigate to **Shadow Link**. ### rpk ```bash rpk shadow list ``` ### Control Plane API ```bash curl 'https://api.redpanda.com/v1/shadow-links' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` To view shadow link configuration details: ### Cloud UI 1. From the **Shadow Link** page, select the shadow link you want to view. 2. Click the **Tasks** tab to view all tasks and their status. ### rpk ```bash rpk shadow describe ``` For detailed command options, see [`rpk shadow list`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-list/) and [`rpk shadow describe`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-describe/). This command shows the complete configuration of the shadow link, including connection settings, filters, and synchronization options. ### Control Plane API ```bash curl 'https://api.redpanda.com/v1/shadow-links/' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` To check your shadow link status and ensure proper operation: ### Cloud UI 1. From the **Shadow Link** page, select the shadow link you want to view. 2. Click the **Tasks** tab to view all tasks and their status. ### rpk ```bash rpk shadow status ``` For troubleshooting specific issues, you can use command options to show individual status sections. See [`rpk shadow status`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-status/) for available status options. The status output includes the following: ### Cloud API ```bash # Get Data Plane API URL of shadow cluster export DATAPLANE_API_URL=`curl https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" | jq .cluster.dataplane_api` curl "https://$DATAPLANE_API_URL/v1/shadowlinks/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" # View topic state curl "https://$DATAPLANE_API_URL/v1/shadowlinks//topic" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` The status includes the following: - **Shadow link state**: Overall operational state (`ACTIVE`, `PAUSED`). - **Individual topic states**: Current state of each replicated topic (`ACTIVE`, `FAULTED`, `FAILING_OVER`, `FAILED_OVER`, `PAUSED`). - **Task status**: Health of replication tasks across brokers (`ACTIVE`, `FAULTED`, `NOT_RUNNING`, `LINK_UNAVAILABLE`). For details about shadow link tasks, see [Shadow link tasks](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). - **Lag information**: Replication lag per partition showing source vs shadow high watermarks (HWM). ## [](#shadow-link-metrics)Metrics Shadowing provides comprehensive metrics to track replication performance and health with the [`public_metrics`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/) endpoint. | Metric | Type | Description | | --- | --- | --- | | redpanda_shadow_link_shadow_lag | Gauge | The lag of the shadow partition against the source partition, calculated as source partition LSO (Last Stable Offset) minus shadow partition HWM (High Watermark). Monitor by shadow_link_name, topic, and partition to understand replication lag for each partition. | | redpanda_shadow_link_total_bytes_fetched | Count | The total number of bytes fetched by a sharded replicator (bytes received by the client). Labeled by shadow_link_name and shard to track data transfer volume from the source cluster. | | redpanda_shadow_link_total_bytes_written | Count | The total number of bytes written by a sharded replicator (bytes written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor data written to the shadow cluster. | | redpanda_shadow_link_client_errors | Count | The number of errors seen by the client. Track by shadow_link_name and shard to identify connection or protocol issues between clusters. | | redpanda_shadow_link_shadow_topic_state | Gauge | Number of shadow topics in the respective states. Labeled by shadow_link_name and state to monitor topic state distribution across your shadow links. | | redpanda_shadow_link_total_records_fetched | Count | The total number of records fetched by the sharded replicator (records received by the client). Monitor by shadow_link_name and shard to track message throughput from the source. | | redpanda_shadow_link_total_records_written | Count | The total number of records written by a sharded replicator (records written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor message throughput to the shadow cluster. | See also: [Metrics Reference](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/) ## [](#monitoring-best-practices)Monitoring best practices ### [](#health-check-procedures)Health check procedures Establish regular monitoring workflows to ensure shadow link health: #### Cloud UI 1. From the **Shadow Link** page, select the shadow link you want to view. 2. Click the **Tasks** tab to view all tasks and their status. #### rpk ```bash # Check all shadow links are active rpk shadow list | grep -v "ACTIVE" || echo "All shadow links healthy" # Monitor lag for critical topics rpk shadow status | grep -E "LAG|Lag" ``` #### Cloud API ```bash # Check all shadow links are active curl 'https://api.redpanda.com/v1/shadow-links' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" | \ jq -r 'if all(.state == "SHADOW_LINK_STATE_ACTIVE") then "All shadow links healthy" else .[] | select(.state != "SHADOW_LINK_STATE_ACTIVE") end' # Monitor lag for critical topics curl "https://$DATAPLANE_API_URL/v1/shadowlinks//topic" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" ``` ### [](#alert-conditions)Alert conditions Configure monitoring alerts for the following conditions, which indicate problems with Shadowing: - **High replication lag**: When `redpanda_shadow_link_shadow_lag` exceeds your RPO requirements - **Connection errors**: When `redpanda_shadow_link_client_errors` increases rapidly - **Topic state changes**: When topics move to `FAULTED` state - **Task failures**: When replication tasks enter `FAULTED` or `NOT_RUNNING` states - **Throughput drops**: When bytes/records fetched drops significantly - **Link unavailability**: When tasks show `LINK_UNAVAILABLE` indicating source cluster connectivity issues For more information about shadow link tasks and their states, see [Shadow link tasks](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). --- # Page 601: Shadowing Overview **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview.md --- # Shadowing Overview > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Shadowing Overview latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/overview page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/overview.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/overview.adoc description: Overview of shadowing for disaster recovery in Redpanda Cloud. page-git-created-date: "2025-12-12" page-git-modified-date: "2026-05-26" --- > 📝 **NOTE** > > Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. Shadowing is Redpanda’s enterprise-grade disaster recovery solution that establishes asynchronous, offset-preserving replication between two distinct Redpanda clusters. A cluster is able to create a dedicated client that continuously replicates source cluster data, including offsets, timestamps, and cluster metadata. This creates a read-only shadow cluster that you can quickly failover to handle production traffic during a disaster. Shadowing keeps data flowing, even during regional outages. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. Unlike traditional replication tools that re-produce messages, Shadowing copies data at the byte level, ensuring shadow topics contain identical copies of source topics with preserved offsets and timestamps. Shadowing replicates: - **Topic data**: All records with preserved offsets and timestamps - **Topic configurations**: Partition counts, retention policies, and other topic properties - **Consumer group offsets**: Enables seamless consumer resumption after failover - **Access control lists (ACLs)**: User permissions and security policies - **Schema Registry data**: Schema definitions, versions, and compatibility settings, replicated from another Redpanda cluster or from a Confluent Schema Registry ## [](#how-shadowing-fits-into-disaster-recovery)How Shadowing fits into disaster recovery Shadowing addresses enterprise disaster recovery requirements driven by regulatory compliance and business continuity needs. Organizations typically want to minimize both recovery time objective (RTO) and recovery point objective (RPO), and Shadowing asynchronous replication helps you achieve both goals by reducing data loss during regional outages and enabling rapid application recovery. The architecture follows an active-passive pattern. The source cluster processes all production traffic while the shadow cluster remains in read-only mode, continuously receiving updates. If a disaster occurs, you can failover the shadow topics, making them fully writable. At that point, you can redirect your applications to the shadow cluster, which becomes the new production cluster. > 📝 **NOTE** > > To avoid a split-brain scenario after failover, ensure that all clients are reconfigured to point to the shadow cluster before resuming write activity. Shadowing complements Redpanda’s existing availability and recovery capabilities. High availability actively protects your day-to-day operations, handling reads and writes seamlessly during node or availability zone failures within a region. Shadowing is your safety net for catastrophic regional disasters. Shadowing delivers near real-time, cross-region replication for mission-critical applications that require rapid failover with minimal data loss. ## [](#limitations)Limitations Shadowing for disaster recovery currently has the following limitations: - Shadowing is designed for active-passive disaster recovery scenarios. Each shadow cluster can maintain only one shadow link. - Shadowing operates exclusively in asynchronous mode and doesn’t support active-active configurations. This means there will always be some replication lag. - [Data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/) are not supported on shadow clusters while Shadowing is active. Writing to shadow topics is blocked. - During a disaster, [audit log](https://docs.redpanda.com/cloud-data-platform/manage/audit-logging/) history from the source cluster is lost, though the shadow cluster begins generating new audit logs immediately after the failover. - After you failover shadow topics, automatic fallback to the original source cluster is not supported. ## [](#shadow-link-tasks)Shadow link tasks Shadow linking operates through specialized tasks that handle different aspects of replication. If you use a `shadow-config.yaml` configuration file to create the shadow link, each task corresponds to a section in the file. Tasks run continuously to maintain synchronization with the source cluster. #### Source Topic Sync The **Source Topic Sync task** manages topic discovery and metadata synchronization. This task periodically queries the source cluster to discover available topics, applies your configured topic filters to determine which topics should become shadow topics, and synchronizes topic properties between clusters. The task is controlled by the `topic_metadata_sync_options` section in the configuration file. It includes: - **Auto-creation filters**: Determines which source topics automatically become shadow topics - **Property synchronization**: Controls which topic properties replicate from source to shadow - **Starting offset**: Sets where new shadow topics begin replication (earliest, latest, or timestamp-based) - **Sync interval**: How frequently to check for new topics and property changes When this task discovers a new topic that matches your filters, it creates the corresponding shadow topic and begins replication from your configured starting offset. #### Consumer Group Shadowing The **Consumer Group Shadowing task** replicates consumer group offsets and membership information from the source cluster. This ensures that consumer applications can resume processing from the correct position after failover. The task is controlled by the `consumer_offset_sync_options` section in the configuration file. It includes: - **Group filters**: Determines which consumer groups have their offsets replicated - **Sync interval**: How frequently to synchronize consumer group offsets - **Offset clamping**: Automatically adjusts replicated offsets to valid ranges on the shadow cluster This task runs on brokers that host the `__consumer_offsets` topic and continuously tracks consumer group coordinators to optimize offset synchronization. #### Security Migrator The **Security Migrator task** replicates security policies, primarily ACLs (access control lists), from the source cluster to maintain consistent authorization across both environments. The task is controlled by the `security_sync_options` section in the configuration file. It includes: - **ACL filters**: Determines which security policies replicate - **Sync interval**: How frequently to synchronize security settings By default, all ACLs replicate to ensure your shadow cluster maintains the same security posture as your source cluster. #### Schema Registry Sync The **Schema Registry Sync task** replicates Schema Registry content so that applications that depend on schemas keep working after failover. The task is controlled by the `schema_registry_sync_options` section in the configuration file. It supports two modes: - **Topic mode** (`shadow_schema_registry_topic`): Shadows the `_schemas` system topic for byte-for-byte replication from another Redpanda cluster. - **API mode** (`shadow_schema_registry_api`): Polls the source Schema Registry over HTTP and imports selected contexts and subjects, with validation. Use this mode to replicate schemas from a Confluent Schema Registry. A shadow link uses one mode or the other, not both. Only API mode runs as a separate task that appears in the shadow link status. Topic mode adds the `_schemas` topic to the set of shadowed topics, so it is monitored like any other shadow topic rather than as a separate task. ### [](#task-status-and-monitoring)Task status and monitoring Each task reports its status through the shadow link status API. Task states include: - **`ACTIVE`**: Task is running normally and performing synchronization - **`PAUSED`**: Task has been manually paused through configuration - **`FAULTED`**: Task encountered an error and requires attention - **`NOT_RUNNING`**: Task is not currently executing - **`LINK_UNAVAILABLE`**: Task cannot communicate with the source cluster You can pause individual tasks by setting the `paused` field to `true` in the corresponding configuration section. This allows you to selectively disable parts of the replication process without affecting the entire shadow link. For monitoring task health and troubleshooting task issues, see [Monitor Shadowing](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/monitor/). ## [](#what-gets-replicated)What gets replicated Shadowing replicates your topic data with complete fidelity, preserving all message records with their original offsets, timestamps, headers, and metadata. The partition structure remains identical between source and shadow clusters, ensuring applications can resume processing from the exact same position after failover. Consumer group data flows according to your group filters, replicating offsets and membership information for matched groups. ACLs replicate based on your security filters. Schema Registry data synchronizes schema definitions, versions, and compatibility settings by shadowing the `_schemas` topic. Partition count is always replicated to ensure the shadow topic matches the source topic’s partition structure. ### [](#topic-properties-replication)Topic properties replication The [Source Topic Sync task](#shadow-link-tasks) handles topic property replication. For topic properties, Redpanda follows these replication rules: **Never replicated** - `redpanda.remote.readreplica` - `redpanda.remote.recovery` - `redpanda.remote.allowgaps` - `redpanda.virtual.cluster.id` - `redpanda.leaders.preference` - `redpanda.cloud_topic.enabled` **Always replicated** - `max.message.bytes` - `cleanup.policy` - `message.timestamp.type` **Always replicated (unless `exclude_default` is `true`)** - `compression.type` - `retention.bytes` - `retention.ms` - `delete.retention.ms` - `replication.factor` - `min.compaction.lag.ms` - `max.compaction.lag.ms` To replicate additional topic properties, explicitly list them in `synced_shadow_topic_properties`. The filtering system you configure determines the precise scope of replication across all components, allowing you to balance comprehensive disaster recovery with operational efficiency. ## [](#best-practices)Best practices To ensure reliable disaster recovery with Shadowing: - **Do not modify shadow topic properties**: Avoid modifying synced topic properties on shadow topics, as these properties automatically revert to source topic values. ## [](#implementation-overview)Implementation overview Choose your implementation approach: - **[Setup and Configuration](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/setup/)**: Initial shadow configuration, authentication, and topic selection - **[Monitoring and Operations](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/monitor/)**: Health checks, lag monitoring, and operational procedures - **[Planned Failover](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover/)**: Controlled disaster recovery testing and migrations - **[Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/)**: Rapid disaster response procedures > 💡 **TIP** > > You can create and manage shadow links with the Redpanda Cloud UI, the [Cloud API](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview), or `rpk`, giving you flexibility in how you interact with your disaster recovery infrastructure. ## [](#next-steps)Next steps After setting up Shadowing for your Redpanda clusters, consider these additional steps: - **Test your disaster recovery procedures**: Regularly practice failover scenarios in a non-production environment. See [Failover Runbook](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/failover-runbook/) for step-by-step disaster procedures. - **Monitor shadow link health**: Set up alerting on the metrics described above to ensure early detection of replication issues. - **Implement automated failover**: Consider developing automation scripts that can detect outages and initiate failover based on predefined criteria. - **Review security policies**: Ensure your ACL filters replicate the appropriate security settings for your disaster recovery environment. - **Document your configuration**: Maintain up-to-date documentation of your shadow link configuration, including network settings, authentication details, and filter definitions. --- # Page 602: Configure Shadowing **URL**: https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/setup.md --- # Configure Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Configure Shadowing latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: disaster-recovery/shadowing/setup page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: disaster-recovery/shadowing/setup.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/setup.adoc description: Learn how to configure shadowing for disaster recovery. page-git-created-date: "2025-12-12" page-git-modified-date: "2026-05-26" --- You can create and manage shadow links with the Redpanda Cloud UI, the [Cloud API](https://docs.redpanda.com/api/doc/cloud-controlplane/topic/topic-cloud-api-overview), or `rpk`, giving you flexibility in how you interact with your disaster recovery infrastructure. > 💡 **TIP** > > Deploy clusters in different geographic regions to protect against regional disasters. ## [](#prerequisites)Prerequisites ### [](#license-and-cluster-requirements)License and cluster requirements Shadowing is supported on BYOC and Dedicated clusters running Redpanda version 25.3 and later. ### [](#cluster-configuration)Cluster configuration The shadow cluster must have the [`enable_shadow_linking`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#enable_shadow_linking) cluster property set to `true`. > 📝 **NOTE** > > Starting with Redpanda v25.3, this cluster property is enabled by default on new Redpanda Cloud clusters. For existing clusters on versions earlier than v25.3, you must enable this property manually. See [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). ### [](#replication-service-permissions)Replication service permissions You must configure a service account on the source cluster with the following [ACL](https://docs.redpanda.com/cloud-data-platform/security/authorization/acl/) permissions for shadow link replication: - **Topics**: `read` permission on all topics you want to replicate - **Topic configurations**: `describe_configs` permission on topics for configuration synchronization - **Consumer groups**: `describe` and `read` permission on consumer groups for offset replication - **ACLs**: `describe` permission on ACL resources to replicate security policies - **Cluster**: `describe` permission on the cluster resource to access ACLs This service account authenticates from the shadow cluster to the source cluster and performs the actual data replication. The credentials for this account are provided when you set up the shadow link. ### [](#network-and-authentication)Network and authentication You must configure network connectivity between clusters with appropriate firewall rules to allow the shadow cluster to connect to the source cluster for data replication. Shadowing uses a pull-based architecture where the shadow cluster fetches data from the source cluster. For detailed networking configuration, see [Networking](#networking). If using [authentication](https://docs.redpanda.com/cloud-data-platform/security/cloud-authentication/) for the shadow link connection, configure the source cluster with your chosen authentication method (SASL/SCRAM, TLS, mTLS) and ensure the shadow cluster has the proper credentials to authenticate to the source cluster. ## [](#set-up-shadowing)Set up Shadowing To set up Shadowing, you need to create a shadow link and configure filters to select which topics, consumer groups, ACLs, and Schema Registry data to replicate. If using the Cloud API to set up Shadowing, you must [authenticate](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication) to the API by including an access token in your requests. ### [](#create-a-shadow-link)Create a shadow link Any BYOC or Dedicated cluster can create a shadow link to a source cluster. > 💡 **TIP** > > You can use `rpk` to generate a sample configuration file with common filter patterns: > > ```bash > # Generate a sample configuration file with placeholder values > rpk shadow config generate --for-cloud -o shadow-config.yaml > ``` > > This creates a complete YAML configuration file that you can customize for your environment. The template includes all available fields with comments explaining their purpose. For detailed command options, see [`rpk shadow config generate --for-cloud`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-config-generate/). Explore the configuration file ```yaml # Sample ShadowLinkConfig YAML with all fields name: # Unique name for this shadow link, example: "production-dr" cloud_options: # Use either source_redpanda_id or bootstrap_servers: only one is required. source_redpanda_id: # Optional: 20 character lowercase ID of the cluster # Example: m7xtv2qq5njbhwruk88f shadow_redpanda_id: # 20 character lowercase ID of the cluster # Example: m7xtv2qq5njbhwruk88f client_options: bootstrap_servers: # Source cluster brokers to connect to - : # Example: "prod-kafka-1.example.com:9092" - : # Example: "prod-kafka-2.example.com:9092" - : # Example: "prod-kafka-3.example.com:9092" source_cluster_id: # Optional: UUID assigned by Redpanda # Example: a882bc98-7aca-40f6-a657-36a0b4daf1fd # This UUID is not available in Redpanda Cloud. # TLS settings using PEM strings tls_settings: enabled: true tls_pem_settings: ca: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- key: ${secrets.} cert: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- # Create SASL credentials in the source cluster. # Then, with this configuration, ensure the shadow cluster uses the credentials # to authenticate to the source cluster. authentication_configuration: # SASL/SCRAM authentication scram_configuration: username: # SASL/SCRAM username, example: "shadow-replication-user" password: ${secrets.} # ID of secret containing SASL/SCRAM password scram_mechanism: SCRAM_SHA_256 # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512" # Connection tuning - adjust based on network characteristics metadata_max_age_ms: 10000 # How often to refresh cluster metadata (default: 10000ms) connection_timeout_ms: 1000 # Connection timeout (default: 1000ms, increase for high latency) retry_backoff_ms: 100 # Backoff between retries (default: 100ms) fetch_wait_max_ms: 500 # Max time to wait for fetch requests (default: 500ms) fetch_min_bytes: 5242880 # Min bytes per fetch (default: 5MB) fetch_max_bytes: 20971520 # Max bytes per fetch (default: 20MB) fetch_partition_max_bytes: 5242880 # Max bytes per partition fetch (default: 5MB) topic_metadata_sync_options: interval: 30s # How often to sync topic metadata (examples: "30s", "1m", "5m") auto_create_shadow_topic_filters: # Filters for automatic topic creation - pattern_type: LITERAL # Include all topics (wildcard) filter_type: INCLUDE name: '*' - pattern_type: PREFIX # Exclude topics with specific prefix filter_type: EXCLUDE name: # Examples: "temp-", "test-", "debug-" synced_shadow_topic_properties: # Additional topic properties to sync (beyond defaults) - retention.ms # Topic retention time - segment.ms # Segment roll time exclude_default: false # Include default properties (compression, retention, etc.) start_at_earliest: {} # Start from the beginning of source topics (default) paused: false # Enable topic metadata synchronization consumer_offset_sync_options: interval: 30s # How often to sync consumer group offsets paused: false # Enable consumer offset synchronization group_filters: # Filters for consumer groups to sync - pattern_type: LITERAL filter_type: INCLUDE name: '*' # Include all consumer groups security_sync_options: interval: 30s # How often to sync security settings paused: false # Enable security settings synchronization acl_filters: # Filters for ACLs to sync - resource_filter: resource_type: TOPIC # Resource type: "TOPIC", "GROUP", "CLUSTER" pattern_type: PREFIXED # Pattern type: "LITERAL", "PREFIXED" name: # Examples: "prod-", "app-data-" access_filter: principal: User: # Principal name, example: "User:app-service" operation: ANY # Operation: "READ", "WRITE", "CREATE", "DELETE", "ALTER", "DESCRIBE", "ANY" permission_type: ALLOW # Permission: "ALLOW" or "DENY" host: '*' # Host pattern, examples: "*", "10.0.0.0/8", "app-server.example.com" schema_registry_sync_options: # Schema Registry synchronization options shadow_schema_registry_topic: {} # Enable byte-for-byte _schemas topic replication ``` Because the shadow cluster pulls from the source cluster, the shadow cluster requires credentials to connect to the source cluster. And because you cannot store plaintext passwords in Redpanda Cloud, you must create a secret to hold the password for the user on the source cluster. If using mTLS, you must also create a secret to hold the key of the client certificate for the client to authenticate. Reference that secret in `client_options.tls_settings.key_file` in the configuration file. 1. In the shadow cluster, create the secret: #### Cloud UI In the shadow cluster, go to the **Secrets Store** page and create a secret for the source cluster user, scoped to Redpanda Cluster. If necessary, first create the user with all ACLs enabled in the source cluster. #### rpk In the shadow cluster, create a secret to store the authentication credential that the cluster will use (`"scram_configuration": "password"` in the example configuration in the next step). Your secret must be scoped to "Redpanda Cluster". Use [`rpk security secret create`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-security/rpk-security-secret-create/) to create the secret from the command line. #### Data Plane API In the shadow cluster, create a secret to store the authentication credential that the cluster will use (`"scram_configuration": "password"` in the example configuration in the next step). Your secret must be scoped to "Redpanda Cluster". Use the [Data Plane API](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-dataplane-api/) to programmatically create the secret. #### Terraform With Terraform, you define the secret and the shadow link together: the [`redpanda_secret` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/secret) in the next step’s Terraform tab creates this secret as part of the same configuration, so there is nothing to do in this step. 2. In the shadow cluster, create a shadow link to the source cluster. #### Cloud UI 1. At the organization level of the Cloud UI, navigate to **Shadow Link**. 2. Click **Create shadow link**. 3. Enter a unique name for the shadow link. The name must start and end with lowercase alphanumeric characters, hyphens allowed. 4. Select the source cluster from which data will be replicated. You can select an existing Redpanda Cloud cluster, or you can enter a bootstrap server URL to connect to any Kafka-compatible cluster. For an existing Redpanda Cloud cluster, you select the specific cluster on the next page. 5. Enter the authorization and authentication details from the source cluster, including the user and the name of the secret containing the password created in the previous step. 6. Optionally, expand **Advanced options** to configure client connection properties. 7. Click **Save** to apply changes. #### rpk 1. Run `rpk cloud login`. Select your shadow cluster when prompted. 2. To create a shadow link with the source cluster using `rpk`, run the following command from the shadow cluster: ```bash # When logged in, optionally create a new rpk profile to easily # switch to the shadow cluster rpk profile create --from-cloud shadow-cluster # Use the generated configuration file to create the shadow link rpk shadow create --config-file shadow-config.yaml ``` For detailed command options, see [`rpk shadow create`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-create/). > 💡 **TIP** > > Use [`rpk profile`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/config-rpk-profile/) to save your cluster connection details and credentials for both source and shadow clusters. This allows you to easily switch between the two configurations. #### Control Plane API To create a shadow link using the Control Plane API, make a `POST /shadow-links` request from the shadow cluster: ```bash curl -X POST 'https://api.redpanda.com/v1/shadow-links' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d '{ "shadow_link": { "shadow_redpanda_id": "", "name": "", "client_options": { "bootstrap_servers": [":", ":", ":"], "tls_settings": { "enabled": true }, "authentication_configuration": { "scram_configuration": { "username": "", "password": "${secrets.}", "scram_mechanism": "SCRAM_MECHANISM_SCRAM_SHA_256" } } }, "topic_metadata_sync_options": { "interval": "30s", "auto_create_shadow_topic_filters": [ { "name": "*", "filter_type": "FILTER_TYPE_INCLUDE", "pattern_type": "PATTERN_TYPE_LITERAL" }, { "name": "", "filter_type": "FILTER_TYPE_EXCLUDE", "pattern_type": "PATTERN_TYPE_PREFIX" } ], "start_at_earliest": {}, "paused": false }, "consumer_offset_sync_options": { "paused": true }, "security_sync_options": { "paused": true } } }' ``` Replace the placeholders with your own values: - ``: ID of the shadow (destination) cluster. - ``: Unique name for this shadow link, for example, `production-dr`. - `:`, `: …​`: Source cluster brokers to connect to, for example, `prod-kafka-1.example.com:9092`, `prod-kafka-2.example.com:9092`. - ``: SASL/SCRAM username, for example, `shadow-replication-user`. You create this user in the source cluster. - ``: The name of the secret containing the SASL/SCRAM password from the source cluster. - ``: Exclude topics that use this prefix, for example, `temp-`, `test-`, `debug-`. The response object represents the [long-running operation](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/#lro) of creating a shadow link. For the full API reference, see [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-shadowlinkservice_createshadowlink). #### Terraform Manage shadow links declaratively with the [`redpanda_shadow_link` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/shadow_link) in the [Redpanda Terraform provider](https://registry.terraform.io/providers/redpanda-data/redpanda/latest) (version 2.0.0 or later). The resource supports the full shadow link lifecycle: create, update, import, and destroy. Store the SASL/SCRAM password for the source cluster user in the shadow cluster’s secret store with the [`redpanda_secret` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/secret) (see [Manage cluster secrets](https://docs.redpanda.com/cloud-data-platform/manage/terraform-provider/#manage-cluster-secrets) for naming, write-only behavior, and rotation), then reference the secret from the shadow link configuration: ```hcl resource "redpanda_secret" "source_password" { name = "" secret_data = var.source_user_password secret_data_version = 1 # Increment when you rotate the password. scopes = ["SCOPE_REDPANDA_CLUSTER"] cluster_api_url = redpanda_cluster.shadow.cluster_api_url } resource "redpanda_shadow_link" "production_dr" { name = "" shadow_redpanda_id = "" source_redpanda_id = "" client_options = { tls_settings = { enabled = true } authentication_configuration = { scram_configuration = { scram_mechanism = "SCRAM_SHA_256" username = "" password = "$${secrets.${redpanda_secret.source_password.name}}" } } } } ``` Replace the placeholders with your own values: - ``: Name for the secret that stores the SASL/SCRAM password. Must be uppercase, matching `^[A-Z][A-Z0-9_]*$`, for example, `SOURCE_SCRAM_PASSWORD`. - ``: Unique name for this shadow link, for example, `production-dr`. - ``: ID of the shadow (destination) cluster. - ``: ID of the source cluster. For a source cluster that is not managed in Redpanda Cloud, omit `source_redpanda_id` and set `client_options.bootstrap_servers` instead. - ``: SASL/SCRAM username, for example, `shadow-replication-user`. You create this user in the source cluster, and you can manage it with the [`redpanda_user` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/user). Grant it the permissions listed in [Replication service permissions](#replication-service-permissions), which you can manage with the [`redpanda_acl` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/acl). To configure topic, consumer group, ACL, and Schema Registry synchronization, add the corresponding option blocks (`topic_metadata_sync_options`, `consumer_offset_sync_options`, `security_sync_options`, `schema_registry_sync_options`) to the resource. For the full schema and a complete working example, see the [`redpanda_shadow_link` reference](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/shadow_link) and the [provider example](https://github.com/redpanda-data/terraform-provider-redpanda/blob/main/examples/shadow_link/main.tf). > 📝 **NOTE** > > If you manage the shadow cluster with the [`redpanda_cluster` resource](https://registry.terraform.io/providers/redpanda-data/redpanda/latest/docs/resources/cluster), you can set the `enable_shadow_linking` cluster property in `cluster_configuration.custom_properties_json`. > > To protect against accidental deletion, Terraform refuses to destroy a shadow link unless the resource sets `allow_deletion = true`. ### [](#set-filters)Set filters Filters determine which resources Shadowing automatically creates when establishing your shadow link. Topic filters select which topics Shadowing automatically creates as shadow topics when they appear on the source cluster. After Shadowing creates a shadow topic, it continues replicating until you failover the topic, delete it, or delete the entire shadow link. Consumer group and ACL filters control which groups and security policies replicate to maintain application functionality. #### [](#filter-types-and-patterns)Filter types and patterns Each filter uses two key settings: - **Pattern type**: Determines how names are matched - `LITERAL`: Matches names exactly (including the special wildcard `*` to match all items) - `PREFIX`: Matches names that start with the specified string - **Filter type**: Specifies whether to INCLUDE or EXCLUDE matching items - `INCLUDE`: Replicate items that match the pattern - `EXCLUDE`: Skip items that match the pattern #### [](#filter-processing-rules)Filter processing rules Redpanda processes filters in the order you define them with EXCLUDE filters taking precedence. Design your filter lists carefully: 1. **Exclude filters win**: If any EXCLUDE filter matches a resource, it is excluded regardless of INCLUDE filters. 2. **Order matters for INCLUDE filters**: Among INCLUDE filters, the first match determines the result. 3. **Default behavior**: Items that don’t match any filter are excluded from replication. #### [](#common-filtering-patterns)Common filtering patterns Replicate all topics except test topics: ```yaml topic_metadata_sync_options: auto_create_shadow_topic_filters: - pattern_type: PREFIX filter_type: EXCLUDE name: test- # Exclude all test topics - pattern_type: LITERAL filter_type: INCLUDE name: '*' # Include all other topics ``` Replicate only production topics: ```yaml topic_metadata_sync_options: auto_create_shadow_topic_filters: - pattern_type: PREFIX filter_type: INCLUDE name: prod- # Include production topics - pattern_type: PREFIX filter_type: INCLUDE name: production- # Alternative production prefix ``` Replicate specific consumer groups: ```yaml consumer_offset_sync_options: group_filters: - pattern_type: LITERAL filter_type: INCLUDE name: critical-app-consumers # Include specific consumer group - pattern_type: PREFIX filter_type: INCLUDE name: prod-consumer- # Include production consumers ``` #### [](#schema-registry-synchronization)Schema Registry synchronization Shadowing can replicate Schema Registry data in one of two modes: - **Topic mode** (`shadow_schema_registry_topic`): Shadows the `_schemas` system topic for byte-for-byte replication of schema definitions, versions, and compatibility settings from another Redpanda cluster. - **API mode** (`shadow_schema_registry_api`): Polls the source Schema Registry over HTTP and imports selected contexts and subjects, with validation. Use this mode to replicate schemas from a Confluent Schema Registry, or to replicate only part of the source registry. A shadow link uses one mode or the other, not both. To enable topic mode, add the following to your shadow link configuration: ```yaml schema_registry_sync_options: shadow_schema_registry_topic: {} ``` Topic mode requirements: - The `_schemas` topic must exist on the source cluster - The `_schemas` topic must not exist on the shadow cluster, or must be empty - Once enabled, the `_schemas` topic will be replicated completely Important: After the `_schemas` topic becomes a shadow topic, it cannot be stopped without either failing over the topic or deleting it entirely. #### [](#system-topic-filtering-rules)System topic filtering rules Redpanda system topics have the following specific filtering restrictions: - Literal filters for `__consumer_offsets` and `_redpanda.audit_log` are rejected. - Prefix filters for topics starting with `_redpanda` or `__redpanda` are rejected. - Wildcard `*` filters will not match topics that start with `_redpanda` or `__redpanda`. - To shadow specific system topics, you must provide explicit literal filters for those individual topics. #### [](#acl-filtering)ACL filtering ACLs are replicated by the [Security Migrator task](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). This is recommended to ensure that your shadow cluster has the same permissions as your source cluster. To configure ACL filters: ```yaml security_sync_options: acl_filters: # Include read permissions for production topics - resource_filter: resource_type: TOPIC # Filter by topic resource pattern_type: PREFIXED # Match by prefix name: prod- # Production topic prefix access_filter: principal: User:app-user # Application service user operation: READ # Read operation permission_type: ALLOW # Allow permission host: '*' # Any host # Include consumer group permissions - resource_filter: resource_type: GROUP # Filter by consumer group pattern_type: LITERAL # Exact match name: '*' # All consumer groups access_filter: principal: User:app-user # Same application user operation: READ # Read operation permission_type: ALLOW # Allow permission host: '*' # Any host ``` #### [](#consumer-group-filtering-and-behavior)Consumer group filtering and behavior Consumer group filters determine which consumer groups have their offsets replicated to the shadow cluster by the [Consumer Group Shadowing task](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). Offset replication operates selectively within each consumer group. Only committed offsets for active shadow topics are synchronized, even if the consumer group has offsets for additional topics that aren’t being shadowed. For example, if consumer group "app-consumers" has committed offsets for "orders", "payments", and "inventory" topics, but only "orders" is an active shadow topic, then only the "orders" offsets will be replicated to the shadow cluster. ```yaml consumer_offset_sync_options: interval: 30s # How often to sync consumer group offsets paused: false # Enable consumer offset synchronization group_filters: - pattern_type: PREFIX filter_type: INCLUDE name: prod-consumer- # Include production consumer groups - pattern_type: LITERAL filter_type: EXCLUDE name: test-consumer-group # Exclude specific test groups ``` ##### [](#important-consumer-group-considerations)Important consumer group considerations **Avoid name conflicts:** If you plan to consume data from the shadow cluster, do not use the same consumer group names as those used on the source cluster. While this won’t break shadow linking, it can impact your RPO/RTO because conflicting group names may interfere with offset replication and consumer resumption during disaster recovery. **Offset clamping:** When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition’s HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster. #### [](#starting-offset-for-new-shadow-topics)Starting offset for new shadow topics When the [Source Topic Sync task](https://docs.redpanda.com/cloud-data-platform/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks) creates a shadow topic for the first time, you can control where replication begins on the source topic. This setting only applies to empty shadow partitions and is crucial for disaster recovery planning. Changing this configuration only affects new shadow topics, existing shadow topics continue replicating from their current position. ```yaml topic_metadata_sync_options: start_at_earliest: {} ``` Alternatively, to start from the most recent offset: ```yaml topic_metadata_sync_options: start_at_latest: {} ``` Or to start from a specific timestamp: ```yaml topic_metadata_sync_options: start_at_timestamp: 2024-01-01T00:00:00Z ``` Starting offset options: - **`earliest`** (default): This replicates all existing data from the source topic. Use this for complete disaster recovery where you need full data history. - **`latest`**: This starts replication from the current end of the source topic, skipping existing data. Use this when you only need new data for disaster recovery and want to minimize initial replication time. - **`timestamp`**: This starts replication from the first record with a timestamp at or after the specified time. Use this for point-in-time disaster recovery scenarios. > ❗ **IMPORTANT** > > The starting offset only affects **new shadow topics**. After a shadow topic exists and has data, changing this setting has no effect on that topic’s replication. #### [](#networking)Networking Configure network connectivity between your source and shadow clusters to enable shadow link replication. The shadow cluster initiates connections to the source cluster using a pull-based architecture. For additional details about networking, see [Network and authentication](#network-and-authentication). ##### [](#connection-requirements)Connection requirements - **Direction**: Shadow cluster connects to source cluster (outbound from shadow, inbound to source) - **Protocol**: Kafka protocol over TCP (default port 9092, or your configured listener ports) - **Persistence**: Connections remain active for continuous replication ##### [](#firewall-configuration)Firewall configuration You must configure firewall rules to allow the shadow cluster to reach the source cluster. **On the source cluster network:** - Allow inbound TCP connections on Kafka listener ports (typically 9092). - Allow connections from the shadow cluster’s IP addresses or subnets. **On the shadow cluster network:** - Allow outbound TCP connections to the source cluster’s Kafka listener ports. - Ensure DNS resolution works for source cluster hostnames. ##### [](#bootstrap-servers)Bootstrap servers Specify multiple bootstrap servers in your shadow link configuration for high availability: ```yaml client_options: bootstrap_servers: # Source cluster brokers to connect to - : # Example: "prod-kafka-1.example.com:9092" - : # Example: "prod-kafka-2.example.com:9092" - : # Example: "prod-kafka-3.example.com:9092" ``` The shadow cluster uses these addresses to discover all brokers in the source cluster. If one bootstrap server is unavailable, the shadow cluster tries the next one in the list. ##### [](#network-security)Network security For production deployments, secure the network connection between clusters: TLS encryption: ```yaml client_options: tls_settings: enabled: true # Enable TLS tls_pem_settings: ca: |- # CA certificate in PEM format -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- key: ${secrets.} # Client private key (can use secrets reference) cert: |- # Optional: Client certificate in PEM format for mutual TLS -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- do_not_set_sni_hostname: false # Optional: Skip SNI hostname when using TLS (default: false) ``` Authentication: ```yaml client_options: authentication_configuration: # SASL/SCRAM authentication. # Create SASL credentials in the source cluster. # Then, with this configuration, ensure the shadow cluster uses the credentials # to authenticate to the source cluster. scram_configuration: username: # SASL/SCRAM username, example: "shadow-replication-user" password: ${secrets.} # ID of secret containing SASL/SCRAM password scram_mechanism: SCRAM_SHA_256 # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512" ``` ##### [](#connection-tuning)Connection tuning Adjust connection parameters based on your network characteristics. For example: ```yaml client_options: # Connection and metadata settings connection_timeout_ms: 1000 # Default 1000ms, increase for high-latency networks retry_backoff_ms: 100 # Default 100ms, backoff between connection retries metadata_max_age_ms: 10000 # Default 10000ms, how often to refresh cluster metadata # Fetch request settings fetch_wait_max_ms: 500 # Default 500ms, max time to wait for fetch requests fetch_min_bytes: 5242880 # Default 5MB, minimum bytes to fetch per request fetch_max_bytes: 20971520 # Default 20MB, maximum bytes to fetch per request fetch_partition_max_bytes: 5242880 # Default 5MB, maximum bytes to fetch per partition ``` ## [](#update-an-existing-shadow-link)Update an existing shadow link To modify a shadow link configuration after creation, run: ### Cloud UI 1. At the organization level of the Cloud UI, navigate to **Shadow Link**. 2. Select the shadow link you want to modify, and click **Edit**. 3. Edit the shadow link settings or the shadowing behavior by specifying which content from the source cluster to shadow (topics, ACLs, consumer groups, Schema Registry). You can also enable additional topic properties to be shadowed or disable optional topic properties from being included in the shadowing. 4. Click **Save** to apply changes. ### rpk ```bash rpk shadow update ``` For detailed command options, see [`rpk shadow update`](https://docs.redpanda.com/cloud-data-platform/reference/rpk/rpk-shadow/rpk-shadow-update/). This opens your default editor to modify the shadow link configuration. Only changed fields are updated on the server. The shadow link name cannot be changed - you must delete and recreate the link to rename it. ### Control Plane API ```bash curl -X PATCH 'https://api.redpanda.com/v1/shadow-links/' \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d '{ "security_sync_options": { "paused": false } }' ``` This endpoint returns a [long-running operation](https://docs.redpanda.com/cloud-data-platform/manage/api/cloud-byoc-controlplane-api/#lro). For the full API reference, see [Control Plane API reference](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-shadowlinkservice_updateshadowlink). ### Terraform Edit the `redpanda_shadow_link` resource configuration and run `terraform apply`. The provider sends only the changed fields to the API. The shadow link name and cluster IDs are immutable: if you change them, Terraform plans a replacement (destroy and recreate) instead of an update. The destroy step of a replacement succeeds only when the resource sets `allow_deletion = true`, the same guard described in the create step. --- # Page 603: Integrate Redpanda with Iceberg **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg.md --- # Integrate Redpanda with Iceberg > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Integrate Redpanda with Iceberg latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/index.adoc description: Generate Iceberg tables for your Redpanda topics for data lakehouse access. page-git-created-date: "2025-04-04" page-git-modified-date: "2025-07-30" --- - [About Iceberg Topics](about-iceberg-topics/) Learn how Redpanda can integrate topics with Apache Iceberg. - [Migrate to Iceberg Topics](migrate-to-iceberg-topics/) Migrate existing Iceberg integrations to Redpanda Iceberg topics. - [Specify Iceberg Schema](specify-iceberg-schema/) Learn about supported Iceberg modes and how you can integrate schemas with Iceberg topics. - [Use Iceberg Catalogs](use-iceberg-catalogs/) Learn how to access Redpanda topic data stored in Iceberg tables, using table metadata or a catalog integration. - [Integrate with REST Catalogs](rest-catalog/) Integrate Redpanda topics with managed Iceberg REST Catalogs. - [Query Iceberg Topics](query-iceberg-topics/) Query Redpanda topic data stored in Iceberg tables, based on the topic Iceberg mode and schema. - [Migrate Iceberg Catalogs](migrate-iceberg-catalog/) Switch the Iceberg catalog backend for an existing Redpanda cluster without losing untranslated topic data. - [Tune Performance for Iceberg Topics](iceberg-performance-tuning/) Optimize query performance and translation throughput for Iceberg topics with partitioning, compaction, lag target tuning, and cluster sizing guidance. - [Troubleshoot Iceberg Topics](iceberg-troubleshooting/) Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. --- # Page 604: About Iceberg Topics **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics.md --- # About Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: About Iceberg Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/about-iceberg-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/about-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/about-iceberg-topics.adoc description: Learn how Redpanda can integrate topics with Apache Iceberg. page-git-created-date: "2025-04-04" page-git-modified-date: "2026-05-26" --- The Apache Iceberg integration for Redpanda allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems, including data warehouses like Snowflake, Databricks, ClickHouse, and Redshift, without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, such as Apache Spark and Flink, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines. Redpanda supports [version 2](https://iceberg.apache.org/spec/#format-versioning) of the Iceberg table format. ## [](#iceberg-concepts)Iceberg concepts [Apache Iceberg](https://iceberg.apache.org) is an open source format specification for defining structured tables in a data lake. The table format lets you quickly and easily manage, query, and process huge amounts of structured and unstructured data. This is similar to the way you would manage and run SQL queries against relational data in a database or data warehouse. The open format lets you use many different languages, tools, and applications to process the same data in a consistent way, so you can avoid vendor lock-in. This data management system is also known as a _data lakehouse_. In the Iceberg specification, tables consist of the following layers: - **Data layer**: Stores the data in data files. The Iceberg integration currently supports the Parquet file format. Parquet files are column-based and suitable for analytical workloads at scale. They come with compression capabilities that optimize files for object storage. - **Metadata layer**: Stores table metadata separately from data files. The metadata layer allows multiple writers to stage metadata changes and apply updates atomically. It also supports database snapshots, and time travel queries that query the database at a previous point in time. - Manifest files: Track data files and contain metadata about these files, such as record count, partition membership, and file paths. - Manifest list: Tracks all the manifest files belonging to a table, including file paths and upper and lower bounds for partition fields. - Metadata file: Stores metadata about the table, including its schema, partition information, and snapshots. Whenever a change is made to the table, a new metadata file is created and becomes the latest version of the metadata in the catalog. For Iceberg-enabled topics, the manifest files are in JSON format. - **Catalog**: Contains the current metadata pointer for the table. Clients reading and writing data to the table see the same version of the current state of the table. The Iceberg integration supports two [catalog integration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) types. You can configure Redpanda to catalog files stored in the same object storage bucket or container where the Iceberg data files are located, or you can configure Redpanda to use an [Iceberg REST catalog](https://iceberg.apache.org/terms/#decoupling-using-the-rest-catalog) endpoint to update an externally-managed catalog when there are changes to the Iceberg data and metadata. ![Redpanda’s Iceberg integration](https://docs.redpanda.com/cloud-data-platform/shared/_images/iceberg-integration-optimized.png) When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers store streaming data in the Iceberg-compatible format in Parquet files in object storage, in addition to the log segments uploaded using Tiered Storage. Storing the streaming data in Iceberg tables in the cloud allows you to derive real-time insights through many compatible data lakehouse, data engineering, and business intelligence [tools](https://iceberg.apache.org/vendors/). ## [](#prerequisites)Prerequisites To enable Iceberg for Redpanda topics, you must have the following: - A running [BYOC](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/) or BYOVPC cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC and BYOVPC, and the cluster properties to configure Iceberg are available with v25.1. - rpk: See [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). - Familiarity with the Redpanda Cloud API. You must [authenticate](https://docs.redpanda.com/api/doc/cloud-controlplane/authentication) to the Cloud API and use the Control Plane API to update your cluster configuration. ## [](#limitations)Limitations - It is not possible to append topic data to an existing Iceberg table that is not created by Redpanda. - If you enable the Iceberg integration on an existing Redpanda topic, Redpanda does not backfill the generated Iceberg table with topic data. - JSON schemas are supported starting with Redpanda version 25.2. ## [](#enable-iceberg-integration)Enable Iceberg integration To create an Iceberg table for a Redpanda topic, you must set the cluster configuration property `[iceberg_enabled](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_enabled)` to `true`, and also configure the topic property `redpanda.iceberg.mode`. You can choose to provide a schema if you need the Iceberg table to be structured with defined columns. 1. Set the `iceberg_enabled` configuration option on your cluster to `true`. When multiple clusters write to the same catalog, each cluster must use a distinct namespace to avoid table name collisions. This is especially critical for REST catalog providers that offer a single global catalog per account (such as AWS Glue), where there is no other isolation mechanism. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace for your cluster’s REST catalog integration, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. You cannot change this property after you enable Iceberg topics on the cluster. #### rpk ```bash rpk cloud login rpk profile create --from-cloud rpk cluster config set iceberg_enabled true # Optional: set a custom namespace (default is "redpanda") # rpk cluster config set iceberg_default_catalog_namespace '[""]' ``` #### Cloud API ```bash # Store your cluster ID in a variable export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret="` # Update cluster configuration to enable Iceberg topics # Optional: to set a custom namespace (default is "redpanda"), # add "iceberg_default_catalog_namespace":[""] to custom_properties curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json'\ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties": {"iceberg_enabled":true}}}' ``` The [`PATCH /clusters/{cluster.id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) request returns the ID of a long-running operation. The operation may take up to ten minutes to complete. You can check the status of the operation by polling the [`GET /operations/{id}`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation) endpoint. 2. (Optional) Create a new topic. ```bash rpk topic create ``` ```bash TOPIC STATUS OK ``` 3. Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following [Iceberg modes](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/): - `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. - `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for this topic, with columns corresponding to each field. You must register a schema in the Schema Registry (see next step), and producers must write to the topic using the Schema Registry wire format. - `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. - `disabled` (default): Disables writing to an Iceberg table for this topic. ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` ```bash TOPIC STATUS OK ``` 4. Register a schema for the topic. This step is required for the `value_schema_id_prefix` and `value_schema_latest` modes. ```bash rpk registry schema create --schema --type ``` ```bash SUBJECT VERSION ID TYPE 1 1 PROTOBUF ``` ### [](#access-iceberg-data)Access Iceberg data To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows: | Cloud provider | Bucket or container name | Iceberg table location | | --- | --- | --- | | AWS | redpanda-cloud-storage- | redpanda-iceberg-catalog/redpanda/ | | Azure | The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. | | GCP | redpanda-cloud-storage- | For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster’s allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. For example, to add subnet IDs to the allow list through the Control Plane API [`PATCH /v1/clusters/`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, run: ```bash curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d @- << EOF { "cloud_storage": { "azure": { "allowed_subnet_ids": [ ] } } } EOF ``` As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to [read the Iceberg topic data](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) in a data lake as you would for a relational database. See also: [Schema types translation](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#schema-types-translation). ### [](#iceberg-data-retention)Iceberg data retention Data in an Iceberg-enabled topic is consumable from Kafka based on the configured [topic retention policy](https://docs.redpanda.com/cloud-data-platform/develop/topics/create-topic/). Conversely, data written to Iceberg remains queryable as Iceberg tables indefinitely. The Iceberg table persists unless you: - Delete the Redpanda topic associated with the Iceberg table. This is the default behavior set by the `[iceberg_delete](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_delete)` cluster property and the `redpanda.iceberg.delete` topic property. If you set this property to `false`, the Iceberg table remains even after you delete the topic. - Explicitly delete data from the Iceberg table using a query engine. - Disable the Iceberg integration for the topic and delete the Parquet files in object storage. The DLQ table (`~dlq`) follows the same persistence rules as the main Iceberg table. ## [](#schema-evolution)Schema evolution Redpanda supports schema evolution in accordance with the [Iceberg specification](https://iceberg.apache.org/spec/#schema-evolution). Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema. For example, if you produce records to a topic `demo-topic` with the following Avro schema: schema\_1.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" } ] } ``` ```bash rpk registry schema create demo-topic-value --schema schema_1.avsc echo '{"user_id":23, "event_type":"BUTTON_CLICK"}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Then, you update the schema to add a new field `ts`, and produce records with the updated schema: schema\_2.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" }, { "name": "ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-millis" } ], "default": null # Default value for the new field } ] } ``` The `ts` field can be either null or a long representing epoch milliseconds. The default value is null. ```bash rpk registry schema create demo-topic-value --schema schema_2.avsc echo '{"user_id":858, "event_type":"BUTTON_CLICK", "ts":1737998723230}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Querying the Iceberg table for `demo-topic` includes the new column `ts`: ```bash +---------+--------------+--------------------------+ | user_id | event_type | ts | +---------+--------------+--------------------------+ | 858 | BUTTON_CLICK | 2025-02-26T20:05:23.230Z | | 23 | BUTTON_CLICK | NULL | +---------+--------------+--------------------------+ ``` ## [](#next-steps)Next steps - [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) - [Tune Performance for Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-performance-tuning/) - [Troubleshoot Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/) ## [](#suggested-reading)Suggested reading - [Understanding Apache Kafka Schema Registry](https://www.redpanda.com/blog/schema-registry-kafka-streaming#how-does-serialization-work-with-schema-registry-in-kafka) --- # Page 605: Tune Performance for Iceberg Topics **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-performance-tuning.md --- # Tune Performance for Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Tune Performance for Iceberg Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/iceberg-performance-tuning page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/iceberg-performance-tuning.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/iceberg-performance-tuning.adoc description: Optimize query performance and translation throughput for Iceberg topics with partitioning, compaction, lag target tuning, and cluster sizing guidance. page-git-created-date: "2026-05-06" page-git-modified-date: "2026-05-26" --- This guide covers strategies for optimizing the performance of Iceberg topics in Redpanda, including improving downstream query performance, tuning the Iceberg translation pipeline, and monitoring translation throughput. After reading this page, you will be able to: - Apply partitioning and compaction strategies to improve query performance - Choose appropriate lag target values for your workload - Identify translation performance signals using Iceberg metrics ## [](#prerequisites)Prerequisites You must be familiar with how Iceberg topics work in Redpanda. See [About Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/). ## [](#optimize-query-performance)Optimize query performance Query engines read Parquet files from object storage to process Iceberg table data. Partitioning, compaction, and schema design affect how efficiently those reads perform. ### [](#use-custom-partitioning)Use custom partitioning To improve query performance, consider implementing custom [partitioning](https://iceberg.apache.org/docs/nightly/partitioning/) for the Iceberg topic. Use the `redpanda.iceberg.partition.spec` topic property to define the partitioning scheme: ```bash # Create new topic with five topic partitions, replication factor 3, and custom table partitioning for Iceberg rpk topic create -p5 -r3 -c redpanda.iceberg.mode=value_schema_id_prefix -c "redpanda.iceberg.partition.spec=(, , ...)" ``` Valid `` values include a source column name or a transformation of a column. The columns referenced can be Redpanda-defined (such as `redpanda.timestamp`) or user-defined based on a schema that you register for the topic. The Iceberg table stores records that share different partition key values in separate files based on this specification. For example: - To partition the table by a single key, such as a column `col1`, use: `redpanda.iceberg.partition.spec=(col1)`. - To partition by multiple columns, use a comma-separated list: `redpanda.iceberg.partition.spec=(col1, col2)`. - To partition by the year of a timestamp column `ts1`, and a string column `col1`, use: `redpanda.iceberg.partition.spec=(year(ts1), col1)`. To learn more about how partitioning schemes can affect query performance, and for details on the partitioning specification such as allowed transforms, see the [Apache Iceberg documentation](https://iceberg.apache.org/spec/#partitioning). > 💡 **TIP** > > - Partition by columns that you frequently use in queries. Columns with relatively few unique values (low cardinality) are good candidates for partitioning. > > - If you must partition based on columns with high cardinality, for example timestamps, use Iceberg’s available transforms such as extracting the year, month, or day to avoid creating too many partitions. Too many partitions can be detrimental to performance because more files need to be scanned and managed. ### [](#compact-iceberg-tables)Compact Iceberg tables Over time, Iceberg translation can produce many small Parquet files, especially with low-throughput topics or short lag targets. Compaction merges small files into larger ones, reducing the number of metadata operations query engines must perform and improving read performance. - Automatic compaction: Some catalog and data platform services, such as AWS Glue and Databricks, automatically compact Iceberg tables. - Manual or scheduled compaction: Tools like [Apache Spark](https://spark.apache.org/) can run compaction jobs on a schedule. This is useful if your catalog or platform does not compact automatically. If you observe degraded read performance or a high number of small files, investigate whether your catalog or platform supports automatic compaction or schedule periodic compaction jobs. ### [](#avoid-high-column-count)Avoid high column count A high column count or schema field count results in more overhead when translating topics to the Iceberg table format. Small message sizes can also increase CPU utilization. To minimize the performance impact on your cluster, keep to a low column count and large message size for Iceberg topics. ## [](#tune-translation-performance)Tune translation performance Translation is the process in which Redpanda converts topic data into Parquet files for the Iceberg table. Each round of translation processes one topic partition at a time. Under typical conditions, Iceberg translation has the following performance characteristics: - Throughput: Approximately 5 MiB/s per core. - Flush threshold: 32 MiB. Each translation process uploads its on-disk data when accumulated data reaches this threshold. This is the primary control for Parquet file size, and is managed by Redpanda Cloud. - Lag target: Controlled by [`iceberg_target_lag_ms`](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_target_lag_ms) (default: 1 minute). Redpanda tries to commit all data produced to an Iceberg-enabled topic within this window. The flush threshold and lag target together determine the size of the Parquet files written to object storage. Larger Parquet files generally improve downstream query performance by reducing the number of metadata operations query engines must perform. ### [](#tune-the-lag-target)Tune the lag target In Redpanda Cloud, `datalake_translator_flush_bytes` is managed by Redpanda Cloud and is not user-tunable. To adjust the size of Parquet files written to object storage, increase the lag target. A larger lag target gives translators more time to accumulate data before committing, resulting in larger Parquet files with more records per file. You can configure the lag target at the cluster level or per-topic: - Cluster-wide: edit `iceberg_target_lag_ms` in the Redpanda Cloud Console. For instructions, see [Configure Cluster Properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/). - Per-topic: set the `redpanda.iceberg.target.lag.ms` topic property. The topic property overrides the cluster default for that topic. > 📝 **NOTE** > > Increasing the lag target means Iceberg tables receive new data less frequently. Choose a lag value that balances file efficiency against how current your downstream data must be. To check the current cluster-wide value: ```bash rpk cluster config get iceberg_target_lag_ms ``` To check topic-level overrides: ```bash rpk topic describe -c ``` ### [](#optimize-message-size)Optimize message size Redpanda has validated 32 MiB as the maximum recommended message size for Iceberg-enabled topics. With large messages, each Parquet file contains fewer records because the flush threshold is reached sooner. This can reduce the efficiency of analytical queries that need to scan many records. If query latency is a concern and your workload produces large messages, consider: - Reducing individual message sizes if your data model allows it. - Increasing `iceberg_target_lag_ms` to produce Parquet files with more records per file. See [Tune the lag target](#tune-the-lag-target). ### [](#size-clusters-for-iceberg-workloads)Size clusters for Iceberg workloads When you enable Iceberg for any substantial workload and start translating topic data to the Iceberg format, you may see most of your cluster’s CPU utilization increase. If this additional workload overwhelms the brokers and causes the Iceberg table lag to exceed the configured target lag, Redpanda automatically increases the scheduling priority of Iceberg translation to help it catch up with incoming data. However, this does not substitute for adequate cluster resources. You may need to increase the size of your Redpanda cluster to accommodate the additional workload. To ensure that your cluster is sized appropriately, contact the Redpanda Customer Success team. ### [](#monitor-translation-performance)Monitor translation performance Use the following [Iceberg metrics](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#iceberg-metrics) to understand whether translation is keeping pace with incoming data: - [`redpanda_iceberg_translation_raw_bytes_processed`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_raw_bytes_processed): Total raw bytes consumed for translation input. Use this to monitor input throughput and compare against the expected 5 MiB/s per core baseline. - [`redpanda_iceberg_translation_parquet_bytes_added`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_parquet_bytes_added): Total bytes written to Parquet files. Divide by `redpanda_iceberg_translation_files_created` to estimate the average file size produced by your workload. - [`redpanda_iceberg_translation_files_created`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_files_created): Number of Parquet files created. A high file creation rate relative to bytes added indicates many small files. Consider increasing `iceberg_target_lag_ms`. - [`redpanda_iceberg_translation_parquet_rows_added`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_parquet_rows_added): Total rows written to Parquet files. Useful for understanding record-level throughput. - [`redpanda_iceberg_translation_translations_finished`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_translations_finished): Number of completed translator executions. A stalling or zero rate indicates translation has stopped. For metrics related to DLQ files, invalid records, and catalog commit failures, see [Troubleshooting metrics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/#troubleshooting-metrics). > 💡 **TIP** > > If translation consistently lags despite available CPU headroom, the workload may be partition-bound. Each core translates its assigned partitions independently, so distributing data across more partitions allows more cores to contribute to translation and can improve total throughput. --- # Page 606: Query Iceberg Topics using AWS Glue **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-aws-glue.md --- # Query Iceberg Topics using AWS Glue > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Query Iceberg Topics using AWS Glue page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/iceberg-topics-aws-glue page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/iceberg-topics-aws-glue.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc description: Add Redpanda topics as Iceberg tables that you can access through the AWS Glue Data Catalog. # Beta release status page-beta: "true" page-git-created-date: "2025-08-05" page-git-modified-date: "2026-05-26" release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- This guide walks you through querying Redpanda topics as Iceberg tables stored in AWS S3, using a catalog integration with [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro). For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/). ## [](#prerequisites)Prerequisites - An AWS account with access to [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html). - AWS Glue Data Catalog must be in the same AWS account and region as the cluster. - Redpanda version 25.2 or later. - [`rpk`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) installed or updated to the latest version. - You can also use the Redpanda Cloud API to [reference secrets in your cluster configuration](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties). - Admin permissions to create IAM policies and roles in AWS. ## [](#limitations)Limitations ### [](#lowercase-field-names-required)Lowercase field names required Use only lowercase field names. AWS Glue converts all table column names to lowercase, and Redpanda requires exact column name matches to manage schemas. Using uppercase letters prevents Redpanda from finding matching columns, which breaks schema management. ### [](#nested-partition-spec-support)Nested partition spec support AWS Glue does not support partitioning on nested fields. If Redpanda detects that the default partitioning `(hour(redpanda.timestamp))` based on the record metadata is in use, it will instead apply an empty partition spec `()`, which means the table will not be partitioned. To use partitioning, you must implement custom partitioning using your own partition columns (that is, columns that are not nested). > 📝 **NOTE** > > In Redpanda versions 25.2.1 and earlier, an empty partition spec `()` can cause a known issue that prevents certain engines like Amazon Redshift from successfully querying the table. To resolve this issue, specify custom partitioning, or upgrade Redpanda to versions 25.2.2 or later. ### [](#manual-deletion-of-iceberg-tables)Manual deletion of Iceberg tables The AWS Glue catalog integration does not support automatic deletion of Iceberg tables from Redpanda. To manually delete Iceberg tables in AWS Glue, you must either: - Set the cluster property `[iceberg_delete](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_delete)` to `false` when you configure the catalog integration. - Override the cluster property `iceberg_delete` by setting the topic property `redpanda.iceberg.delete` to `false` for the topic you want to delete. When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic. ## [](#authorize-access-to-aws-glue)Authorize access to AWS Glue For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the cluster’s IAM role when Iceberg is enabled. You don’t need to manually create IAM policies or roles for Glue access. For clusters created before March 2026, you must re-run `rpk cloud byoc aws apply --redpanda-id=` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster’s IAM role with the necessary Glue permissions. ## [](#configure-authentication-and-credentials)Configure authentication and credentials You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: - Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach, especially in BYOC deployments where the cluster’s existing AWS credentials already include the necessary Glue permissions. For an example cluster configuration that uses the same IAM credentials for both S3 and AWS Glue, see the **Use cluster’s IAM credentials** tab in the [next section](#update-cluster-configuration). - If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: - `[iceberg_rest_catalog_credentials_source](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_rest_catalog_credentials_source)` - Set the property to `sts` if you want to use the cluster’s default IAM role. - Set to `config_file` if you want to scope Glue access through your own IAM user and policy instead of the cluster’s default IAM role, or if you want to use static credentials. - `[iceberg_rest_catalog_aws_access_key](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_access_key)` (static credentials only) - `[iceberg_rest_catalog_aws_secret_key](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_secret_key)` (static credentials only), added as a secret value (see the [next section](#update-cluster-configuration) for details) - `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` For an example cluster configuration that uses separate access keys for AWS Glue, see the **Use static credentials (override IAM)** tab in the [next section](#update-cluster-configuration). ## [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with the AWS Glue Data Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. Because AWS Glue provides a single catalog per account, each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. To set a unique namespace, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. This property cannot be changed after Iceberg is enabled. Use `rpk` as shown in the following examples, or [use the Cloud API](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties) to update these cluster properties. The update might take several minutes to complete. ### Use cluster’s IAM credentials ```bash # Glue requires Redpanda Iceberg tables to be manually deleted # so iceberg_delete is set to false. rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ iceberg_delete=false \ iceberg_default_catalog_namespace='[""]' \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ iceberg_rest_catalog_credentials_source=sts \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_base_location=s3:/// ``` ### Use static credentials (override IAM) ```bash # Glue requires Redpanda Iceberg tables to be manually deleted # so iceberg_delete is set to false. rpk cluster config set \ iceberg_enabled=true \ iceberg_delete=false \ iceberg_default_catalog_namespace='[""]' \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ iceberg_rest_catalog_credentials_source=config_file \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_aws_access_key= \ iceberg_rest_catalog_aws_secret_key='${secrets.}' \ iceberg_rest_catalog_base_location=s3:/// ``` Use your own values for the following placeholders: - ``: A unique namespace for this cluster’s Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used. - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` property. - `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. - Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the object storage bucket you created as a [customer-managed resource](https://docs.redpanda.com/cloud-data-platform/get-started/cluster-types/byoc/aws/vpc-byo-aws/#configure-the-redpanda-network-and-cluster). This must be the same bucket used for your cluster’s object storage. You cannot specify a different bucket for Iceberg data. - Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster. As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with the rest of your cluster’s data in object storage. - `` (static credentials only): The AWS access key ID for your Glue service account. - `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first [store the secret value](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#store-a-secret-for-rest-catalog-authentication). ```bash Successfully updated configuration. New configuration version is 2. ``` 2. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 3. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` You should see the topic as a table with data in AWS Glue Data Catalog. The data may take some time to become visible, depending on your `[iceberg_target_lag_ms](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_target_lag_ms)` setting. 1. In AWS Glue Studio, go to Databases. 2. Select the `redpanda` database. The `redpanda` database and the table within are automatically added for you. The table name is the same as the topic name. ## [](#query-iceberg-table)Query Iceberg table You can query the Iceberg table using different engines, such as Amazon Athena, PyIceberg, or Apache Spark. To query the table or view the table data in AWS Glue, ensure that your account has the necessary permissions to access the catalog, database, and table. To query the table in Amazon Athena: 1. On the list of tables in AWS Glue Studio, click "Table data" under the **View data** column. 2. Click "Proceed" to be redirected to the Athena query editor. 3. In the query editor, select AwsDataCatalog as the data source, and select the `redpanda` database. 4. The SQL query editor should be pre-populated with a query that selects 10 rows from the Iceberg table. Run the query to see a preview of the table data. ```sql SELECT * FROM "AwsDataCatalog"."redpanda"."" limit 10; ``` Your query results should look like the following: ```sql +-----------------------------------------------------+----------------+ | redpanda | value | +-----------------------------------------------------+----------------+ | {partition=0, offset=0, timestamp=2025-07-21 | 77 6f 72 6c 64 | | 18:11:25.070000, headers=null, key=[B@1900af31} | | +-----------------------------------------------------+----------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and the AWS Glue Data Catalog. To grant your end users and query engines (such as Amazon Athena or Apache Spark) read access to the Iceberg tables, use [AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) to assign table-level and column-level permissions. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) --- # Page 607: Query Iceberg Topics using Databricks and Unity Catalog **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-databricks-unity.md --- # Query Iceberg Topics using Databricks and Unity Catalog > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Query Iceberg Topics using Databricks and Unity Catalog latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/iceberg-topics-databricks-unity page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/iceberg-topics-databricks-unity.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc description: Add Redpanda topics as Iceberg tables that you can query in Databricks managed by Unity Catalog. page-git-created-date: "2025-06-12" page-git-modified-date: "2026-05-26" --- This guide walks you through querying Redpanda topics as managed Iceberg tables in Databricks, with AWS S3 as object storage and a catalog integration using [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog). For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/). After reading this page, you will be able to: - Configure a Unity Catalog integration for Redpanda Iceberg topics with AWS S3 - Query Redpanda topic data as Iceberg tables in Databricks SQL ## [](#prerequisites)Prerequisites - A Databricks workspace in the same region as your S3 bucket. See the [list of supported AWS regions](https://docs.databricks.com/aws/en/resources/supported-regions#supported-regions-list). - Unity Catalog enabled in your Databricks workspace. See the [Databricks documentation](https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started) to set up Unity Catalog for your workspace. - [Predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-predictive-optimization) enabled for Unity Catalog. > 📝 **NOTE** > > When you enable predictive optimization, you must also set the following configurations in your Databricks workspace. These configurations allow predictive optimization to automatically generate column statistics and carry out background compaction for Iceberg tables: > > ```sql > SET spark.databricks.delta.liquid.lazyClustering.backfillStats=true; > SET spark.databricks.delta.computeStats.autoConflictResolution=true; > > /* > After setting these configurations, you can optionally run OPTIMIZE to > immediately trigger compaction and liquid clustering, or let predictive > optimization handle it automatically later. > */ > OPTIMIZE ``.redpanda.``; > ``` - [External data access](https://docs.databricks.com/aws/en/external-access/admin) enabled in your metastore. - Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster’s object storage bucket to Databricks. ## [](#limitations)Limitations The following data types are not currently supported for managed Iceberg tables: | Iceberg type | Equivalent Avro type | | --- | --- | | uuid | uuid | | fixed(L) | fixed | | time | time-millis, time-micros | There are no limitations for Protobuf types. ## [](#create-a-unity-catalog-storage-credential)Create a Unity Catalog storage credential A storage credential is a Databricks object that controls access to external object storage, in this case S3. You associate a storage credential with an AWS IAM role that defines what actions Unity Catalog can perform in the S3 bucket. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials) to create an AWS IAM role that has the required permissions for the bucket. When you have completed these steps, you should have the following configured in AWS and Databricks: - A self-assuming IAM role, meaning you’ve defined the role trust policy so the role trusts itself. - Two IAM policies attached to the IAM role. The first policy grants Unity Catalog read and write access to the bucket. The second policy allows Unity Catalog to configure file events. - A storage credential in Databricks associated with the IAM role, using the role’s ARN. You also use the storage credential’s external ID in the role’s trust relationship policy to make the role self-assuming. ## [](#create-a-unity-catalog-external-location)Create a Unity Catalog external location The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for object storage for your Redpanda cluster. For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations) to **manually** create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing object storage bucket URL, `s3://`. ## [](#choose-a-catalog-setup)Choose a catalog setup You can either create a new catalog dedicated to Redpanda topics or use an existing catalog. If you create a new catalog, Redpanda automatically creates the required schema for you. If you need to integrate with an existing catalog, you must manually create the schema in that catalog before Redpanda creates any Iceberg tables. After you set up your catalog, the authorization and Redpanda configuration steps are the same for both options. ### [](#option-1-create-a-new-catalog-recommended)Option 1: Create a new catalog (recommended) Follow the steps in the Databricks documentation to [create a standard catalog](https://docs.databricks.com/aws/en/catalogs/create-catalog). When you create the catalog, specify the external location you created in the previous step as the storage location. In this setup, Redpanda creates the default `redpanda` schema for you. You use the catalog name when you set the Iceberg cluster configuration properties in Redpanda in a later step. ### [](#option-2-use-an-existing-catalog-with-a-pre-created-schema)Option 2: Use an existing catalog with a pre-created schema If you need to integrate Redpanda with an existing Unity Catalog catalog object, follow the steps to [create a schema](https://docs.databricks.com/aws/en/schemas/create-schema) in the catalog. - By default, Redpanda creates tables in a schema named `redpanda`. If you want to use a different schema, set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` before enabling Iceberg, then manually create that schema in the catalog. - Set the schema’s managed storage location to the same S3 bucket used for object storage, using the external location you created in the previous step. Unity Catalog resolves managed storage locations through a hierarchy of metastore > catalog > schema. If you assign the schema its own managed storage location, Redpanda can use the existing catalog while the schema stores its managed Iceberg data in the schema-specific location. For example: - Your existing Unity Catalog catalog stores managed data in `s3://`. - You manually create a `redpanda` schema in that catalog and override its managed storage location, through the external location, to the S3 bucket that Redpanda uses for your cluster’s object storage (`s3://redpanda-cloud-storage-` for BYOC, or your customer-managed bucket for BYOVPC). For more information, see the [Unity Catalog managed storage location hierarchy](https://docs.databricks.com/aws/en/data-governance/unity-catalog/#managed-storage-location-hierarchy) in the Databricks documentation. ## [](#authorize-access-to-unity-catalog)Authorize access to Unity Catalog Redpanda recommends using OAuth for service principals to grant Redpanda access to Unity Catalog. 1. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) to create a service principal, and then generate an OAuth secret. You use the client ID and secret to set Iceberg cluster configuration properties in Redpanda in the next step. 2. Open your catalog in the Catalog Explorer, then click **Permissions**. 3. Click **Grant** to grant the service principal the following permissions on the catalog: - `ALL PRIVILEGES` - `EXTERNAL USE SCHEMA` The Iceberg integration for Redpanda also supports using bearer tokens. ## [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Unity Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Use `rpk` like in the following example, or use the Cloud API to [update these cluster properties](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties). The update might take several minutes to complete. To reference a secret in a cluster property, you must first [store the secret value](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#store-a-secret-for-rest-catalog-authentication). ```bash rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https:///api/2.1/unity-catalog/iceberg-rest \ iceberg_rest_catalog_authentication_mode=oauth2 \ iceberg_rest_catalog_oauth2_server_uri=https:///oidc/v1/token \ iceberg_rest_catalog_oauth2_scope=all-apis \ iceberg_rest_catalog_client_id= \ iceberg_rest_catalog_client_secret='${secrets.}' \ iceberg_rest_catalog_warehouse= \ iceberg_disable_snapshot_tagging=true # Optional. Set a custom namespace only if you want to use a schema other than the default `redpanda` # iceberg_default_catalog_namespace='[""]' ``` Use your own values for the following placeholders: - ``: The URL of your [Databricks workspace instance](https://docs.databricks.com/aws/en/workspace/workspace-details#workspace-instance-names-urls-and-ids); for example, `cust-success.cloud.databricks.com`. - ``: The client ID of the service principal you created in an earlier step. - ``: The name of the client secret of the service principal you created in an earlier step. - ``: The name of your catalog in Unity Catalog. ```bash Successfully updated configuration. New configuration version is 2. ``` 2. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 3. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` You should see the topic as a table with data in Unity Catalog. The data may take some time to become visible, depending on your `[iceberg_target_lag_ms](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_target_lag_ms)` setting. 1. In Catalog Explorer, open your catalog. You should see a `redpanda` schema (or the namespace you configured with `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`), in addition to `default` and `information_schema`. 2. The schema and the table residing within it are automatically added for you. The table name is the same as the topic name. ## [](#query-iceberg-table-using-databricks-sql)Query Iceberg table using Databricks SQL You can query the Iceberg table using different engines, such as Databricks SQL, PyIceberg, or Apache Spark. To query the table or view the table data in Catalog Explorer, ensure that your account has the necessary permissions to read the table. The following example shows how to query the Iceberg table using SQL in Databricks SQL. 1. In the Databricks console, open **SQL Editor**. 2. In the query editor, run: ```sql /* Ensure that the catalog and table name are correctly parsed in case they contain special characters. If you set iceberg_default_catalog_namespace to a custom namespace, replace `redpanda` with that namespace in the query below. */ SELECT * FROM ``.redpanda.`` LIMIT 10; ``` Your query results should look like the following: ```sql -- Example for redpanda.iceberg.mode=key_value with 1 record produced to topic +----------------------------------------------------------------------+------------+ | redpanda | value | +----------------------------------------------------------------------+------------+ | {"partition":0,"offset":"0","timestamp":"2025-04-02T18:57:11.127Z", | 776f726c64 | | "headers":null,"key":"68656c6c6f"} | | +----------------------------------------------------------------------+------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and Unity Catalog. To grant your end users or query engines read access to the Iceberg tables, use Unity Catalog to assign the appropriate privileges. Review the Databricks documentation on [granting permissions to objects](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/?language=SQL#grant-permissions-on-objects-in-a-unity-catalog-metastore) and [Unity Catalog privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges) for details. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) --- # Page 608: Use Iceberg Topics with GCP Lakehouse **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-gcp-biglake.md --- # Use Iceberg Topics with GCP Lakehouse > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Use Iceberg Topics with GCP Lakehouse latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/iceberg-topics-gcp-biglake page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/iceberg-topics-gcp-biglake.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc description: Add Redpanda topics as Iceberg tables to Google Lakehouse for Apache Iceberg that you can query from Google BigQuery. page-git-created-date: "2026-06-11" page-git-modified-date: "2026-06-11" --- > 💡 **TIP** > > This guide is for integrating Iceberg topics with a managed REST catalog. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. This guide walks you through querying Redpanda topics as Iceberg tables stored in Google Cloud Storage, using a REST catalog integration with [Google Lakehouse for Apache Iceberg](https://docs.cloud.google.com/lakehouse/docs/introduction) (formerly BigLake). After completing this guide, you will be able to: - Create a catalog in GCP Lakehouse for Iceberg topic data. - Configure a Redpanda cluster to use GCP Lakehouse as an Iceberg REST catalog. - Query Iceberg topic data from Google BigQuery. For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/). > 📝 **NOTE** > > Check the [Lakehouse product page](https://docs.cloud.google.com/lakehouse/docs) for the latest status and availability of the REST Catalog API. ## [](#prerequisites)Prerequisites - A Google Cloud Platform (GCP) project. - Lakehouse must be in the same GCP project as the cluster. Cross-project Lakehouse is not supported. If you do not have permissions to manage GCP resources such as VMs, storage buckets, and service accounts in your project, ask your project owner to create or update them for you. - The [`gcloud` CLI](https://docs.cloud.google.com/sdk/docs/install) installed and configured for your GCP project. - [Lakehouse (BigLake) API](https://cloud.google.com/biglake/docs/enable-biglake-api) enabled for your GCP project. - Redpanda version 25.3 or later. - `rpk` [installed or updated](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) to the latest version. - You can also use the Redpanda Cloud API to [reference secrets in your cluster configuration](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties). > 📝 **NOTE** > > For BYOC clusters created before June 9, 2026, you must re-run `rpk cloud byoc gcp apply --redpanda-id= --project-id=` to enable the required API services before following this guide. This is a one-time operation. ## [](#limitations)Limitations ### [](#multi-region-bucket-support)Multi-region bucket support The Lakehouse runtime catalog does not support multi-region buckets. Use single-region buckets to store your Iceberg topics. ### [](#catalog-deletion)Catalog deletion Currently, it is not possible to delete non-empty Lakehouse Iceberg catalogs through the Lakehouse interface. If you need to reconfigure your setup, create a new bucket or use the REST API to remove the existing catalog. ### [](#topic-names)Topic names Lakehouse does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through Lakehouse, either: - Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`). - Ensure that the new topic names do not include dots. You must also set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). See [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) for the list of cluster properties to set when enabling the Lakehouse REST catalog integration. ## [](#set-up-google-cloud-resources)Set up Google Cloud resources For BYOC clusters, the required Lakehouse IAM permissions are automatically provisioned and attached to the cluster’s service account when Iceberg is enabled with a Lakehouse endpoint. You can skip to [Create a Lakehouse catalog](#create-a-lakehouse-catalog). For BYOVPC clusters, you must grant the required permissions to your cluster’s service account and enable the `biglake.googleapis.com` and `bigquery.googleapis.com` APIs in your GCP project. ### [](#grant-required-permissions)Grant required permissions Grant the necessary permissions to your service account. To run the following commands, replace the placeholder values: - ``: The name of your service account. - ``: The name of your storage bucket. 1. Grant the service account the [Storage Object Admin role](https://docs.cloud.google.com/storage/docs/access-control/iam-roles) to access the bucket: ```bash gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" ``` 2. Grant [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage) and [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake#biglake.editor) roles for using the Iceberg REST catalog: ```bash gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/serviceusage.serviceUsageConsumer" gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/biglake.editor" ``` ### [](#create-a-lakehouse-catalog)Create a Lakehouse catalog Create a Lakehouse Iceberg REST catalog using the [`gcloud biglake`](https://docs.cloud.google.com/sdk/gcloud/reference/biglake/iceberg/catalogs/create) command: ```bash gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project= ``` Replace the placeholder values: - ``: Use the name of your storage bucket as the catalog ID. - ``: Your GCP project ID. ## [](#configure-redpanda-for-iceberg)Configure Redpanda for Iceberg 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Use `rpk` as shown in the following example, or [use the Cloud API](https://docs.redpanda.com/cloud-data-platform/manage/cluster-maintenance/config-cluster/#set-cluster-configuration-properties) to update these cluster properties. The update might take several minutes to complete. ```bash rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://biglake.googleapis.com/iceberg/v1/restcatalog \ iceberg_rest_catalog_authentication_mode=gcp \ iceberg_rest_catalog_warehouse=gs:/// \ iceberg_rest_catalog_gcp_user_project= \ iceberg_dlq_table_suffix=_dlq ``` - ``: Your Redpanda cluster ID. - ``: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the object storage bucket you created as a customer-managed resource. - ``: Your GCP project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). 2. Enable the REST catalog integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` Iceberg data can take a few moments to become available in Lakehouse. ## [](#query-iceberg-topics-in-bigquery)Query Iceberg topics in BigQuery 1. Navigate to the [BigQuery console](https://console.cloud.google.com/bigquery). 2. Query your Iceberg topic using SQL. For example, to query the `transactions` topic in the quickstart cluster: ```sql SELECT * FROM `>redpanda`.transactions ORDER BY redpanda.timestamp DESC LIMIT 10 ``` Replace `` with your bucket name. Your Redpanda topic is now available as Iceberg tables in Lakehouse, allowing you to run analytics queries directly on your streaming data. ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and the BigLake catalog. To grant your end users and query engines read access to the Iceberg tables in BigQuery, see [Grant permissions for BigLake tables](https://cloud.google.com/bigquery/docs/manage-open-source-metadata#grant_permissions) in the Google Cloud documentation. ## [](#suggested-reading)Suggested reading - [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/) - [Query Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) - [Google Lakehouse for Apache Iceberg documentation](https://docs.cloud.google.com/lakehouse/docs/introduction) --- # Page 609: Troubleshoot Iceberg Topics **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting.md --- # Troubleshoot Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Troubleshoot Iceberg Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/iceberg-troubleshooting page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/iceberg-troubleshooting.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc description: Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. page-git-created-date: "2026-05-06" page-git-modified-date: "2026-05-26" --- Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. Use this page to: - Diagnose Iceberg translation errors using DLQ tables and metrics - Reprocess or drop invalid records from the DLQ table ## [](#dead-letter-queue)Dead-letter queue If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate DLQ Iceberg table named `~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format: - Redpanda cannot find the embedded schema ID in the Schema Registry. - Redpanda fails to translate one or more schema data types to an Iceberg type. - In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte. The DLQ table itself uses the `key_value` schema, consisting of two columns: the record metadata including the key, and a binary column for the record’s value. > 📝 **NOTE** > > Topic property misconfiguration, such as [overriding the default behavior of `value_schema_latest` mode](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#override-value-schema-latest-default) but not specifying the fully qualified Protobuf message name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. ### [](#inspect-dlq-table)Inspect DLQ table You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. The following example produces a record to a topic named `ClickEvent` and does not use the Schema Registry wire format that includes the magic byte and schema ID: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent --format='%k %v\n' ``` Querying the DLQ table returns the record that was not translated: ```sql SELECT value FROM ."ClickEvent~dlq"; -- Fully qualified table name ``` ```bash +-------------------------------------------------+ | value | +-------------------------------------------------+ | 7b 22 75 73 65 72 5f 69 64 22 3a 32 33 32 34 2c | | 22 65 76 65 6e 74 5f 74 79 70 65 22 3a 22 42 55 | | 54 54 4f 4e 5f 43 4c 49 43 4b 22 2c 22 74 73 22 | | 3a 22 32 30 32 34 2d 31 31 2d 32 35 54 32 30 3a | | 32 33 3a 35 39 2e 33 38 30 5a 22 7d | +-------------------------------------------------+ ``` The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema. ### [](#reprocess-dlq-records)Reprocess DLQ records You can apply a transformation and reprocess the record in your data lakehouse to the original Iceberg table. In this case, you have a JSON value represented as a UTF-8 binary. Depending on your query engine, you might need to decode the binary value first before extracting the JSON fields. Some query engines decode the binary value automatically: ClickHouse SQL example to reprocess DLQ record ```sql SELECT CAST(jsonExtractString(json, 'user_id') AS Int32) AS user_id, jsonExtractString(json, 'event_type') AS event_type, jsonExtractString(json, 'ts') AS ts FROM ( SELECT CAST(value AS String) AS json FROM .`ClickEvent~dlq` -- Ensure that the table name is properly parsed ); ``` ```bash +---------+--------------+--------------------------+ | user_id | event_type | ts | +---------+--------------+--------------------------+ | 2324 | BUTTON_CLICK | 2024-11-25T20:23:59.380Z | +---------+--------------+--------------------------+ ``` You can now insert the transformed record back into the main Iceberg table. Redpanda recommends using an exactly-once processing strategy to avoid duplicates when reprocessing records. ### [](#drop-invalid-records)Drop invalid records To disable the default behavior and drop an invalid record, set the `redpanda.iceberg.invalid.record.action` topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. ## [](#troubleshooting-metrics)Troubleshooting metrics The following [Iceberg metrics](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#iceberg-metrics) help identify translation errors, invalid records, and catalog connectivity issues: - [`redpanda_iceberg_translation_dlq_files_created`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_dlq_files_created): Number of DLQ Parquet files created. A non-zero and increasing value indicates records are failing to translate. See [Inspect DLQ table](#inspect-dlq-table) to examine the failed records. - [`redpanda_iceberg_translation_invalid_records`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_translation_invalid_records): Number of invalid records encountered during translation, labeled by cause. See [Drop invalid records](#drop-invalid-records) to configure how Redpanda handles these records. - [`redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#redpanda_iceberg_rest_client_num_commit_table_update_requests_failed): Failed table commit requests to the REST catalog. Applies only when using a REST catalog (`iceberg_catalog_type: rest`). Persistent failures indicate catalog connectivity or permission issues. --- # Page 610: Migrate Iceberg Catalogs **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/migrate-iceberg-catalog.md --- # Migrate Iceberg Catalogs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Migrate Iceberg Catalogs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/migrate-iceberg-catalog page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/migrate-iceberg-catalog.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/migrate-iceberg-catalog.adoc description: Switch the Iceberg catalog backend for an existing Redpanda cluster without losing untranslated topic data. page-git-created-date: "2026-05-30" page-git-modified-date: "2026-05-30" --- Use the procedures in this topic when moving from the filesystem-based `object_storage` catalog to a managed REST catalog, or when changing between REST catalogs. By doing so, you can switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data. This procedure guides you through pausing Iceberg translation per topic, draining pending commits to the old catalog, applying the new catalog configuration, and restarting the cluster. After completing these steps, you will be able to: - Verify that a target Iceberg catalog supports your existing schemas and partition specs - Pause Iceberg translation and drain pending commits without losing untranslated data - Apply new catalog configuration and resume translation safely > ❗ **IMPORTANT** > > Do not change `[iceberg_catalog_type](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_catalog_type)` or any other catalog cluster property in place without following this procedure. In-flight commits and untranslated data can be lost or stuck if the catalog changes mid-translation. ## [](#prerequisites)Prerequisites - Iceberg topics enabled and running on your Redpanda cluster. - Network connectivity from all brokers to the new catalog endpoint. - Credentials configured for the new catalog (REST endpoint, authentication mode, secret or token). For configuration guidance for each catalog type, see [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/). - The new catalog must support the current schema and partition spec of every Iceberg topic. See [Verify catalog compatibility](#verify-catalog-compatibility). ## [](#verify-catalog-compatibility)Verify catalog compatibility Before starting the migration, verify that the new catalog can host every Iceberg topic’s table with its existing schema and partition spec. If the new catalog is incompatible with your existing schemas or partition specs, already-translated Parquet files will fail to commit, and translation will stall in a state that is difficult to recover from. The simplest validation is to manually create a test table in the new catalog with the same schema and partition spec as one of your Iceberg topics. If the create call fails, fix the partition spec or schema before migrating. Delete the test table after validation. > ⚠️ **CAUTION** > > AWS Glue does not support partitioning on a nested field, which is Redpanda’s default partition spec for Iceberg topics. If you migrate to AWS Glue, you must change the partition spec to a Glue-compatible form before starting the migration procedure. ## [](#run-the-migration)Run the migration 1. Save the current `retention.ms` and `retention.bytes` values for every Iceberg topic, then set both to `-1` (infinite retention): ```bash rpk topic alter-config --set retention.ms=-1 --set retention.bytes=-1 ``` While Iceberg translation is paused in the next step, the topic’s retention anchor on the log is released. Without infinite retention, the cluster could delete untranslated data before the migration completes. 2. Pause Iceberg translation on every Iceberg topic by setting `redpanda.iceberg.mode` to `disabled`. Save each topic’s previous mode value so you can restore it later. ```bash rpk topic alter-config --set redpanda.iceberg.mode=disabled ``` Setting the mode to `disabled` stops new translation while letting already-translated data finish committing to the old catalog. For more about Iceberg modes, see [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/). > 📝 **NOTE** > > Do not change `[iceberg_enabled](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_enabled)` at the cluster level. The Iceberg integration must remain enabled at the cluster level so that pending commits can drain to the old catalog. 3. Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` metric until it reaches `0` for every Iceberg topic-partition. This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. A non-zero value here is expected while translation is paused, and reflects new records the cluster has not yet translated. > 💡 **TIP** > > If you scrape Prometheus, the following expression returns `0` only when every Iceberg-topic partition has fully drained: > > ```promql > sum(redpanda_iceberg_pending_commit_lag) > ``` 4. Apply the new catalog configuration. For example, to switch from `object_storage` to a REST catalog, update the catalog cluster properties: ```bash rpk cluster config set iceberg_catalog_type rest rpk cluster config set iceberg_rest_catalog_endpoint rpk cluster config set iceberg_rest_catalog_authentication_mode oauth2 # Set additional credential properties for your chosen authentication mode. ``` For full guidance on setting catalog cluster properties, see [Connect to a REST catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#rest) and the individual [REST catalog integration pages](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/rest-catalog/). 5. Restart Redpanda. The catalog cluster properties require a restart to take effect. Coordinate the restart with Redpanda Support. The restart must occur after `redpanda_iceberg_pending_commit_lag` has reached `0` and before you resume translation. 6. After the cluster comes up, check broker logs for successful catalog requests and the absence of authentication errors to verify the new catalog connection. 7. Resume Iceberg translation by restoring `redpanda.iceberg.mode` on every Iceberg topic to its previous value: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` 8. Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved before starting the migration. ## [](#verify-the-migration)Verify the migration After the migration completes, confirm that new data is reaching the new catalog: - Query an Iceberg table in your query engine using the new catalog and confirm that row counts continue to increase as your topic produces new records. - Check broker logs for any commit failures referencing the new catalog. Repeated failures often indicate a schema or partition spec mismatch. See [Troubleshooting](#troubleshooting) for details. ## [](#troubleshooting)Troubleshooting - Pending commits stuck after restart: A schema or partition spec mismatch between the original tables and the new catalog is the most common cause. See [Verify catalog compatibility](#verify-catalog-compatibility). If you cannot resolve the mismatch, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). - Authentication errors against the new REST catalog: Verify that the credential cluster properties (for example, `iceberg_rest_catalog_client_id`, `iceberg_rest_catalog_client_secret`, `iceberg_rest_catalog_token`) match what the new catalog expects. For OAuth, also check `iceberg_rest_catalog_oauth2_server_uri`. - Translation does not resume after restoring `redpanda.iceberg.mode`: Check that `redpanda_iceberg_pending_translation_lag` is increasing as new records are produced. If it remains `0`, the cluster is not translating new records. Verify that your producer is still writing to the topic and that the topic’s mode value is one of `key_value`, `value_schema_id_prefix`, or `value_schema_latest`. --- # Page 611: Migrate to Iceberg Topics **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/migrate-to-iceberg-topics.md --- # Migrate to Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Migrate to Iceberg Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/migrate-to-iceberg-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/migrate-to-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/migrate-to-iceberg-topics.adoc description: Migrate existing Iceberg integrations to Redpanda Iceberg topics. page-topic-type: how-to learning-objective-1: Compare external Iceberg integrations with Iceberg Topics architectures learning-objective-2: Implement data merge strategies using SQL patterns learning-objective-3: Execute validation checks and perform cutover procedures page-git-created-date: "2026-02-28" page-git-modified-date: "2026-05-26" --- Migrate existing Iceberg pipelines to Redpanda Iceberg topics to simplify your architecture and reduce operational overhead. After reading this page, you will be able to: - Compare external Iceberg integrations with Iceberg Topics architectures - Implement data merge strategies using SQL patterns - Execute validation checks and perform cutover procedures ## [](#why-migrate-to-iceberg-topics)Why migrate to Iceberg Topics Redpanda’s built-in Iceberg-enabled topics offer a simpler alternative to external Iceberg integrations for writing streaming data to Iceberg tables. > 📝 **NOTE** > > This page focuses on migrating from Kafka Connect Iceberg Sink. The migration patterns and SQL examples can be adapted for other Iceberg sources such as Apache Flink or Spark. ### [](#kafka-connect-iceberg-sink-comparison)Kafka Connect Iceberg Sink comparison The following table compares Kafka Connect Iceberg Sink with Redpanda Iceberg Topics: | Aspect | Kafka Connect Iceberg Sink | Iceberg Topics | | --- | --- | --- | | Infrastructure | Requires external Kafka Connect cluster | Built into Redpanda brokers | | Dependencies | Separate service to manage | No external dependencies | | Setup time | Medium (deploy connector) | Fast (enable topic property and post schema) | ## [](#prerequisites)Prerequisites To migrate from an existing Iceberg integration to Iceberg Topics, you must have: - [Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/) enabled on your Redpanda cluster. - Understanding of your current schema format (Avro, Protobuf, or JSON Schema). - For Kafka Connect migrations, knowledge of your Kafka Connect configuration, especially if using `iceberg.tables.route-field` for multi-table routing. - If migrating multi-table fan-out patterns, [data transforms](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/how-transforms-work/) enabled on your cluster. - Access to both source and target (Iceberg Topics) tables in your query engine. - Query engine access (Snowflake, Databricks, ClickHouse, or Spark) for data merging. ## [](#migration-steps)Migration steps Redpanda recommends following a phased approach to ensure data consistency and minimize risk: 1. Enable Iceberg on target topics and verify new data flows. 2. Run both systems concurrently during transition. 3. Choose a strategy to combine historical and new data. 4. Verify data completeness and accuracy. 5. Disable the external Iceberg integration. > ❗ **IMPORTANT** > > Iceberg Topics cannot append to existing Iceberg tables that are not created by Redpanda. You must create new Iceberg tables and merge historical data separately. ### [](#enable-iceberg-topics)Enable Iceberg Topics For simple migrations (one topic mapping to one Iceberg table), enable the Iceberg integration for your Redpanda topics. 1. Set the `iceberg_enabled` configuration property on your cluster to `true`: ###### rpk ```bash rpk cloud login rpk profile create --from-cloud rpk cluster config set iceberg_enabled true ``` ###### Cloud API ```bash # Store your cluster ID in a variable export RP_CLUSTER_ID= # Retrieve a Redpanda Cloud access token export RP_CLOUD_TOKEN=$(curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "content-type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=" \ -d "client_secret=") # Update cluster configuration to enable Iceberg topics curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \ "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H 'accept: application/json' \ -H 'content-type: application/json' \ -d '{"cluster_configuration":{"custom_properties": {"iceberg_enabled":true}}}' ``` 2. Configure the `redpanda.iceberg.mode` property for the topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` Choose the mode based on your message format and schema configuration. For Kafka Connect migrations, use this mapping: | Kafka Connect Converter | Recommended Iceberg Mode | | --- | --- | | io.confluent.connect.avro.AvroConverter | value_schema_id_prefix (messages already use Schema Registry wire format) | | io.confluent.connect.protobuf.ProtobufConverter | value_schema_id_prefix (messages already use Schema Registry wire format) | | org.apache.kafka.connect.json.JsonConverter with schemas | value_schema_latest (Schema Registry resolves schema automatically) | | org.apache.kafka.connect.json.JsonConverter with embedded schemas | key_value (schema included with each message) | See [Specify Iceberg Schema](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/) to learn more about the different Iceberg modes. 3. If using `value_schema_id_prefix` or `value_schema_latest` modes, register a schema for the topic: ```bash rpk registry schema create -value --schema --type ``` > ❗ **IMPORTANT** > > If using the `value_schema_id_prefix` mode, schema subjects must use the `-value` [naming convention](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-id-validation/#set-subject-name-strategy-per-topic) (TopicNameStrategy). Note the schema ID returned, in case you need it for troubleshooting. 4. Verify that new records are being written to the Iceberg table: - Check that data appears in your query engine. - Validate that the schema translation is correct. - Confirm record counts are increasing. #### [](#multi-table-fan-out-pattern)Multi-table fan-out pattern If your existing integration routes records to multiple Iceberg tables based on a field value (for example, Kafka Connect’s `iceberg.tables.route-field` property), you need to implement equivalent routing logic. You create separate Iceberg-enabled topics for each target table, and Redpanda automatically creates corresponding Iceberg tables. Use either of the following approaches to route records to the correct topic: ##### [](#option-1-data-transforms-with-separate-topics-recommended)Option 1: Data transforms with separate topics (recommended) Use a data transform to read the routing field from each message and write records to separate Iceberg-enabled topics. This approach keeps routing logic within Redpanda and avoids external dependencies. When using Iceberg modes that require schema validation, the transform can register schemas dynamically and encode messages with the appropriate format. 1. Enable data transforms on your cluster: ```bash rpk cluster config set data_transforms_enabled true ``` 2. Create output topics and enable Iceberg with Schema Registry validation: ```bash rpk topic create rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix ``` 3. Implement a transform function that: 1. Reads the routing field from each input message. 2. If using Schema Registry validation, registers schemas dynamically and encodes messages with the appropriate format. 3. Writes to a specific output topic based on the routing field. 4. Deploy the transform, specifying multiple output topics: ```bash rpk transform deploy \ --file transform.wasm \ --name \ --input-topic \ --output-topic \ --output-topic \ --output-topic ``` 5. Validate the fanout by checking that each output topic receives the correct records. For a complete implementation example with dynamic schema registration, see [Multi-topic fan-out with Schema Registry](https://docs.redpanda.com/cloud-data-platform/develop/data-transforms/build/#multi-topic-fanout). The example demonstrates Schema Registry wire format encoding for use with `value_schema_id_prefix` mode. ##### [](#option-2-external-stream-processor)Option 2: External stream processor Use an external stream processor for complex routing logic: 1. Use a stream processor ([Redpanda Connect](https://docs.redpanda.com/cloud-data-platform/develop/connect/about/) or Flink) to split records. 2. Write to separate Iceberg-enabled topics. This approach is more complex but offers more flexibility for advanced routing requirements not supported by data transforms. ### [](#validate-schema-registry-integration)Validate Schema Registry integration If using [`value_schema_id_prefix`](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/#value_schema_id_prefix) mode, verify that messages use the Schema Registry [wire format](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/#wire-format). ```bash rpk topic consume --num=1 --format='%v\n' | xxd | head -n 1 ``` If the first byte is not `00` (magic byte), you must configure your producer to use the wire format. The `value_schema_id_prefix` mode also requires that schema subjects follow the TopicNameStrategy: `-value`. Verify your schemas use the correct naming: ```bash rpk registry schema list ``` #### [](#verify-no-records-in-dlq)Verify no records in DLQ Check that no records failed validation and were written to the dead-letter queue. If records are present, see [Records in DLQ table](#records-in-dlq-table) for resolution steps. ```sql SELECT COUNT(*) FROM ."~dlq"; ``` ### [](#run-systems-in-parallel)Run systems in parallel Keep your existing Iceberg integration running while Iceberg Topics is enabled. This provides a safety net during the transition period: - New data flows to both the source tables and new Iceberg Topics tables. - You can validate data consistency between both systems. - You have a fallback option if issues arise. Run a query to compare record counts between systems: ```sql -- Source table SELECT COUNT(*) AS source_count FROM .; -- Iceberg Topics table SELECT COUNT(*) AS iceberg_topics_count FROM .; ``` Record counts should increase at similar rates, accounting for the time Iceberg Topics was enabled. Check for DLQ records (see [Records in DLQ table](#records-in-dlq-table)). Monitor Iceberg topic metrics to validate that data is flowing at expected rates: - `redpanda_iceberg_translation_parquet_rows_added`: Track rows written to Iceberg tables (compare with source write rate) - `redpanda_iceberg_translation_translations_finished`: Number of completed translation executions - `redpanda_iceberg_translation_invalid_records`: Records that failed validation - `redpanda_iceberg_translation_dlq_files_created`: Dead-letter queue activity - `redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`: Failed table commits to catalog If using data transforms for multi-table fanout, also monitor: - `redpanda_transform_processor_lag`: Records pending processing in transform input topic For a complete list of Iceberg metrics, see the [Iceberg metrics reference](https://docs.redpanda.com/cloud-data-platform/reference/public-metrics-reference/#iceberg-metrics). > 💡 **TIP** > > Run both systems for at least 24-48 hours to ensure stability before proceeding with data merge. ### [](#merge-historical-data)Merge historical data Choose a strategy to combine your historical data with new Iceberg Topics data. #### [](#option-1-insert-into-pattern-recommended)Option 1: INSERT INTO pattern (recommended) Use this approach to create a unified table with all data, taking into consideration the following: - You want a single table for queries. - You can afford the one-time data copy cost. - You need optimal query performance. This SQL pattern uses partition and offset metadata to identify and copy only records not yet in the target table: ```sql -- Step 1: Find the latest offset per partition in the target (Iceberg Topics) table WITH latest_offsets AS ( SELECT partition, MAX(offset) AS max_offset FROM target_iceberg_topics_table GROUP BY partition ) -- Step 2: Insert records from source table that don't exist in target INSERT INTO target_iceberg_topics_table SELECT s.* FROM source_table AS s LEFT JOIN latest_offsets AS t ON s.partition = t.partition WHERE t.max_offset IS NULL -- Partition not seen before in target OR s.offset > t.max_offset; -- Record is newer than target's latest offset ``` - The `latest_offsets` CTE finds the highest offset in the target table for each partition. - The `LEFT JOIN` ensures you include partitions never seen before in the target (`t.max_offset IS NULL`). - The `WHERE` clause filters to only records with offsets greater than the target’s latest. - This avoids duplicates by using Kafka partition and offset as the deduplication key. This approach may take significant time for large datasets. Consider executing this process during low-query periods. You can also execute on an incremental basis to ease the load on your query engine, for example, by date or partition ranges. #### [](#option-2-view-based-query-federation)Option 2: View-based query federation Use this approach to query both tables without copying data if: - You cannot afford data copy time or cost. - You need immediate access to a unified view. - Query complexity and performance are acceptable with federated queries. - You may consolidate data later. Create a view that queries both tables and deduplicates on the fly: ```sql CREATE VIEW unified_iceberg_view AS WITH latest_offsets AS ( SELECT partition, MAX(offset) AS max_offset FROM target_iceberg_topics_table GROUP BY partition ), historical_data AS ( SELECT s.* FROM source_table AS s LEFT JOIN latest_offsets AS t ON s.partition = t.partition WHERE t.max_offset IS NULL OR s.offset <= t.max_offset -- Only historical records not in target ), new_data AS ( SELECT * FROM target_iceberg_topics_table ) SELECT * FROM historical_data UNION ALL SELECT * FROM new_data; ``` Most Iceberg-compatible query engines support views, including Snowflake, Databricks, ClickHouse, and Spark. ### [](#validate-the-migration)Validate the migration After completing the data merge, verify the migration before cutting over: - Record counts match between source and target: ```sql -- Compare record counts SELECT 'Source' AS table_name, COUNT(*) AS record_count FROM . UNION ALL SELECT 'Target', COUNT(*) FROM .; ``` - All partitions are represented in the target: ```sql -- Check for missing partitions SELECT DISTINCT partition FROM . EXCEPT SELECT DISTINCT partition FROM .; -- Should return no rows ``` - Date ranges cover the full historical period. Compare `MIN(timestamp)` and `MAX(timestamp)` between source and target tables to ensure the target covers the same time range. - No gaps in offset sequences: ```sql -- Check for offset gaps (may indicate missing data) WITH offset_check AS ( SELECT partition, offset, LAG(offset) OVER (PARTITION BY partition ORDER BY offset) AS prev_offset FROM . ) SELECT * FROM offset_check WHERE offset - prev_offset > 1; -- Should return no rows ``` - Sample queries return expected results. Spot check specific records by ID to verify data accuracy. - Schema translation is correct. Run `DESCRIBE` on both tables and verify all fields are present with correct data types. - New records are flowing to Iceberg Topics. Check record count for a recent time window (for example, the last hour). - Query performance is acceptable. - Monitoring and alerts are configured. - No records in DLQ (see [Records in DLQ table](#records-in-dlq-table)). ### [](#troubleshoot-common-migration-issues)Troubleshoot common migration issues #### [](#records-in-dlq-table)Records in DLQ table Iceberg Topics write records that fail validation to a dead-letter queue (DLQ) table. Records may appear in the DLQ due to: - Schema Registry issues. For example, using the wrong schema subject name, or Redpanda cannot find the embedded schema ID in Schema Registry. - When using `value_schema_id_prefix` mode: messages not encoded with Schema Registry wire format. - Incompatible schema changes. For example, changing field types or removing required fields. - Data type translation failures. To check for DLQ records during migration: ```sql SELECT COUNT(*) FROM ."~dlq"; ``` If the count is greater than zero, inspect the failed records. See [Troubleshoot Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/) for steps to inspect and reprocess DLQ records. #### [](#multi-table-fan-out-transform-issues)Multi-table fan-out transform issues If the transform does not process messages, check if: - The specified output topics don’t exist or aren’t enabled with Iceberg. - The routing logic in the transform is incorrect, or the routing field is missing from input messages. - (When using Schema Registry validation) The schema registration failed during initialization, preventing the transform from starting. To check the transform status: ```bash rpk transform list ``` To view logs and check for errors: ```bash rpk transform logs ``` To check for routing errors: ```bash rpk transform logs | grep -i "unknown\|error" ``` If using Schema Registry validation, verify schema registration: ```bash # Check transform logs for schema registration messages rpk transform logs | grep -i "schema" # List registered schemas rpk registry schema list ``` ### [](#plan-for-rollback)Plan for rollback Before cutting over, ensure you have a rollback strategy. See the [Pre-cutover checklist](#pre-cutover-checklist) in the cutover section to verify you’re ready. #### [](#rollback-during-parallel-operation)Rollback during parallel operation If you discover issues while both systems are running: 1. Keep producing to both systems. 2. Point consumers back to source tables. 3. Investigate Iceberg Topics issues using troubleshooting section. 4. Fix issues and re-validate. 5. Attempt cutover again when ready. #### [](#rollback-after-external-integration-disabled)Rollback after external integration disabled > ⚠️ **WARNING** > > Rollback after stopping your external Iceberg integration may result in data loss or gaps. If you must rollback after disabling the external integration: 1. Restart your external Iceberg integration immediately. 2. Identify data written only to Iceberg Topics during the gap. 3. Export that data from Iceberg Topics tables: ```sql SELECT * FROM iceberg_topics_table WHERE timestamp > ''; ``` 4. Write exported data back to the source system (for example, Kafka Connect input topics or directly to source tables). 5. Verify data completeness across both systems. 6. Resume operations on the external integration. Redpanda recommends maintaining the ability to rollback for at least seven days after cutover to allow for issue discovery. ### [](#cut-over-to-iceberg-topics)Cut over to Iceberg Topics #### [](#pre-cutover-checklist)Pre-cutover checklist Before disabling your external Iceberg integration, ensure you have completed all validation steps: - All historical data is successfully merged (see [Merge historical data](#merge-historical-data)). - Parallel operation is complete and stable for at least 24-48 hours. - All validation queries pass (see [Validate the migration](#validate-the-migration)). - No records in DLQ tables, or all DLQ records are investigated and resolved. - Query performance meets requirements. - Downstream consumers are successfully tested with Iceberg Topics tables. - Monitoring and alerts are configured. - Rollback plan is verified and documented. #### [](#cutover-procedure)Cutover procedure 1. Set an appropriate maintenance window, ideally during low-traffic periods. 2. Stop your external Iceberg integration. **For Kafka Connect:** ```bash # Stop connector curl -X PUT http:///kafka-connect/clusters/iceberg-sink-connector/stop # Or delete connector (permanent) curl -X DELETE http:///kafka-connect/clusters/iceberg-sink-connector ``` 3. Monitor Iceberg Topics to ensure data continues flowing. 4. Verify that no new records are being written to source tables: ```sql SELECT MAX(timestamp) FROM .; -- Should not change after integration is stopped ``` 5. Run validation queries from [Validate the migration](#validate-the-migration) after 1-2 hours of operation. 6. Wait for a short period, such as 24-48 hours, to monitor and validate stability. 7. If migrating to a unified table of historical plus new data, optionally delete old source tables after an extended validation period (for example, at least seven days): > 📝 **NOTE** > > Ensure you have backups before deleting historical data. Some organizations keep old tables for compliance or audit purposes. ```sql DROP TABLE .; ``` 8. Decommission external Iceberg infrastructure after an extended safety period (30+ days, for example). If any issues arise during cutover, see [Plan for rollback](#plan-for-rollback). ## [](#next-steps)Next steps - [Query Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) - [About Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/) --- # Page 612: Query Iceberg Topics **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics.md --- # Query Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Query Iceberg Topics latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/query-iceberg-topics page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/query-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/query-iceberg-topics.adoc description: Query Redpanda topic data stored in Iceberg tables, based on the topic Iceberg mode and schema. page-git-created-date: "2025-04-04" page-git-modified-date: "2026-05-26" --- When you access Iceberg topics from a data lakehouse or other Iceberg-compatible tools, how you consume the data depends on the topic [Iceberg mode](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema/) and whether you’ve registered a schema for the topic in the [Redpanda Schema Registry](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/). You do not need to rely on complex ETL jobs or pipelines to access real-time data from Redpanda. After reading this page, you will be able to: - Query Redpanda topic data from an Iceberg-compatible engine - Grant end-user and query engine access to Iceberg data ## [](#access-iceberg-tables)Access Iceberg tables Redpanda generates an Iceberg table with the same name as the topic. Depending on the processing engine and your Iceberg catalog implementation, you may also need to define the table (for example using `CREATE TABLE`) to point the data lakehouse to its location in the catalog. For BYOC clusters, the bucket name and table location are as follows: | Cloud provider | Bucket or container name | Iceberg table location | | --- | --- | --- | | AWS | redpanda-cloud-storage- | redpanda-iceberg-catalog/redpanda/ | | Azure | The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. | | GCP | redpanda-cloud-storage- | For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster’s allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. For example, to add subnet IDs to the allow list through the Control Plane API [`PATCH /v1/clusters/`](https://docs.redpanda.com/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster) endpoint, run: ```bash curl -X PATCH https://api.cloud.redpanda.com/v1/clusters/ \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" \ -d @- << EOF { "cloud_storage": { "azure": { "allowed_subnet_ids": [ ] } } } EOF ``` ### [](#grant-access-to-query-engine-users)Grant access to query engine users Redpanda manages the service-to-service permissions between Redpanda and the catalog (see [Use Iceberg Catalogs](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/)). However, you are responsible for granting your end users and query engines (such as Amazon Athena, Apache Spark, Trino, or Snowflake) read access to the Iceberg data. Use either or both of the following approaches to control access: #### [](#cloud-storage-prefix-level-access)Cloud storage prefix-level access Grant query engine roles or users read access to the Iceberg data prefix in the cluster’s storage bucket. This controls who can read the underlying data and metadata files. Scope permissions to specific prefixes to restrict access to individual tables. - AWS (S3): Use IAM policies to grant `s3:GetObject` and `s3:ListBucket` on the Iceberg prefix (for example, `/redpanda-iceberg-catalog/*`). See [Using IAM policies with Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-iam-policies.html). - GCP (GCS): Use IAM conditions or bucket-level policies to grant `storage.objects.get` and `storage.objects.list` on the Iceberg prefix. See [GCS IAM permissions](https://cloud.google.com/storage/docs/access-control/iam). - Azure (Blob Storage): Use Azure RBAC roles such as Storage Blob Data Reader scoped to the container or prefix. See [Authorize access to blob data](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory). #### [](#catalog-level-table-access)Catalog-level table access If you use a REST catalog, you can control access at the table level through the catalog’s own access control layer. Use this approach when query engines access tables through the catalog rather than reading files directly. - AWS Glue: Use [AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) to grant table-level and column-level permissions. - Databricks Unity Catalog: See the [Unity Catalog privileges documentation](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html). - Snowflake Open Catalog: See [Open Catalog access control](https://docs.snowflake.com/en/user-guide/opencatalog/access-control). - GCP BigLake: See [BigLake table permissions](https://cloud.google.com/bigquery/docs/manage-open-source-metadata#grant_permissions). ### [](#refresh-table-data)Refresh table data Some query engines may require you to manually refresh the Iceberg table snapshot (for example, by running a command like `ALTER TABLE REFRESH;`) to see the latest data. If your engine needs the full JSON metadata path, use the following: ```none redpanda-iceberg-catalog/redpanda//metadata/v.metadata.json ``` This provides read access to all snapshots written as of the specified table version (denoted by `version-number`). > 📝 **NOTE** > > Redpanda automatically removes expired snapshots on a periodic basis. Snapshot expiry helps maintain a smaller metadata size and reduces the window available for [time travel](#time-travel-queries). ## [](#query-examples)Query examples To follow along with the examples on this page, suppose you produce the same stream of events to a topic `ClickEvent`, which uses a schema, and another topic `ClickEvent_key_value`, which uses the key-value mode. The topic’s Iceberg data is stored in an AWS S3 bucket. A sample record contains the following data: ```bash {"user_id": 2324, "event_type": "BUTTON_CLICK", "ts": "2024-11-25T20:23:59.380Z"} ``` > 📝 **NOTE** > > The query examples on this page use `redpanda` as the Iceberg namespace, which is the default. If you configured a different namespace using `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`, replace `redpanda` with your configured namespace. ### [](#topic-with-schema-value_schema_id_prefix-mode)Topic with schema (`value_schema_id_prefix` mode) > 📝 **NOTE** > > The steps in this section also apply to the `value_schema_latest` mode, except the produce step. The `value_schema_latest` mode is not compatible with the Schema Registry wire format. The [`rpk topic produce`](#reference:rpk/rpk-topic/rpk-topic-produce) command embeds the wire format header, so you must use your own producer code with `value_schema_latest`. Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`: `schema.avsc` ```avro { "type" : "record", "namespace" : "com.redpanda.examples.avro", "name" : "ClickEvent", "fields" : [ { "name": "user_id", "type" : "int" }, { "name": "event_type", "type" : "string" }, { "name": "ts", "type": "string" } ] } ``` 1. Register the schema under the `ClickEvent-value` subject: ```bash rpk registry schema create ClickEvent-value --schema path/to/schema.avsc --type avro ``` 2. Produce to the `ClickEvent` topic using the following format: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent --format='%k %v\n' --schema-id=topic ``` The `value_schema_id_prefix` mode requires that you produce to a topic using the [Schema Registry wire format](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/#wire-format), which includes the magic byte and schema ID in the prefix of the message payload. This allows Redpanda to identify the correct schema version in the Schema Registry for a record. 3. The following Spark SQL query returns values from columns in the `ClickEvent` table, with the table structure derived from the schema, and column names matching the schema fields. If you’ve integrated a catalog, query engines such as Spark SQL provide Iceberg integrations that allow easy discovery and access to existing Iceberg tables in object storage. ```sql SELECT * FROM ``.redpanda.ClickEvent; ``` ```bash +-----------------------------------+---------+--------------+--------------------------+ | redpanda | user_id | event_type | ts | +-----------------------------------+---------+--------------+--------------------------+ | {"partition":0,"offset":0,"timestamp":2025-03-05 15:09:20.436,"headers":null,"key":null} | 2324 | BUTTON_CLICK | 2024-11-25T20:23:59.380Z | +-----------------------------------+---------+--------------+--------------------------+ ``` ### [](#topic-in-key-value-mode)Topic in key-value mode In `key_value` mode, you do not associate the topic with a schema in the Schema Registry, which means using semi-structured data in Iceberg. The record keys and values can have an arbitrary structure, so Redpanda stores them in [binary format](https://apache.github.io/iceberg/spec/?h=spec#primitive-types) in Iceberg. In this example, assume that you have created the `ClickEvent_key_value` topic, and set `redpanda.iceberg.mode` to `key_value`. 1. Produce to the `ClickEvent_key_value` topic using the following format: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent_key_value --format='%k %v\n' ``` 2. The following Spark SQL query returns the semi-structured data in the `ClickEvent_key_value` table. The table consists of two columns: one named `redpanda`, containing the record key and other metadata, and another binary column named `value` for the record’s value: ```sql SELECT * FROM ``.redpanda.ClickEvent_key_value; ``` ```bash +-----------------------------------+------------------------------------------------------------------------------+ | redpanda | value | +-----------------------------------+------------------------------------------------------------------------------+ | {"partition":0,"offset":0,"timestamp":2025-03-05 15:14:30.931,"headers":null,"key":key1} | {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"} | +-----------------------------------+------------------------------------------------------------------------------+ ``` Depending on your query engine, you might need to first decode the binary value to display the record key and value using a SQL helper function. For example, see the [`decode` and `unhex`](https://spark.apache.org/docs/latest/api/sql/index.html#unhex) Spark SQL functions, or the [HEX\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/hex_decode_string) Snowflake function. Some engines may also automatically decode the binary value for you. ### [](#time-travel-queries)Time travel queries Some query engines, such as Spark, support time travel with Iceberg, allowing you to query the table as it existed at a specific point in the past. You can run a time travel query by specifying a timestamp or version number. Redpanda automatically removes expired snapshots on a periodic basis, which also reduces the window available for time travel queries. By default, Redpanda retains snapshots for five days, so you can query Iceberg tables as of up to five days ago. The following example queries a `ClickEvent` table at a specific timestamp in Spark: ```sql SELECT * FROM ``.redpanda.ClickEvent TIMESTAMP AS OF '2025-03-02 10:00:00'; ``` --- # Page 613: Query Iceberg Topics using Snowflake and Open Catalog **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog.md --- # Query Iceberg Topics using Snowflake and Open Catalog > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Query Iceberg Topics using Snowflake and Open Catalog latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/redpanda-topics-iceberg-snowflake-catalog page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc description: Add Redpanda topics as Iceberg tables that you can query in Snowflake using an Open Catalog integration. page-git-created-date: "2025-05-21" page-git-modified-date: "2026-05-26" --- This guide walks you through querying Redpanda topics as Iceberg tables in [Snowflake](https://docs.snowflake.com/en/user-guide/tables-iceberg), with Amazon S3 as object storage and a catalog integration using [Open Catalog](https://docs.snowflake.com/en/user-guide/opencatalog/overview). After reading this page, you will be able to: - Configure AWS IAM credentials granting Open Catalog access to your S3 bucket - Integrate Redpanda Iceberg topics with Snowflake using Open Catalog ## [](#prerequisites)Prerequisites - `rpk` or familiarity with the Redpanda Cloud API to use secrets in your cluster configuration. For `rpk`, see [Install or Update rpk](https://docs.redpanda.com/cloud-data-platform/manage/rpk/rpk-install/). For the Cloud API, you must [authenticate](https://docs.redpanda.com/api/cloud-controlplane/authentication) using a service account. - A Snowflake account. - An Open Catalog account. To [create an Open Catalog account](https://other-docs.snowflake.com/en/opencatalog/create-open-catalog-account), you require ORGADMIN access in Snowflake. ## [](#authorize-access-to-open-catalog)Authorize access to Open Catalog You must create an AWS IAM policy and role that Open Catalog and Snowflake use to access the S3 bucket where your Iceberg data is stored. Redpanda writes Iceberg data and metadata files to the bucket using your cluster’s existing object storage credentials, so Redpanda does not need additional IAM configuration for its own S3 access. You finish configuring this role’s trust policy when you create the catalog and the external volume, using values that Open Catalog and Snowflake generate for you. ### [](#create-an-iam-policy)Create an IAM policy Create an IAM policy with the following S3 permissions, scoped to your cluster’s storage bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion" ], "Resource": "arn:aws:s3:::/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::" } ] } ``` Replace `` with the name of your cluster’s object storage bucket. You use this same IAM policy for both the Open Catalog role and the Snowflake external volume. ### [](#create-an-iam-role)Create an IAM role Create an IAM role and attach the IAM policy you created. Open Catalog and Snowflake each need this role’s ARN before they can generate the IAM user and external ID values you use to finish configuring the trust policy, so create the role with a temporary trust relationship for now: 1. In the AWS IAM console, create a new role. 2. For the trusted entity type, select **AWS account**. Under **An AWS account**, select **This account**. 3. Attach the IAM policy you created. 4. Note the role’s ARN (``). You update this role’s trust policy after Open Catalog and Snowflake generate the values you need, when you create the catalog and the external volume. ## [](#create-a-catalog-in-open-catalog)Create a catalog in Open Catalog Create the catalog that Redpanda’s Iceberg data and metadata are registered to, with your Tiered Storage S3 bucket configured as external storage: 1. In Open Catalog, create a new catalog. 2. For **S3 role ARN**, enter ``, the ARN of the IAM role you created. 3. Configure the catalog’s external storage location to point to your Tiered Storage S3 bucket. > 📝 **NOTE** > > Your Open Catalog account must be in the same AWS region as your S3 bucket. 4. On the Open Catalog home page, in the **Catalogs** pane, select the catalog you created. Under **Storage Details**, copy the **IAM user arn** (``). 5. If you didn’t specify an external ID when you created the IAM role, Open Catalog generates one for you (``). Record this value. For complete steps, see the [Open Catalog documentation](https://docs.snowflake.com/en/user-guide/opencatalog/create-catalog). ### [](#update-the-iam-role-trust-policy-for-open-catalog)Update the IAM role trust policy for Open Catalog Update the trust policy for the IAM role you created, using the IAM user ARN and external ID that Open Catalog generated when you created the catalog: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` After you update the trust policy, Open Catalog can assume the role to read and write Iceberg data and metadata in your S3 bucket. ## [](#create-a-snowflake-external-volume)Create a Snowflake external volume Create a Snowflake external volume using the same IAM role you created for Open Catalog. Snowflake provisions its own IAM user to assume the role, so you add a second statement to the role’s trust policy rather than replacing the statement that trusts Open Catalog: 1. In Snowflake, run `CREATE EXTERNAL VOLUME`, pointing `STORAGE_AWS_ROLE_ARN` to the IAM role you created: ```sql CREATE OR REPLACE EXTERNAL VOLUME STORAGE_LOCATIONS = ( ( NAME = '' STORAGE_PROVIDER = 'S3' STORAGE_BASE_URL = 's3:///' STORAGE_AWS_ROLE_ARN = '' STORAGE_AWS_EXTERNAL_ID = '' ) ) ALLOW_WRITES = TRUE; ``` Use your own values for the following placeholders: - ``: Provide a name for your external volume in Snowflake. - ``: Provide a name for the storage location. - ``: Choose an external ID for the volume’s trust relationship, distinct from the external ID Open Catalog uses. If you don’t set `STORAGE_AWS_EXTERNAL_ID`, Snowflake generates one for you. 2. Run `DESC EXTERNAL VOLUME` to retrieve the IAM user ARN that Snowflake generated for the volume: ```sql DESC EXTERNAL VOLUME ; ``` Record the `STORAGE_AWS_IAM_USER_ARN` value from the output (``). 3. Add a new statement to the IAM role’s trust policy for the volume’s IAM user ARN and external ID, alongside the existing statement that trusts Open Catalog: > ❗ **IMPORTANT** > > Add this as a new statement in the trust policy’s `Statement` array. If you replace the existing trust policy instead, Open Catalog loses access to the bucket. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` 4. Run `SYSTEM$VERIFY_EXTERNAL_VOLUME` to confirm Snowflake can access the bucket: ```sql SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME(''); ``` For complete steps, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume-s3). ## [](#set-up-catalog-integration-using-open-catalog)Set up catalog integration using Open Catalog To integrate Iceberg-enabled topics with Open Catalog, create a service connection and configure catalog roles. ### [](#create-a-new-open-catalog-service-connection-for-redpanda)Create a new Open Catalog service connection for Redpanda To create a new service connection to integrate the Iceberg-enabled topics into Open Catalog: 1. In Open Catalog, select **Connections**, then **\+ Connection**. 2. In **Configure Service Connection**, provide a name. Open Catalog creates a new principal with this name. 3. Make sure **Create new principal role** is selected. 4. Enter a name for the principal role. Then, click **Create**. After you create the connection, get the client ID and client secret. Save these credentials to add to your cluster configuration in a later step. ### [](#create-a-catalog-role)Create a catalog role Grant privileges to the principal created in the previous step: 1. In Open Catalog, select **Catalogs**, and select your catalog. 2. On the **Roles** tab of your catalog, click **\+ Catalog Role**. 3. Give the catalog role a name. 4. Under **Privileges**, select `CATALOG_MANAGE_CONTENT`. This provides full management [privileges](https://docs.snowflake.com/en/user-guide/opencatalog/access-control#catalog-privileges) for the catalog. Then, click **Create**. 5. On the **Roles** tab of the catalog, click **Grant to Principal Role**. 6. Select the catalog role you just created. 7. Select the principal role you created earlier. Click **Grant**. ### [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Open Catalog: 1. [Store the Open Catalog client secret in your cluster](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#store-a-secret-for-rest-catalog-authentication) using `rpk` or the Data Plane API. 2. [Edit your cluster configuration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#use-a-secret-in-cluster-configuration) to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below using `rpk` or the Control Plane API. For example, to use `rpk cluster config set`, run: ```bash rpk cluster config set \ iceberg_enabled=true \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://-.snowflakecomputing.com/polaris/api/catalog \ iceberg_rest_catalog_authentication_mode=oauth2 \ iceberg_rest_catalog_client_id= \ iceberg_rest_catalog_client_secret='${secrets.}' \ iceberg_rest_catalog_warehouse= # Optional properties: # iceberg_translation_interval_ms_default=1000 # iceberg_catalog_commit_interval_ms=1000 ``` Use your own values for the following placeholders: - `` and ``: Your [Open Catalog account URI](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters) is composed of these values. > 💡 **TIP** > > In Snowflake, navigate to **Admin**, then **Accounts**. Click the ellipsis near your Open Catalog account name, and select **Manage URLs**. The **Current URL** contains `` and ``. - ``: The client ID of the service connection you created in an earlier step. - ``: The name of the secret you created in the previous step. You must pass the secret name to the `${secrets.}` placeholder, not the secret value itself. - ``: The name of your catalog in Open Catalog. ```bash Successfully updated configuration. New configuration version is 2. ``` 3. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns: one for the record metadata including the key, and another binary column for the record’s value. See [Enable Iceberg integration](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration) for more details on Iceberg modes. Use any of the following to set `redpanda.iceberg.mode`: - `rpk`. See the following examples to run `rpk topic` commands. - The Cloud UI. Navigate to **Topics** to create a new topic and specify `redpanda.iceberg.mode` in **Additional Configuration**, or edit an existing topic under the topic’s **Configuration** tab. - The Data Plane API to [create a new topic](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-topicservice_createtopic) or [update a property for an existing topic](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-topicservice_updatetopicconfigurations). Specify the key-value pair for `redpanda.iceberg.mode` in the request body. The following examples show how to use `rpk` to create a new topic or alter the configuration for an existing topic, setting the Iceberg mode to `key_value`. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 4. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` The topic appears as a table in Open Catalog. 1. In Open Catalog, select **Catalogs**, then open your catalog. 2. Under your catalog, you will see the `redpanda` namespace (or the namespace you configured with `[iceberg_default_catalog_namespace](https://docs.redpanda.com/cloud-data-platform/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`), and a table with the name of your topic. The namespace and the table are automatically added for you. ## [](#query-iceberg-table-in-snowflake)Query Iceberg table in Snowflake To query the topic in Snowflake, you must create a [catalog integration](https://docs.snowflake.com/en/user-guide/tables-iceberg#catalog-integration) so that Snowflake has access to the table data and metadata. ### [](#configure-catalog-integration-with-snowflake)Configure catalog integration with Snowflake 1. Run the [`CREATE CATALOG INTEGRATION`](https://docs.snowflake.com/sql-reference/sql/create-catalog-integration-open-catalog) command in Snowflake: ```sql CREATE CATALOG INTEGRATION CATALOG_SOURCE = POLARIS TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'redpanda' REST_CONFIG = ( CATALOG_URI = '' WAREHOUSE = '' ) REST_AUTHENTICATION = ( TYPE = OAUTH OAUTH_CLIENT_ID = '' OAUTH_CLIENT_SECRET = '' OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL') ) REFRESH_INTERVAL_SECONDS = 30 ENABLED = TRUE; ``` Use your own values for the following placeholders: - ``: Provide a name for your Iceberg catalog integration in Snowflake. - ``: Your [Open Catalog account URI](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters) (`[https://-.snowflakecomputing.com/polaris/api/catalog](https://\-\.snowflakecomputing.com/polaris/api/catalog)`). - ``: The name of your catalog in Open Catalog. - ``: The client ID of the service connection you created in an earlier step. - ``: The client secret of the service connection you created in an earlier step. 2. Run the following command to verify that the catalog is integrated correctly: ```sql SELECT SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG(''); ``` ```bash # Example result for redpanda.iceberg.mode=key_value +-----------------------------------------------------------------------+ | SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('') | +-----------------------------------------------------------------------+ | [{"namespace":"redpanda","name":""}] | +-----------------------------------------------------------------------+ ``` ### [](#create-iceberg-table-in-snowflake)Create Iceberg table in Snowflake After creating the catalog integration, you must create an externally-managed table in Snowflake. You must run your Snowflake queries against this table. In your Snowflake database, run the [CREATE ICEBERG TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-rest) command. The following example also specifies that the table should automatically refresh metadata: ```sql CREATE ICEBERG TABLE CATALOG = '' EXTERNAL_VOLUME = '' CATALOG_TABLE_NAME = '' AUTO_REFRESH = TRUE ``` Use your own values for the following placeholders: - ``: Provide a name for your table in Snowflake. - ``: The name of the catalog integration you configured in an earlier step. - ``: The name of the external volume you configured using the Tiered Storage bucket. - ``: The name of the table in your catalog, which is the same as your Redpanda topic name. ### [](#query-the-iceberg-table)Query the Iceberg table To verify that Snowflake has successfully created the table containing the topic data, run the following: ```sql SELECT * FROM ; ``` Your query results should look like the following: ```bash # Example for redpanda.iceberg.mode=key_value with 3 records produced to topic +--------------------------------------------------------------------------------------------------------------+------------+ | REDPANDA | VALUE | +--------------------------------------------------------------------------------------------------------------+------------+ | { "partition": 0, "offset": 0, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "68656C6C6F"} | 776F726C64 | | { "partition": 0, "offset": 1, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "666F6F"} | 626172 | | { "partition": 0, "offset": 2, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "62617A" } | 717578 | +--------------------------------------------------------------------------------------------------------------+------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and Open Catalog. To grant your Snowflake users or other query engines read access to the Iceberg tables, use [Open Catalog access control](https://docs.snowflake.com/en/user-guide/opencatalog/access-control) to assign catalog privileges. For example, you can grant `TABLE_READ_DATA` to a read-only role rather than the `CATALOG_MANAGE_CONTENT` privilege used by the Redpanda service principal. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/query-iceberg-topics/) --- # Page 614: Integrate with REST Catalogs **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/rest-catalog.md --- # Integrate with REST Catalogs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Integrate with REST Catalogs latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/rest-catalog/index page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/rest-catalog/index.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/rest-catalog/index.adoc description: Integrate Redpanda topics with managed Iceberg REST Catalogs. page-git-created-date: "2025-08-05" page-git-modified-date: "2025-11-27" --- > 💡 **TIP** > > These guides are for integrating Iceberg topics with managed REST catalogs. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. - [Query Iceberg Topics using AWS Glue](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-aws-glue/) Add Redpanda topics as Iceberg tables that you can access through the AWS Glue Data Catalog. - [Query Iceberg Topics using Databricks and Unity Catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-databricks-unity/) Add Redpanda topics as Iceberg tables that you can query in Databricks managed by Unity Catalog. - [Use Iceberg Topics with GCP Lakehouse](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-topics-gcp-biglake/) Add Redpanda topics as Iceberg tables to Google Lakehouse for Apache Iceberg that you can query from Google BigQuery. - [Query Iceberg Topics using Snowflake and Open Catalog](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog/) Add Redpanda topics as Iceberg tables that you can query in Snowflake using an Open Catalog integration. --- # Page 615: Specify Iceberg Schema **URL**: https://docs.redpanda.com/cloud-data-platform/manage/iceberg/specify-iceberg-schema.md --- # Specify Iceberg Schema > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [cloud-data-platform-full.txt](https://docs.redpanda.com/cloud-data-platform-full.txt) --- title: Specify Iceberg Schema latest-operator-version: v26.2.1 latest-console-tag: v3.10.0 latest-connect-version: 4.105.0 latest-redpanda-tag: v26.2.1 docname: iceberg/specify-iceberg-schema page-component-name: cloud-data-platform page-version: master page-component-version: master page-component-title: Cloud page-relative-src-path: iceberg/specify-iceberg-schema.adoc page-edit-url: https://github.com/redpanda-data/cloud-docs/edit/main/modules/manage/pages/iceberg/specify-iceberg-schema.adoc description: Learn about supported Iceberg modes and how you can integrate schemas with Iceberg topics. page-git-created-date: "2025-07-31" page-git-modified-date: "2026-05-26" --- In [Iceberg-enabled clusters](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration), the `redpanda.iceberg.mode` topic property determines how Redpanda maps topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a schema in Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. After reading this page, you will be able to: - Configure the redpanda.iceberg.mode property when you create or update a topic - Choose the Iceberg mode that produces the table structure your data consumers need - Apply independent translation for record keys, values, and headers ## [](#supported-iceberg-modes)Supported Iceberg modes Redpanda supports the following modes for Iceberg topics: ### [](#key_value)key_value Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. ### [](#value_schema_id_prefix)value_schema_id_prefix Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in [Schema Registry](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/) and producers must write to the topic using the Schema Registry wire format. In the [Schema Registry wire format](https://docs.redpanda.com/cloud-data-platform/manage/schema-reg/schema-reg-overview/#wire-format), a "magic byte" and schema ID are embedded in the message payload header. Producers to the topic must use the wire format in the serialization process so Redpanda can determine the schema used for each record, use the schema to define the Iceberg table, and store the topic values in the corresponding table columns. ### [](#value_schema_latest)value_schema_latest Creates an Iceberg table whose structure matches the latest schema registered for the subject in Schema Registry. You must register a schema in Schema Registry. Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is without the magic byte or schema ID prefix in the record value. > 📝 **NOTE** > > The `value_schema_latest` mode is not compatible with the [`rpk topic produce`](#reference:rpk/rpk-topic/rpk-topic-produce) command which embeds the wire format header. You must use your own producer code to produce to topics in `value_schema_latest` mode. The latest schema is cached periodically. The cache period is defined by the cluster property `iceberg_latest_schema_cache_ttl_ms` (default: 5 minutes). ### [](#disabled)disabled Default for `redpanda.iceberg.mode`. Disables writing to an Iceberg table for the topic. > 📝 **NOTE** > > The following modes are compatible with producing to an Iceberg topic using Redpanda Console: > > - `key_value` > > - Starting in version 25.2, `value_schema_latest` with a JSON schema > > > Otherwise, records may fail to write to the Iceberg table and instead write to the [dead-letter queue](https://docs.redpanda.com/cloud-data-platform/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). ## [](#configure-iceberg-mode-for-a-topic)Configure Iceberg mode for a topic You can set the Iceberg mode for a topic when you create the topic, or you can update the mode for an existing topic. Option 1. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode= ``` Option 2. Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` ### [](#override-value-schema-latest-default)Override `value_schema_latest` default In `value_schema_latest` mode, you only need to set the property value to the string `value_schema_latest`. This enables the default behavior of `value_schema_latest` mode, which determines the subject for the topic using the TopicNameStrategy. For example, if your topic is named `sensor` the schema is looked up in the `sensor-value` subject. For Protobuf data, the default behavior also deserializes records using the first message defined in the corresponding Protobuf schema stored in Schema Registry. If you use a different strategy other than the topic name to derive the subject name, you can override the default behavior of `value_schema_latest` mode and explicitly set the subject name. To override the default behavior, use the following optional syntax: ```bash value_schema_latest:subject=,protobuf_name= ``` - For both Avro and Protobuf, specify a different subject name by using the key-value pair `subject=`, for example `value_schema_latest:subject=sensor-data`. - For Protobuf only: - Specify a different message definition by using a key-value pair `protobuf_name=`. You must use the fully qualified name, which includes the package name, for example, `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`. - To specify both a different subject and message definition, separate the key-value pairs with a comma, for example: `value_schema_latest:subject=my_protobuf_schema,protobuf_name=com.example.manufacturing.SensorData`. > 📝 **NOTE** > > If you don’t specify the fully qualified Protobuf message name, Redpanda pauses the data translation to the Iceberg table until you fix the topic misconfiguration. ## [](#configure-key-value-and-header-translation)Configure key, value, and header translation In addition to the [supported modes](#supported-iceberg-modes), `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections (see [Iceberg mode shorthands](#iceberg-mode-shorthands)). The `key` and `headers` sections change fields inside the `redpanda` system struct column (`redpanda.key` and the `value` field of each entry in `redpanda.headers`), while the `value` section changes the columns outside that struct. See [How Iceberg modes translate to table format](#how-iceberg-modes-translate-to-table-format) for the base row structure that every generated table includes. Use the following syntax to configure one or more sections: ```bash
: